New languages

28 Apr 2016 news, languages

Today is a really exciting day for us and for all the developers that were patiently waiting for the support of new languages in Wit.

We are releasing 39 new languages in Beta. They will be added to our existing languages (English, French, German, Italian, Dutch, Spanish, Polish, Swedish, Portuguese, Estonian and Russian)

Albanian Croatian Icelandic Persian Turkish
Arabic Czech Indonesian Romanian Ukrainian
Azerbaijani Danish Japanese Serbian Vietnamese
Bengali Finnish Korean Slovak  
Bosnian Georgian Latin Slovenian  
Bulgarian Greek Lithuanian Swahili  
Burmese Hebrew Macedonian Tagalog  
Catalan Hindi Malay Tamil  
Chinese Hungarian Norwegian Thai  

As you may now, Wit heavily relies on machine learning techniques. This means that these new languages will become better over time (like a fine wine). For example, a built-in entity like wit/location may take a couple of weeks/months to be as good as the one in existing supported languages. The more it is used and trained by different Wit apps, the better it will get.

Wit also relies on a hybrid system based on rules and examples called Duckling for a few specific entities like wit/datetime, wit/duration or wit/number. We open-sourced Duckling on October 1st, 2014. Any help from our community of native speakers is more than welcome to:

  • improve the existing languages
  • create built-in entities for the new languages

To contribute to our open-sourced parser, visit duckling.wit.ai.

Team Wit



Bot Engine, day 7

19 Apr 2016 news

Last Friday, we reflected on the announcement of our new feature, Bot Engine, and announced our public roadmap to give you visibility on where Bot Engine is going.

We started executing on this short-term roadmap right away and today, we’re excited to release 3 items from the roadmap.

Chat window

This feature allows you to talk to your bot without leaving the Wit Console. Press ~ or click on the launcher at the bottom-right corner of the screen to start a conversation with your bot.

There’s one caveat though. Because (1) your bot’s actions are run on your server and (2) you don’t want to run destructive actions when just testing your bot, no actions are actually run. Consequently, you will have to manually update the context as you chat with your bot. When the bot triggers an action that usually updates the context, we will proactively show you the usual context updates for you to pick from. Once you pick your context updates, the conversation can continue.

Roles

We have added our famous Roles feature to Bot Engine. Roles allow you to differentiate two entities of the same type when you need to. The canonical example is a flight booking. You have two airports in the sentence and you need to know which is the origin and which is the destination. You can learn more in the docs, under the Roles recipe.

Action removal

As you build your app, you may be creating a lot of throw-away actions and templates. You can now go into the Actions tab and remove actions that are not used in your stories.

Team Wit



Bot Engine, day 3

15 Apr 2016 news

We’ve received a lot of great feedback from the community since the release of Bot Engine 3 days ago. Thanks everyone for reaching out.

I’d like to share the key findings and our short-term roadmap.

Findings

  • Developers are excited about Bot Engine! 8000 new signups in 72 hours.
  • The Story paradigm seems to be well received… but the console UX is a bit confusing
  • It’s too hard to test your bots
  • Some “pioneers” Wit.ai developers are looking for the good old intents
  • How to know when the user message is out of scope?
  • Bugs and incomplete documentation

So what are we doing about this? Here is what we are working on right now:

Short-term Roadmap

  • Chat with your bot in Console directly
  • Bring Roles back
  • Delete action/response in Actions tab
  • Automatic intent embedding, with two consequences
    • It will simplify intent handling
    • It will enable bots to detect if the message is totally out of scope
  • More compact view of stories
  • Redesigned Logs view
  • Conversations Inbox (turn logs into stories)
  • Conversation-aware entity extraction
  • Structured input (handle Messenger’s CTA – buttons clicks – for instance)
  • New languages
  • More tutorials and sample apps

… and much more to come! These are really the most urgent improvements we want to ship.

We keep this list up to date on the Github repo.

We’re spending a lot of time responding to tickets. Apologies if our response comes late sometimes, it’s hard to keep up with the huge interest. That said we are trying to answer to everybody, so keep sending your feedback!

Thanks again for your continued interest and support. Onwards!

Team Wit



Bot Engine

12 Apr 2016 news

Today we are releasing the first step of our vision for conversational bots: an early beta of Bot Engine.

Since we joined Facebook 15 months ago, the Wit community grew from 6,000 to 21,500 developers. Many of you are already making bots, leveraging Wit to parse messages into structured, actionable data. We are also building Facebook M, your personal assistant in Messenger. We’ve learned a lot in the process.

Making a bot seems easy until you try. And then, the curse of combinatorics bites you. Understanding the user intent is a necessary step, but that’s not enough. How should the bot respond? What actions should it perform? What question should it ask? This is the problem we are trying to help solve with Bot Engine. Here is how:

Key concept 1: Stories

To make a bot, there are two schools: rules or machine learning. (Everybody claims rules are bad and their bot is powered by AI, but when you really look under the hood, the core is often imperative.)

Machine learning is of course more desirable, but the problem is the training dataset. Training a Wit intent with a dozen examples works well, and it’s easy to leverage the community to get more examples. But in order to entirely learn the business logic of a bot of medium complexity, we would need many, many thousands of example conversations.

Rules (or any kind of imperative approach, including plain script/program) are kind of the opposite. The good thing with rules is, you can have a demo working after you write two rules. As long as you follow the script carefully, your bot will work and your audience will be impressed. But as you discover new “paths” in the dialog, you’ll add more and more rules, until one day everything collapses. You’re doomed by the curse of combinatorics. Any new rule conflicts with old rules that you totally forgot the reason for. Your bot cannot improve anymore.

Bot Engine is trained with Stories. Stories are example conversations. On top of user’s and bot’s messages, Stories also contain the bot actions:

When you create your bot, you just start with a few stories that describe the most probable conversations paths. At this stage, Bot Engine will build a machine learning model that deliberately overfits the stories dataset. Practically, it means that stories will behave almost like rules. It enables you to start beta testing your bot and collect conversations. You will then turn these conversations into new stories (exactly like you turn logs into new expressions in the original Wit). The more stories you have, the better the model becomes. Unlike rules, Stories can ignore each other; when you discover a new use case, you can add a new story without the need to take into account all previous stories.

With this approach, we are trying to get the best of both worlds: rules when you get started and don’t have much data, and AI once your dataset grows.

Key concept 2: Action

Trained by Stories, Bot Engine predicts the next action your bot may execute at each step of the conversation. This action is executed on your side, with your code. We’ve made this decision because we believe that developers need total freedom of platform and execution scope for their bot’s actions. You should be able to use any programming language of your choice and call any API you need. That wouldn’t be practical if actions were executed on our side.

An action typically modifies the context of the conversation. You pass the new context to Bot Engine when you need to predict the next action.

Actions also help relieve the pressure on the machine learning model: they encapsulate some business logic that would require a huge dataset to learn end-to-end. It enables Wit to train efficient models before you have hundreds of Stories.

Key concept 3: Inbox

In line with the original Wit philosophy, we think that you really discover your users’ expectations and logic only after you release a first version of your bot. As a result, the initial build should be as lightweight as possible, and the platform optimized around continuous learning and improvement based on actual usage.

In Bot Engine, we are extending the concept of the Wit Inbox to the conversation logs: we will provide an easy way to turn logs into new Stories. This is the learning loop. Each time you add a new Story, your bot’s models are rebuilt in real time.

We are trying to solve a very hard problem, and we are not pretending that we have the definitive solution. There are many kinds of bots, and many different ways to address the problem. This is an early beta: our hope is that you won’t hesitate to contact us when you are trying to do something that doesn’t seem to be well handled. We are committed to spend a significant amount of time supporting the community and learning more about the problem.

While at first glance it may look easier to just use scripts, AIML, or slot-based workflow templates to build bots, we believe that these approaches soon lead to bottlenecks. Bot Engine might not be as easy as putting a few rules together in a basic bot, but it’s designed to handle scaling complexity with simplicity (special dedication to our hero Rich Hickey), as additional stories don’t need to comply with a complex net of existing rules.

Some of us at Wit have been building bots for 15 years. We’ve tried many things that don’t work, and we’re still looking for what could work. Please give Bot Engine a try, and give us your feedback!

Alex Lebrun & Team Wit


Note: If you already have a Wit app

  • First of all, thanks! We are thrilled to continue to serve you.
  • Your existing app will continue to work as before. The /message API is unchanged.
  • You won’t have the Stories tab in your existing app. We’ll add a migration process very soon.
  • If you create a new app, you’ll get the Stories tab. You will also notice a change in the Console: we’ve merged the concepts of intents and entities under an Understanding tab, and slightly updated the training UX.



20,000!

17 Mar 2016 community

Our Wit community just crossed 20,000 developers. This is a very important milestone for us as developers are the core of our project. We started Wit.ai with the vision that no solution can be made using a closed, centralized and managed approach. We’ve been building an open, distributed, community-based platform that makes it easy for developers to build apps that users can talk to.

Our community went from 6,000 at the beginning of 2015 to 20,000 developers powering thousands of apps and devices. It spans 3,000+ cities and 141 countries!

We wouldn’t be where we are without you so we wanted to take a moment to thank you for your involvement.

Team Wit