Speech interaction studies

Original supporting material for Shipping the Element of speech. These studies preserve the original wording and diagrams.

Defining work streams: Bringing UXR findings, engineering technology and UX together

  1. Pardon the interruption - The UX need was based on the tendency when hearing the response from the agent they are communicating with but the mic generally was not open during speech. Engineering need was how do we utilize our NLP stack to fulfill this… by utilizing NLU or parsing text and using RNN (recurrent neural networks) to comprehend meaning. And the UXR finding kept pointing to people needing less rules in conversation like breaking norms and interrupting the agent.

  2. Partial Fulfillment - The UX need here was to begin to reflect state to show speed and develop an app framework that utilizes a natural interaction and reduces wait time. We eventually filed a patent for this under the idea of "anticipatory computing" or the idea that you could work through an app, building on needs, refining them, etc. similar to scenarios one might recall from Hollywood (Ironman) but more like help me find a restaurant I ate at last night… near kings cross… not inside the station… yes that one!"

  3. Speak at your own pace - The UX need here came from the big roadblock that kept coming up in UXR about how people feel pressured like they are in a spelling bee or similar…

Original interaction studies

  1. Convey degree of certainty - UX - voice plate appears but text is not bold

  2. Understand unfinished utterances - UX - UI pattern of timer loaded and but we don't assume time letting user fill in the rest

    Original speech interaction diagram

  3. Adjust listening endpoint - UX - pause the fulfillment and keep listening until we are certain a user is done

    Original animation

  4. Understand emotional prosody - when uncertain show a disambiguation screen relating to context (e.g. calls)

    Original animation

  5. Buy time - Show UI of action happening but pause the actual call for 2 seconds for user to give them that they are certain

    Complete original storyboardView original storyboard ↗

  6. Detect return addressee - possible side-speech update on UI and assistant responds minimally to indicate listening

    Complete original storyboardView original storyboard ↗

  7. Resolve inexact object reference - don't be pedantic by pointing out an error, just focus on fulfillment based on context

  8. Listen while speaking - a user can barge in and ask for volume up (e.g. long name of a song)

    Original animation

  9. Correct in-progress action - punt should be helpful and when info isn't available give tips related to similar queries

    Original animation

  10. Modify free text field

    Original animation

  11. Confirmation

    Original animation

  12. Correct completed action - UX here is similar to voice typing and a user says scratch that, etc.