Archived from my previous site · Original wording retained

Shipping the Element of speech

Mar 11, 2025

Watch the original speech demonstration on YouTube ↗

What: Talking to robots

Starting from Assistant and the years it took for us to get to Gemini Live I was the lead design for a nascent Speech team. I worked to demonstrate a world where a user could speak more naturally to address an agent. For this breakdown I plan to focus on how we actualized linguistic concepts through interaction breakdown, prototypes and body storming. A lot of our work centered around three of the big four with linguistics. Mainly pragmatics or the study of how language is used in context, on a smaller scale sociolinguistics (language & society) and historical linguistics (language change).

Table represents a sample of NLP linguistics and IXD considerations.

Defining work streams: Bringing UXR findings, engineering technology and UX together

1. Pardon the interruption - The UX need was based on the tendency when hearing the response from the agent they are communicating with but the mic generally was not open during speech. Engineering need was how do we utilize our NLP stack to fulfill this… by utilizing NLU or parsing text and using RNN (recurrent neural networks) to comprehend meaning. And the UXR finding kept pointing to people needing less rules in conversation like breaking norms and interrupting the agent.

   
   

2. Partial Fulfillment - The UX need here was to begin to reflect state to show speed and develop an app framework that utilizes a natural interaction and reduces wait time. We eventually filed a patent for this under the idea of "anticipatory computing" or the idea that you could work through an app, building on needs, refining them, etc. similar to scenarios one might recall from Hollywood (Ironman) but more like help me find a restaurant I ate at last night… near kings cross… not inside the station… yes that one!"

   
   

3. Speak at your own pace - The UX need here came from the big roadblock that kept coming up in UXR about how people feel pressured like they are in a spelling bee or similar…

ESL UXR participant: "Alexa usually interrupts me so I get nervous and say as few words as possible."

How: Bringing it all together

Utilizing an e2e traditional IXD method of breaking down micro-interactions and addressing the tech, the spec and a degree of what users would experience in their day to day.

We aimed to direct our product roadmap through explorations that get into the nuances of linguistics. When we design all these systems we need to keep in mind there is a tolerance of false accept and that a user may not be pausing and then we make processing longer, etc. so we aim to have aspirations:

1. Convey degree of certainty - UX - voice plate appears but text is not bold

2. Understand unfinished utterances - UX - UI pattern of timer loaded and but we don't assume time letting user fill in the rest

   

3. Adjust listening endpoint - UX - pause the fulfillment and keep listening until we are certain a user is done

   

4. Understand emotional prosody - when uncertain show a disambiguation screen relating to context (e.g. calls)

   

5. Buy time - Show UI of action happening but pause the actual call for 2 seconds for user to give them that they are certain

   

6. Detect return addressee - possible side-speech update on UI and assistant responds minimally to indicate listening

   

7. Resolve inexact object reference - don't be pedantic by pointing out an error, just focus on fulfillment based on context

8. Listen while speaking - a user can barge in and ask for volume up (e.g. long name of a song)

   

9. Correct in-progress action - punt should be helpful and when info isn't available give tips related to similar queries

   

10. Modify free text field

    

11. Confirmation

    

12. Correct completed action - UX here is similar to voice typing and a user says scratch that, etc.

    
    

Why: Many users rule out chatting with agents

1. Users don't know the rules. We used to expect people to know how a robot should be talked to and generally it was command centric. Even when one off commands are accurately accomplished a users expectations are generally quite low due to lack of shared context awareness.

Wrap up: Three things (what is natural?)

What is natural? In the changing space of LLMs we need to temper this as our expectations change to quickly between task oriented and conversational. It is in these two "modals" that context will be key do users need a full UI? Do they just need better under the hood AI API operators?

1. Understanding unfinished utterances, buying time and generally delaying helps keep the thread of a users need, may keep a users attention and avoid the paralysis a pregnant pause brings (e.g. "un hunh", "mmm") It also ensures that we do not jump in before truly reflecting.

   
   

2. Listening while speaking or… Interrupting LLMs when they go on too long is paramount and a common pattern that exists to have a natural and straightforward dialog while maintaining context.

   
   

3. Be relevant and be out of the way… for LLMs and for Gemini we have a conversation space and are working through command and control with the retirement of Google Assistant.