Multi-modal for car experience

Role

Conversation Designer

Multi-Modal UX Designer

UI Designer

Deliverables

Multi-modal UX design

Low- fidelity wireframes with multi-modal design

UI Design

High-fidelity mockups and prototypes
UI elements

Conversation Design

Competitive analysis
User research
User journeys and task flows
Sample dialogues and error handling solutions
Intent design and prompt design
Usability tests and findings

Project Specifications

Duration

8 WEEKS

Company

NUANCE (Cerence)

Tools

Omnigraffle
Sketch

Introduction

At the heart of Nuance's ambition to redefine the car experience with AI-powered solutions, I embarked on a journey to integrate eye-gaze detection into Dragon Drive, now celebrated as Cerence Drive. This narrative unfolds over eight transformative weeks, where I navigated through the roles of Conversation Designer, Multi-Modal UX Designer, and UI Designer, crafting a multi-modal UX design that bridges the car with the world outside.

The Challenge

Nuance envisioned a future where the car is not just a vehicle but a hub of connectivity. The mission was clear: to design features showcasing this vision through visuals, eye gaze, and voice. The challenge was to not only envision but also execute a multi-modal in-car experience that could seamlessly integrate with the driver's environment.

Research: unveiling the potential

My journey began with a deep dive into existing research on the Nuance Dragon Drive head unit. By studying usage frequency and feature-scenario connections, I gleaned insights into the features that drivers value in their interactions with their cars.

Brainstorming: Imagining the Possibilities

Envisioning a user immersed in the driving experience, I asked, "What does the user see, and what information do they seek?" This led to the conception of GAZE functionality, focusing on both the interior and exterior of the car. Sketching these scenarios, I explored the connectivity between the car and the external world, notably through restaurants, hotels, shopping malls, and general queries.

Objects outside a car

Concept validation

After considering the available data sources, existing voice supported domains, user pain points, popular scenarios, and optimal outcomes, I narrowed the ideas down to a few topics:

restaurants
hotels
general object query
shopping malls

After this, I set out to explore the restaurant topic in detail through use case mapping.

Designing the Experience

From Use Case Mapping to Storyboarding

Collaboration with PMs, developers, and speech engineers helped refine and validate the feasibility of our ideas. Storyboarding became a pivotal tool in communicating UX ideas, receiving stakeholder feedback on technology constraints and feasibility.

VUI Design: Crafting Conversations

I initiated the VUI design with a focus on the "happy path," progressively integrating complexity. This approach ensured a natural flow, mimicking human conversation while collaborating closely with engineers to align on system logic and database parameters.

A Happy Path Example

VUI elements

To complete dialog flow, I usually carefully think through all the intents, slots, utterances, prompts, GUI screens, earcon, hints, and system logic. During this phase, I worked very closely with engineers to make sure I have all the parameters that I needed in the database, and to make sure that the system logic is feasible.

GUI Design: Enhancing Interaction

Acknowledging the voice-first nature of in-car features, I aimed to augment this with an intuitive graphical user interface. Starting with low-fidelity wireframes, I engaged stakeholders in validating the design's feasibility, eventually evolving these into high-fidelity mockups. This phase underscored the importance of simplicity and clarity in in-car screens, where driver attention is paramount.

My full set of deliverables for this project includes: VUI flow, GUI wireframes & hi-fi mockups, prompt lists, and intent lists. After finishing all the designs, I moved to the testing phase.

Testing and Refinement

A week of intensive usability testing across three offices brought invaluable insights, leading to several refinements in our design. This phase was crucial for validating the user experience and ensuring the design met our users' needs and expectations.

AI Training

We constantly collected possible utterances/phrases to deliver to the development team, so that they could better organize the data and train the AI.

Results and Reflections

The feature's launch marked a significant milestone, with its popularity among users affirming the project's success. Nuance, now Cerence, not only secured new clients but also featured this innovation prominently on their website.

Reflecting on this journey, I recognize the evolving nature of multi-modal design and its expanding technological landscape. This project underscored the importance of considering multiple dimensions of user interaction, from visual and auditory to future possibilities of sensory engagement.

Final Thoughts

This project was a testament to the power of collaboration, innovation, and user-centered design. It taught me the intricacies of multi-modal design and the need for a holistic approach to UX, blending touch, gaze, voice, and potentially even more senses in the future.