Our assignment was to integrate a voice user interface, aiming to elevate the user experience and increase engagement. To assess the effectiveness of this new feature compared to the existing one, we conducted a usability test, utilizing both control and treatment groups for comprehensive evaluation.

Karen Ting - UX Designer

July 2022 | 1 week

UX Design, User Research, Prototyping

Figma, Pencil & Paper

As Berkeley students receive free bus passes, the AC Transit app has become a popular choice among them. Primarily used to check upcoming bus times, the app also offers additional features such as bus routes and navigation. However, many students tend to prefer using other applications for navigation instead. Moreover, some students tend to avoid using the app altogether because they find it challenging to navigate and difficult to use, especially when they are under time constraints.

Research students' bus riding habits

Introduce a novel feature to discourage their reliance on external app

Conduct a thorough usability testing, employing both control and treatment groups for a comprehensive evaluation

We conducted interviews with three Berkeley students aged between 18 to 25, with a mix of some heavily relying on the bus for transportation and others not relying on it as much. None of the three consistently used the AC Transit app. During the interviews, we delved into their transportation habits and also asked them to interact with the application through a sample scenario to gain insights into their experiences and perceptions.

Here are some key observations:

Users only use the app for bus schedules and the bus routes with some preferring Google maps over the transit app

Everyone rated the difficulty level in using the app to navigate a two

Difficulty navigating since the application does not allow first person viewing for navigation

Loading time which can cause the user to miss the bus

Afterward, we proceeded to create a persona, which became essential guides during our design process. These personas helped us pinpoint specific task flows that needed improvement. By utilizing the insights gathered from the interviews with students, we were able to define realistic user goals and identify common frustrations.

GOAL

more efficient way to line-up the times she goes to the bus on a time constraint

FRUSTRATION

misses bus when she has to search on the app

Then we created Sam.

Sam is an inclusive and genderless system designed to engage users with a tone of casual friendliness, emanating a caring and compassionate aura. Their primary goal is to assist users in reaching their bus on time and navigate the bus system effortlessly. Sam's voice exudes a smooth, calming, and soft-spoken quality, with a mellow pitch that strikes a balance between not being too high-pitched and not excessively deep.

Their character is characterized by trustworthiness, respect, patience, and amiability, creating a sense of reliability and comfort for users. Sam will gently encourage users to make it to the bus stop on time, but they will also exhibit patience and soothing support if users encounter frustration. Thanks to their unwavering trustworthiness, users can always rely on Sam, knowing they are readily available whenever needed.

After creating Sam, we envisioned three narrative scenarios illustrating interactions between Sam and Shelly:

Shelly, already familiar with the route and buses, requests bus times from Sam. Sam promptly narrates the upcoming bus schedules. Additionally, Sam provides Shelly with the option to set a countdown timer, give step-by-step directions, and even narrate the estimated travel duration.

While walking alone at night, Shelly activates safe mode on Sam. Sam becomes her guiding companion, leading Shelly to the bus stop and recording the duration of her walk for added safety and reassurance.

Shelly needs to know the bus times but is more comfortable with Spanish. Thankfully, Sam easily accommodates her language preference and delivers the bus schedules in Spanish, ensuring seamless communication and understanding for Shelly.

Then, we proceeded to create dialogue flows, which helped us gain a deeper understanding of the interactions that Sam would have with users.

After developing the dialogue flows, we tailored two distinct versions for the control and treatment groups. The control group was presented with a map interface, requiring them to navigate to the bus stop independently. In contrast, the treatment group was provided with a map interface along with an additional narration feature for added guidance and convenience.

Following rough sketches, we proceeded to develop a prototype using Figma. Considering AC Transit's existing app, we integrated a dedicated interface for interacting with Sam, complemented by a voice button accessible when users open the map. Within the dialogue interface, users can review the conversation transcript, with a microphone interaction indicating when Sam is listening.

Furthermore, we introduced a guided map feature that offers users two options: a self-guided mode or a narrated guide. This enhancement ensures users are directed to the nearest bus stop efficiently, with the added convenience of Sam's voice providing directions in the treatment group

home page

after clicking activation button

suggestions on what to ask sam and are able to click mic button to ask something else

transcript of conversation

users can see what they say to ensure it is corrected

lines next to mic button shows that Sam is listening in

end of conversation

Sam tells user what is happening next

control group - self guided map

Sam activation button top right if user wants to speak to Sam again

current map on app with static image and estimated arrival time on the bottom

treatment group - guided map

Sam announces the estimated time arrival at the start of narration

directions on the bottom along with narration

narrated guide

narrated guide

We designed an experiment with a control and treatment option with the overall goal being to assist users in finding bus times, nearby stops, and directions in a more efficient manner.

KEY QUESTION Do the users want a visual list of the results? Narration? Or both?

HYPOTHESIS Both in case the user doesn’t hear what Sam says and people digest information differently.

Following the prototype development, we recruited three Berkeley students to participate in the study. Each student experienced both the control and treatment options, with the order in which they encountered each option varied. For the treatment option, we utilized the Wizard of Oz tactic by having my partner assumed the role of Sam, while I took notes on the interactions and observations throughout the sessions.

Here are some key observations:

No significant difference in the time it took for participants to reach the bus stop between the treatment and control options

Spent less time looking at their phones when the narration feature was present

Participant 2 expressed a preference for verbal navigation, stating “it made walking much easier”

Participant 3 showed a preference for no narration during shorter trips, while they found the narration beneficial during longer trips

Our initial hypothesis was that users would prefer and find it easier to use both the visual map and narration together. However, when comparing the time taken for the treatment and control options, we did not observe a significant difference; the time was roughly the same with or without the narration.

Despite the lack of explicit verbal feedback from most participants regarding their preference for narration, we noticed a pattern where narration led users to check their phones less frequently during navigation.

Based on our observations, it appears that providing the option for users to choose between a visual map with or without narration would be beneficial. This way, users can decide whether they want the added guidance or prefer to navigate independently.

Given the project's constraints and requirements, we acknowledged that the idea and experiment could have been more comprehensive and fleshed out. In hindsight, we would have recruited more participants for the initial research and the experiment itself to enhance the study's validity and to gain more insight.

A significant observation was that participants took longer during their first navigation to the bus stop compared to subsequent attempts because of their familiarity with the route. To achieve a more accurate comparison, we would have chosen a second bus stop with similar navigation characteristics for the experiment. Although we wished to incorporate more features for Sam, time constraints limited our ability to test them thoroughly.

This project provided valuable insights into user research and experience. Our focus shifted from interface to user experience, especially considering that this was an existing application. For the first time, we drafted an experiment design with a hypothesis to test, which was an enlightening experience compared to previous usability tests that primarily aimed to gauge users' interactions and thoughts about the product.

Overall, the project emphasized the importance of research and testing in validating our assumptions and understanding user preferences accurately. The experience taught us the significance of data-driven decision-making and how it shapes the success of a project. As we continue to gain experience, we recognize the potential for iterative improvements in future experiments, leading to more robust and insightful results!