Designing real-time pocket AI for sign language translation to bring people together. 

Project background

There are over 466 million deaf and hard-of-hearing people worldwide. HandPro, Mongolia’s first virtual sign language AI, emerges as a pocket translator to bring people together through technology and communication. 

The HandPro project was handpicked and selected from 30 sustainable start-ups with a mission to solve a major social issue. 

CLIENT

HandPro

COMPLETION

November 2022

DURATION

10 weeks

ROLE

Design Lead

UX Designer

UI Designer 

CONTRIBUTORS

Sanjaa (Junior Designer)

Uunza( Design Sprint Lead)

Tulga ( Senior Designer)

HandPro (Development) 

THE ASK

Create a friendly and accessible tool to serve individuals with a disability and caregivers as learning, interpreting, job-finding, and emergency calls.

Problem

HandPro was born out of a mission to reach the deaf and disabled community through inclusive design and technological innovation by launching Mongolia’s first virtual sign language interpreter. 

THE GOAL

How might we design an all-around accessible experience using a personable 3D translator that automatically translates text and audio to sign language?

Secondly, how might we design a tool that can help individuals with a disability integrate into society?

Thirdly, how might we also serve the caregivers of disabled individuals to have an intuitive learning experience that helps them connect to their loved ones?

STEP 1

Discover

Research

Starting out, we had no prior experience designing a real-time sign language translator with AI, so we went into deep research to make HandPro the best accessible app on the market. 

FINDINGS

1. Based on our research and feedback, we designed a whole new design system that is fit to be inclusive for the target users
2. We had to make sure everything was accessible from the home screen of the application - like the emergency buttons and sign language interpreter
3. Every expendable button and message has to have a sign language interpreter video because from our studies the majority of individuals who are deaf or hard of hearing often struggle to develop or improve crucial literacy skills. 

Design Sprint brainstorming and sketching with clients

User Testing

We let go of what we already knew and attended to and interviewed deaf individuals by hiring sign language interpreters to find out what their needs and wants were in HandPro. 

USABILITY TESTING FINDINGS

1. The social learning aspect of the app was not clearly displayed on the prototype, instead, it displayed too many in-app features that weren’t relevant to the main goals.

2. Users did not understand what the chatroom was for. This showed that to create a useful social learning platform, we need to keep everything as minimal and straightforward as possible, to not distract and confuse the user from their learning experience.

STEP 2

Ideate

Low-fidelity prototype

We spent a strong focus on applying UX principles to the prototypes. From our usability testing findings and from weekly feedback from our client, we created multiple iterations of the sign language translator page, because we wanted to make sure the application was accessible to everyone. 

STEP 3

Design

UI Design

Once all wireframes and lo-fi prototypes were complete and approved, we completed the final UI design. Referencing back to the approved visual direction, we created a design that is approachable by utilizing a lighter color palette than the original branding colors and rounded corners. We also designed in-app accessibility buttons and SOS action buttons and placed them on the home screen of the app, so the user can access them in one-step action. 

OUTCOME

The HandPro team hosted Mongolia's first Disability Expo in November 2022, partnering with Microsoft, Unicef, and other major partners. At the event, HandPro soft-launched the HandPro application and introduced Mongolia's first and only AI sign language interpreter, Anu. 

In later versions, Handpro will be a tool for not only Sign Language interpretation but for all disabled people to find jobs through job listings, emergency calls, and courses for SL learning. 

Disability Expo 2022 Moments

Thanks for stopping by,

Let's chat!

LinkedIn

Dribbble

Email

Resume