Smart glove that translates sign language into English, Chinese text wins contest

SINGAPORE – A smart glove that instantly translates sign language into English or Chinese text is being developed in Singapore, potentially enabling deaf people to communicate without needing an interpreter.

The glove tracks the hand movement of sign language users with sensors that run along the four fingers and thumb, before interpreting the words in text via an app paired via Bluetooth.

The app also animates spoken English words that it records into sign language to help deaf people understand those who speak to them.

Titled See Your Voice, the project was the winning pitch at 2022’s Huawei Tech4City Finals on Tuesday, beating 140 other teams of students in the Inaugural competition that challenges students to use technology to tackle local issues.

Mr Gong He, 22, who is the business lead in the team of four, said they wanted to use technology to assist deaf people to communicate with others after the team learned about the daily struggles faced by some deaf friends.

More than 70 million people globally are deaf while the number of individuals with hearing loss in Singapore is estimated to be around 500,000, Mr Gong cited in his presentation on Tuesday.

According to the Singapore Association for the Deaf, there are more than 5,400 people here with hearing loss registered. This is on the rise due to an aging population, it reported.

Several deaf people interviewed said it was often challenging to communicate with those who couldn’t speak in sign language, and that they preferred to keep to themselves or only with other signers, said Mr Gong.

There is also a shortage of sign language interpreters and a lack of translation services in mainstream media today, he added.

Mr Gong, who just completed his Masters in Business Analytics in the National University of Singapore (NUS), said: “We may easily neglect our deaf friends in our community… We realized that there was a need by the deaf community to be understood and thought we could contribute with technology.”

The passion project started in 2020 among Mr Gong, NUS data science and machine learning student Sun Shizhuo, 23, NUS mechanical engineering student Huang Yunqi, 24, and computer science PhD student Xu Xuanqi, 22, from Peking University.

Using self-developed algorithms, they trained the glove to detect the hand movements of 8,000 words in sign language – a number which is expected to grow as they work with Huawei to bolster the system’s machine learning capabilities.