Lip-reading computers can recognize different languages

Lip-reading computers for deaf people are a step closer with scientists successfully teaching computers to recognize different languages from the shapes and movements of people’s mouths.

Researchers at the School of Computing Sciences at the University of East Anglia conducted statistical modeling of the lip motions of 23 bilingual and trilingual speakers. The languages tested included English, French, German, Arabic, Mandarin, Cantonese, Italian, Polish and Russian, and the system was able to identify with very high degree of accuracy which language was spoken by an individual speaker.

“This is an exciting advance in automatic lip-reading technology and the first scientific confirmation of something we already intuitively suspected -­ that when people speak different languages, they use different mouth shapes in different sequences,” said Professor Cox who, along with PhD student Jake Newman, led the team.

“For example, we found frequent ‘lip-rounding’ among French speakers and more prominent tongue movements among Arabic speakers.”

Improving the system's ability to identify individual characteristics and ways of speaking is the next step. The research is part of a wider UEA project on automatic lip-reading funded by the Engineering and Physical Sciences Research Council (EPSRC). The current research builds on earlier work conducted by the University in conjunction with Surrey University. In 2008 it announced the development of a lip-reading system that converts videos of conversations into written transcripts. The researchers developed software to track face and lip movements and a database of expressions and lip movements corresponding to letter combinations.

Language recognition offers enormous potential to help deaf people and those who have difficulty hearing in noisy environments.

The technology could also be of value to law enforcement agencies in the fight against terrorism, crime and antisocial behavior. But while most of us have nothing to hide, it's also likely to raise the usual questions and concerns about privacy and increasing levels of (often covert) surveillance and capture of data.

Watch a video about this research over at New Scientist.

Karen Sprey