LookTel smartphone system tells you what you're looking at
By Paul Ridden
March 28, 2010
Shortly available for consumer beta, LookTel is a smartphone object recognition package which is intended to help the visually impaired correctly identify such things as money, CD titles, landmarks and so on. An image captured by the phone's camera is sent to a PC which then quickly scans through a database and when a match is made, the result is returned to the phone and spoken to the user. Sighted user assist, a handy text-to-speech function and guided interface control also feature.
Capturing video and using sound to act as a guide for the partially sighted or blind has been reported in Gizmag before, the Cognitive Aid System for Blind People creates an acoustic map of a user's surroundings using lasers and video and then feeds this to the user via headphones.
LookTel's approach caters for individual objects, combining already widely available smartphone technology, image recognition software and vocal descriptions.
After years of intensive testing LookTel will be made available in beta form in the Spring and aims to provide those who suffer from impaired vision with a little more independence. The system is said to work on any Windows Mobile smartphone and turns the device's camera into an object scanner. Pointing the phone in the direction of the item to be identified captures and sends an image back to a PC.
BaseStation software waits patiently on a computer for a signal to come in and then searches through an image library until it finds a match. For items around the home that can't easily be determined by packaging or design (such as glass jars or plastic containers), special labels are available allowing the user to tag such things up and then record a custom description which is stored in the software's database.
Once identified, the result is sent back to the phone and the user is told what's in front of them through the smartphone's internal speaker. The system can be "taught to recognize all the objects and landmarks you wish to identify" and with some assistance from a sighted helper, can become a "helpful assistant for many tasks where vision makes a difference in your independence".
More than object recognition
As you can see in the demonstration video below, having the system read out a page from a book or magazine is as simple as taking a photograph and waiting a short while for the software to translate the image into an audio reading. Useful for those with failing vision who have difficulty seeing small text on a page.
If a user is out and about and gets into trouble, a live video feed can be sent to a sighted person for assistance. As well as seeing the user's surroundings, the helper is also provided with a position on Google Maps via the GPS capabilities of the smartphone so they will be able to either offer turn by turn guidance to get the user to safety or come to the rescue themselves.
With phones that support the speech recognition features of Microsoft's Voice Command, LookTel can help initiate calls or ask for the time and so on. The company has also worked on ways to turn a smartphone's touchscreen user interface into a voice-guided tactile experience, identifying menu items as a finger slides around a display grid and then allowing the user double-tap activation.
As the system uses familiar everyday technology such as computers and smartphones, there is great scope for bringing this visual assistance system within practical and affordable reach of those with failing eyesight or suffering some impairment. How much use it would be to those totally without vision will no doubt be revealed during the beta development phase.