A smartphone app using the camera to describe all sorts of objects and text around the user
- Google Inc.
- Start Year
- First published
“I have tried for many years to find an app that is easy to use and performs the functions that this one does.” Anonymous user due to data protection issues
Google, part of the US multinational technology company Alphabet, introduced Lookout in 2019, an app that uses the smartphone camera to assist people with low vision or blindness. The user takes a picture and gets a text or speech message with a description of objects, printed text, handwriting, and barcode/OR code identification. Google does not publish user numbers, but between 2019 and 2022 there have been more than 100,000 downloads.
People with visual impairments can have difficulty completing common tasks such as reading mail, identifying packaged food to cook a meal, or identifying currencies.
Solution, Innovation and Impact
Lookout uses the camera on a person’s smartphone to get more information about the world around the user, thus ensuring easy access on all Android devices. The aim is to make daily tasks more efficient, such as sorting mail and putting groceries away. Lookout can understand both printed text and handwriting, and it can recognize products not just by their barcodes or QR-codes but also by their packaging (since barcodes can be hard to find). The app can also provide captions for images, describe any people present, highlight particular details, and read any text in the image. Lookout works on all Android smartphones and has been localized to 24 languages. People with disabilities are involved throughout the research and design process.
Funding, Outlook and Transferability
The app is free to download and has no commercial goals for Google. Instead, the aim is to improve Google’s understanding of how to apply technology to assistive use cases. This includes both improving technologies such as text recognition and discovering better human-computer interfaces so that all users can easily interact with Lookout. This expertise then trickles through to other Google products. In the coming years Google aims to increase the focus on user needs, such as working in as many languages as possible and building models that are accurate and useful.