A smartphone app using the camera to describe all sorts of objects and text around the user

Google introduced the app ‘Lookout’ in 2019 whereby users who take a photo with their smartphone receive a text or voice message with a description of objects, printed text, handwriting, and barcode/OR-code recognition. By 2022 the app had been more than 100,000 downloads.

Google, part of the US multinational technology company Alphabet, introduced Lookout in 2019, an app that uses the smartphone camera to assist people with low vision or blindness. The user takes a picture and gets a text or speech message with a description of objects, printed text, handwriting, and barcode/OR code identification. Google does not publish user numbers, but between 2019 and 2022 there have been more than 100,000 downloads.

Problems Targeted

People with visual impairments can have difficulty completing common tasks such as reading mail, identifying packaged food to cook a meal, or identifying currencies.

Solution, Innovation and Impact

Lookout uses the camera on a person’s smartphone to get more information about the world around the user, thus ensuring easy access on all Android devices. The aim is to make daily tasks more efficient, such as sorting mail and putting groceries away. Lookout can understand both printed text and handwriting, and it can recognize products not just by their barcodes or QR-codes but also by their packaging (since barcodes can be hard to find). The app can also provide captions for images, describe any people present, highlight particular details, and read any text in the image. Lookout works on all Android smartphones and has been localized to 24 languages. People with disabilities are involved throughout the research and design process.

Funding, Outlook and Transferability

The app is free to download and has no commercial goals for Google. Instead, the aim is to improve Google’s understanding of how to apply technology to assistive use cases. This includes both improving technologies such as text recognition and discovering better human-computer interfaces so that all users can easily interact with Lookout. This expertise then trickles through to other Google products. In the coming years Google aims to increase the focus on user needs, such as working in as many languages as possible and building models that are accurate and useful.

Pictures