Talking camera app for people who are blind using AI and augmented reality

Seeing AI is a free app that uses the camera of devices to identify people and objects and describes them acoustically. Seeing AI can read texts, describe landscapes and emotions, and read the barcodes of products, thus helping the user to focus. In 2020 the app was available in nine languages and 70 countries.

Microsoft Inc., the ICT-multinational company, developed Seeing AI, a free mobile talking camera app using Artificial Intelligence and augmented reality. The app helps people who are blind or have low vision to know more about who and what is around them. It assists users to perform daily tasks, such as reading a document, recognizing people, or identifying products. Seeing AI is available in 70 countries and nine languages and has assisted with over 20 million tasks.

Problems Targeted

There are few free tools available to convert visual information into audio in multiple languages.

Solution, Innovation and Impact

The Seeing AI app is available on iPhones and iPads. Users download the free app and can select a variety of functions to support them to complete tasks, such as reading text and handwriting, recognizing known people, distinguishing between currencies, and detecting colours and light. Users just hold up their phone or take a photo to hear a description of what is captured by the camera. The app was developed with academic researchers and the blind community, being continually tested with a diverse user-base around the world with the help of NGOs. Seeing AI is fully compatible with screen readers, and it supports higher contrast colours and larger font sizes for increased legibility. It is currently available on iPhones and iPads in nine languages: English, Dutch, French, German, Italian, Japanese, Turkish, Spanish, and (Brazilian) Portuguese. Seeing AI was originally created for people who are blind or have low vision, but it is also benefitting people with learning differences and English language learners. An estimated 20 million tasks have been completed with Seeing AI in 70 countries from 2017 to 2020.

Funding, Outlook and Transferability

The app was developed by the Tech-for-Good team, which is part of Microsoft’s AI Ethics & Society group. The group combines emerging technologies with human-centred experience to design and build ICT solutions. The team considers disability to be a driver of innovation, believing that solutions using inclusive design for persons with disabilities will eventually become mainstream for all customers. The Tech-for-Good team will continue to work with the blind community to develop and refine solutions that leverage the latest technologies to support independent living.

Pictures