Innovative Practice 2021 on Inclusive Employment and ICT

Talking camera app for people who are blind using AI and augmented reality

Microsoft Inc., the ICT-multinational company, developed Seeing AI, a free mobile talking camera app using Artificial Intelligence and augmented reality. The app helps people who are blind or have low vision to know more about who and what is around them. It assists users to perform daily tasks, such as reading a document, recognizing people, or identifying products. Seeing AI is available in 70 countries and nine languages and has assisted with over 20 million tasks.

“At Seeing AI we are always talking to people with visual impairments to understand their challenges.”

About the practice at a glance
Name of Innovative Practice:Seeing AI
Organisation:Microsoft
Country
of Implementation
United States and 70 countries worldwide
Start Year2017
Sub-CategoryA project related to ICT

FACTS & FIGURES

  • Seeing AI is available in 70 countries
  • The app is adding seven new languages in 2020, which brings the total to 16

PROBLEMS TARGETED

There are few free tools available to convert visual information into audio in multiple languages.

SOLUTION, INNOVATION, AND IMPACT

The Seeing AI app is available on iPhones and iPads. Users download the free app and can select a variety of functions to support them to complete tasks, such as reading text and handwriting, recognizing known people, distinguishing between currencies, and detecting colours and light. Users just hold up their phone or take a photo to hear a description of what is captured by the camera.

The app was developed with academic researchers and the blind community, being continually tested with a diverse user-base around the world with the help of NGOs.
Seeing AI is fully compatible with screen readers, and it supports higher contrast colours and larger font sizes for increased legibility. It is currently available on iPhones and iPads in nine languages: English, Dutch, French, German, Italian, Japanese, Turkish, Spanish, and (Brazilian) Portuguese.

Seeing AI was originally created for people who are blind or have low vision, but it is also benefitting people with learning differences and English language learners.
An estimated 20 million tasks have been completed with Seeing AI in 70 countries from 2017 to 2020.

A blind man holding a white cane in holding up a smartphone and smiling into the camera.

Microsoft Seeing AI Engineer Saqib Shaikh photographed at Studio C on the Microsoft campus. ©Red Box Pictures P.O. Box 9901 Seattle, WA 98109

FUNDING, OUTLOOK AND TRANSFERABILITY

The app was developed by the Tech-for-Good team, which is part of Microsoft’s AI Ethics & Society group. The group combines emerging technologies with human-centred experience to design and build ICT solutions. The team considers disability to be a driver of innovation, believing that solutions using inclusive design for persons with disabilities will eventually become mainstream for all customers.

The Tech-for-Good team will continue to work with the blind community to develop and refine solutions that leverage the latest technologies to support independent living.

FACTSHEET

Download factsheet as accessible word
download