Was Blind, But Now Eyes See

Imagine trying to draw a car on a computer screen with the monitor shut off. With a computer program designed by a UC Berkeley engineer, this seemingly impossible task has become a tangible reality.

For the visually impaired, the inability to see a computer screen often poses a problem, especially as computer usage continues to become an increasingly essential component of life. In response to this problem, a visually-impaired UC Berkeley student has developed a computer software program to help alleviate this problem and enable the blind to successfully draw detailed images with their computers.

Researcher Hesham Kemal, a UC Berkeley electrical engineering and computer science department graduate student, often ran into obstacles while preparing for presentations he had to give during his graduate work on campus.

"(Hesham) had to give a talk in my class and had to do PowerPoint slides," said James Landlay, a professor of computer science. "When he wanted to have diagrams, he had to have someone else do it for him."

Although most people were often more than willing to help him design slides for his presentations, a problem frequently arose when others did not have time or were not available to lend a hand, Kamel said. Kamel wanted to develop a visually-impaired friendly device that would enable him to do all the work without the aid of a sighted person. He came up with the idea of creating a drawing board program for the visually impaired called the Integrated Communication 2 Draw program.

The ability to use a mouse and a computer monitor in drawing programs is often taken for granted, he said. Vision allows users to locate where they have left off on the screen. For the visually impaired, however, drawing is like trying to use a mouse with the monitor shut off.

"If you imaging a drawing program, imagine trying to draw with your screen turned off," said Landay. "You have no feedback about where you are and you have no way to reference back into things you draw."

The drawing program tackles the problem that visually impaired people have with computer-aided drawing, especially the problem of having to use a mouse and go back and relocate a previous point for modification.

Kemal wanted to provide visually-impaired users with a way to imagine a map on the computer screen. To do this, he used the metaphor of a telephone keypad in his program. Kemal chose the telephone touchpad because most people, blind or sighted, are familiar with its layout.

"My assumption is that if I give this to someone who can see and turn of the monitor, he will be able to draw," he said. "I wanted to use the telephone metaphor because I wanted to know exactly where the cursor is on the screen. If I move it away, I want to get it back."

The software program is based on a three by three, nine-cell, grid system similar in layout to that of a telephone keypad. The number one corresponds to the upper left hand corner, the number three to the upper right hand corner, and so on. The partitioning of the monitor into nine "cells" gives the user a sense of the layout of the screen and helps navigation and movement.

The program uses audio feedback devices to identify to a blind user exactly what position the cursor is located at on the screen. Kemal wanted to take advantage of the heightened hearing senses of the visually impaired and incorporated audio feedback mechanisms into the program, where different sounds and dings tell users where they have moved relative to the screen.

"There's both voice and non-voice feedback," Landay said. "Audio feedback is important."

Allowing visually impaired users to focus on their sense of hearing rather than touch makes the program much more compact and convenient for users.

"One strong point about the project is that it does not use big devices like braille printers and tactile displays," Kamel said. "We decided to have something portable, like software and the map function keys (correspond) to keys on the keyboard."

With the drawing program, even a person who is not visually impaired could draw an image onto the computer with their eyes closed, or even with the monitor turned off, Kemal said.

The project is one component of a larger project aimed at combining a number of departments on campus.

"The whole effort in human-centered computing. We want to build a center on campus with people from psychology, sociology, education, computer sciences, the Haas school, etcetera," Landay said. "It's really by a joint deal. It's a big effort to design technology with the human in the center of it.

In addition to locating their precise location on the screen, users of the program can also gain access to various shape and color pallets to make their drawings more detailed and colorful. The pallets themselves are also organized into nine cells and replace the traditional pull-down menus that computer users usually access via pull-down menus, which are usually accessed via a mouse. Users can then choose from a variety of shapes and colors for their drawing.

For detailed drawings, each of the nine boxes can in addition be broken into smaller nine-celled boxes for greater detail in drawing.

"There can also be a hierarchy," Kamel said. "If you want to construct the front wheel of a car, you can draw each object and group them all together. Inside each cell, we're using the same metaphor. The subgrid can move inside with you. In there, you can even go more detailed."

Kamel has already used his software program successfully for many of his projects.

"I use IC2D on my presentations," Kamel said. "The research papers I write with James, most of the figures I do using IC2D."

An early trial on three visually impaired subjects also proved successful. The three blind people were able to use the tool with little training.

"One of them drew a car," Landay said. "That was really interesting."

Kamel continues to modify the software to make it even more useful for the visually impaired.

"I just worked on a labeling method to label each object and give it a meaningful name," he said. "As you navigate the cursor around now, the voice feedback will read out the textual label of the object."

In addition to allowing a blind person to draw, the software also allows a blind person to imagine what the drawing made by someone who is sighted. The visually impaired person can use the gridding system and labeling system to reconstruct in his imagination a picture that appears on the screen.

"Labeling is very important because it will open up communication between the blind and others," Kamel said. "This tool would allow me as a blind person to communicate with a sighted person. It opens up communication between blind and blind as well as blind and sighted."

Currently, the labeling system and audio component of the project consists of an automated voice. Kamel said he hopes to improve this as well.

"Hopefully beyond this semester, the IC2D would have its own voice feedback mechanism," he said.

Tags: