Johns Hopkins Gazette: March 24, 1997

Would You Trust
A Robot To Sort
Your Chocolates?

Technology: Lights and
camera provide action
for visual identification

Phil Sneiderman
News and Information

A packaging robot, working in a candy factory in the near future, aims its electronic eye at a tray crammed with chocolates. The robot must fill a box with a specific mix of candy, but how can it tell a cream-filled morsel from a chocolate-covered peanut cluster?

A new object recognition technique developed at Hopkins' Whiting School of Engineering might help this robot hold onto its job.

Using three lights, a video camera and a computer, researcher Elli Angelopoulou has devised a new way to transform visual images into electronic "signatures." Each is a distinctive series of 11 numbers. By comparing these signatures, a computer can tell how closely two objects resemble one another. For instance, a smooth, dome-shaped candy and a bumpy nut cluster would have far different signatures.

How A Computer 'Sees' Chocolates

Images of two distinct chocolate candies.

Computer generated images of the same chocolates after the DeCov transformation.

Angelopoulou, a doctoral student in the Department of Computer Science--and an admitted chocoholic--has used the technique to identify coffee mugs, disposable razors, children's toys, and, yes, chocolate candies. It can even distinguish between two objects that have identical shapes but different colors. Her technique is primarily for recognizing objects with curved surfaces, a task that has long troubled computer vision scientists. The next step, Hopkins researchers say, is for robotics experts to incorporate Angelopoulou's breakthrough into a mechanical packaging machine.

"As part of an industrial object recognition system, this technique could expand a robot's capability," says Larry Wolff, who supervises the Computer Vision Laboratory at the school.

A mechanical device that interacts with its environment must have some sort of visual information about its surroundings, says Angelopoulou (pronounced an-jell-o-POO-loo). "A robot or a vehicle that moves by itself needs to 'see' where it's going," she says. "We need a way for it to recognize objects."

Scientists have invented electronic ways to identify some objects the way people often do: by following the edges until a rectangular shape, for example, is recognized as a door. But such systems do not work well on curved objects, such as vases, that have few edges to follow.

In 1995, Angelopoulou decided on a different approach: measuring the light that bounces off an object and into the lens of a video camera.

"My goal was to find an object recognition technique by looking at the intensity values," she says. "That meant I was not going to look at the edges. I was going to look at every pixel and the value the camera recorded. I believed I could collect enough information for an object signature just by looking at the intensity values recorded by the camera under three lighting conditions."

Two years later, here's how her system works: Angelopoulou aims one lamp in the same direction as her camera lens and positions the other two on opposite sides of an object such as a coffee mug. She shoots the mug three times, first switching one lamp on and off, then repeating the process with the other two. The camera captures this mug in about 10,000 pixels--the tiny points on a video screen that blend together and form an image. For each pixel, Angelopoulou's software computes a DeCov value, short for determinant of the photometric covariance matrix. These values provide critical information about how the object curves. Within a few minutes, the computer sorts and reduces these values to a distinctive series of 11 numbers. These become the signature for the mug. If another picture produces the same or nearly the same series of numbers, the computer "recognizes" that object, too, as a coffee mug. Different colors reflect light differently, so the system can also separate blue and red mugs that have the same shape.

Elli Angelopoulou, a researcher at Hopkins, aims her lights and camera at a piece of chocolate that her computer will identify, using her new object recognition system. Behind her, the bottom monitor displays the camera's view of the candy, while the top monitor shows the image after it is processed by the computer. Photo By Doug Barber -- 3/97.

This system is less sensitive to the electronic "noise" or interference that hampers other computer vision systems, Angelopoulou says. And, unlike those systems, it does not require extremely precise placement and measurement of lighting. But the process does have limitations. It will recognize a mug from many upright angles, but the computer will be fooled if the camera looks at the flat bottom of the cup. Also, the computer will only recognize identical objects if the lighting remains unchanged.

A factory could provide such controlled lighting, and Angelopoulou hopes other scientists will someday install her system in a robot that could sort and pack mechanical parts, toys or even chocolates. "Usually, what we do in smaller labs is just try to solve a small part of the problem," she says. "Then we hope someone else will integrate it into a device that solves a bigger part of the problem."

Go back to Previous Page

Go to Gazette Homepage