Monday, February 15, 2010

Why Can't My Computer Recognize a Red Wall?

People often ask why the bucket still makes so many mistakes. Why won't we make software that is able to distinguish a wall or a door from the rest of the picture. Why not indeed?

The short answer: Because we have no clue how the human eye sees a wall or a door.

There's a lot we know about human perception. However, there's a lot more that we don't know about it. Perception is still largely unchartered scientific territory.

Let me explain this in more detail with an example. Here's a picture of a room with white and red walls. Common sense says it should be easy to distinguish a red wall from a white wall. After all, the walls are clearly visable in the picture.
To the computer, however, they are no more than a collection of colored pixels.
If so, why not take out the white pixels and the red pixels seperately?
Because the white wall isn't white and the red wall isn't red. And to complicate matters even more, there's a lot of red and white to be found outside the white and red walls. Here's a random collection of pixels from the white wall: As you can see, there are a lot of colors in the white wall, some of them even white.
But surely the red wall must be easier? After all, this is a bold color, which can clearly be distinghuished from the rest. Well, here are some of the pixels in the red wall: Do you see the bold red from the paint can that was applied onto that wall?
Me neither.
Interesting, isn't it? The pixels of the red wall don't even contain the shade of red it was painted with. That's because the color of the pixels result from the reflection of the light on the painted surface. Not only the paint determines the color, the light does so too. What we think of as red, is in fact not red at all. How on earth we are then able to recognize a red wall as a red wall is an enigma which still baffles the scientists. The truth is that scientists are now only investigating IF computers will ever be able to recognize a red wall. The HOW is probably something for our grandchildren to worry about. So don't expect a perfect bucket tool any time soon.

That said, there are ways to make things a bit easier for the computer as well as the user. The bucket is one of them.
Then how does the bucket work?
Contrary to popular belief (and contrary to Photoshop, for that matter) it does not search for similarly colored pixels. Instead it uses an advanced algorithm, involving statistics, to make a calculated guess as to which pixels belong to the red wall. It is so advanced that if you use the bucket on a textured surface such as a brick wall, you will find that it will interpret the brick as well as the mortar as one and the same surface. It can pull that trick utilizing some very cool technology that we developed in cooperation with the University of Amsterdam.
We have some more tricks up our sleeves. But we haven't come round yet to implementing it in Colorjive because it involves quite complicated math which would draw too much computing power off the computer.

No comments:

Post a Comment