Natural Motion

Looking at images with judder-free motion

Options

+ Search Research
+ Printable version of this page

Natural Motion has so far received two important awards. During last year's Internationale Funk Ausstellung (IFA) in Berlin, it received the 'Video Innovation Award.' This prize was for the innovation that this technique brought to the field of television and video technology. In July this year, during the International Conference on Consumer Electronics in Chicago, a further prize - the 'Outstanding Paper Award' - was presented. This prize was for the best article written on a topic in the field of consumer electronics.

Dr Gerard de Haan, electronics engineer with Philips Research in Eindhoven, is rather proud of these awards. But on questioning him more closely, he points out that in fact entire teams have earned the awards and share the honor. "I am far from being the only person to be involved with this success. A new idea on how to improve television pictures is still only a concept and not a product. The concept has first to be converted into practical working parts. The electronics, which is now in the form of a chip, is also not something which consumers are sitting around waiting for. It needs a TV chassis around it that exhibits the advantages which we have demonstrated. It was therefore a project in which a large number and variety of people were involved, both at Philips Research and outside. Cooperation was perfect during the project. And, it has paid off!"

To show what it is all about, De Haan switches on two televisions. The one on the left, a 100 hertz set, is the ultimate in viewing pleasure. The one on the right, also a 100 hertz set, appears to be the same. But when the two sets are switched on, there is a clear difference.

Both sets show the same pictures simultaneously. There is a sequence of pictures featuring a building complex filmed from a helicopter hovering in the air above. Similarly, there is a sequence filmed from a vehicle which is overtaking parked cars along a street. In another scene, the scattered seeds of a dandelion go floating by. And finally, there is sport, as various types of ball go whizzing across the screen.

When you compare the two television pictures, the picture on the left appears to judder when images move. Only the picture on the right shows images with a natural flowing movement. Those who have experienced the difference for themselves, go on viewing the screen on the right.

"The left-hand picture is the best that we could produce up to now", says Gerard de Haan. "With stationary pictures, we can achieve the same quality as a color transparency. But moving images that were filmed with a movie camera as opposed to a video camera, result in pictures that judder. Now to the right-hand set. It doesn't only just give an ultra-sharp static picture, it also offers a very good picture when images are moving."

But how do we know that this movement is really natural? Anyway, aren't images that move on television always a true reproduction of a movement? "Yes, that's correct," says De Haan with a grin. "But that's exactly where the problem lies, because with a sequence of movie film or video shots, only a limited number of frames per second are recorded."

"As early as the previous century", continues De Haan, "people realized that 12 to 16 frames per second were necessary to capture objects that didn't move too quickly. However, it was immediately apparent that at 12 to 16 frames per second, the projected picture flickered enormously. To show a picture that didn't flicker required more frames per second. But instead of taking these extra frames, each frame was projected several times in order to save film material."

"In conjunction with this, it must be realized that so-called 'large-area flicker' is something that you also see in static pictures. This has nothing to do with motion," says De Haan. "Using a pulsed light source, between 40 and 100 frames per second are necessary to project a slide so that the picture is flicker-free. At 40 frames per second it is reasonably flicker-free as long as pictures are not too large and bright. If that is the case, then 100 frames per second are necessary. This can also be traced back to the television standards: in Europe they are based on 50 hertz for normal television; in other words picture information is transmitted 50 times a second. In the USA and some other countries, it is 60 hertz. This is related to the electricity supply, which is either 50 or 60 hertz."

"Over the years, things have developed to the stage where motion pictures are filmed at 24 frames per second. In order to show that on television, the film must first of all be shown at a somewhat quicker speed, i.e. at 25 frames per second. The 25 frames are afterwards shown as 50 complementary half-pictures or 'fields' per second (see separate text: Television pictures).

"At Philips Research we have developed the television even further. At the beginning of the 1980's we realized that the larger and brighter that TV tubes became, so also did the amount of flicker. Our solution was the 100 Hz TV, where 50 fields per second are shown twice every second. However, that required quite expensive memories and the scanning frequencies increased. But the resulting picture had the quality of a color transparency. And when we introduced the system to the market, it was such a success that even our competitors copied us and also bought our chips for signal processing."

But is the 100 Hz TV now at its optimum, or can it still be improved? "For static images, perfection has just about been achieved. But moving images can still be improved, especially where the transmission of motion pictures is concerned. And, these account for more than half of the programs transmitted on television today. But motion pictures are still filmed at 25 frames per second, each frame being shown repeatedly in order to make the 50 fields that are necessary for transmission of pictures according to the 50Hz standard. This causes moving images to judder a lot, regardless of the type or make of the television set. Although the judder observed on a standard 100 Hz TV appears less obvious because each picture is shown four times, a judder-free picture combined with the flicker-free 100 Hz frequency would give the ultimate picture."

If the speed of moving objects that were filmed for the movies and shown on the TV screen was known, there would be no need to repeat pictures but to simply show the intermediate pictures with the moving objects at intermediate positions. This would give natural flowing, judder-free movement. What De Haan was looking for was an effective motion estimator: a way of finding out which parts of a picture change position, and thus be able to calculate an intermediate position from two known positions.

"Work in the field of 'Motion Estimation' has been going on for twenty years," continues De Haan. "A totally confrontational approach is to divide the picture into blocks. If you want to know how a certain block moves, you create a search area around it. Then, in the same area of the next picture, you look for a block that looks exactly like the first one. When I say 'Looking exactly like', I don't mean visual identification with the naked eye. It is done 'pixel-for-pixel' with motion vectors that give the direction and velocity of the moving pixels."

"There are, however, some real problems with such a method. With normal television speeds, the search area means that 4000 different vectors must be checked. A television picture is built-up out of approximately 400,000 pixels. Thus, if you were to do this for all pixels, it works out at a billion checks per picture, each requiring tens of operations. This is not possible with 'real-time' television pictures. You could do it with a very powerful computer, but not for a consumer price."

It seemed as though an impasse had been reached. Sophisticated motion estimators are used in large studios, but they cost around quarter of a million dollars. Of course everybody wants a perfect television picture, but there aren't so many millionaires in the world. To achieve the ultimate picture, De Haan had to conceive an entirely new type of motion estimator.

How was that done? "When you first hear how it happened, it almost sounds stupid. I settled on a motion estimator which always concluded that a scene was static. It didn't need to do much, it gave a perfect result and it cost hardly nothing at all. And the result was better than the enormously expensive estimator which, due to the large number of calculations it has to make, frequently runs into trouble."

"Then I thought: if a perfect picture is so important, I need to make an estimator that can estimate related vectors. Not by checking all of the vectors, but by looking at what had already been calculated in the vicinity of a particular block. This way you also take the following block into account. You simply assume that a moving object has certain dimensions and that the motion of one block very probably is linked to the motion of an adjacent block. That's roughly all there was to it!"

However simple it sounds, De Haan's method of '3-D Recursive Search Block-Matching' led to an enormous reduction in the number of picture processing steps. It was possible to reduce it to 10 per pixel. The complexity of the motion estimator was reduced by a factor of 200, while the quality increased by a factor of 9. De Haan: "It was right on the border between something that was impossible to make, and something that could be made to work very well indeed. It is something you dream of when you are in research. All of a sudden it became possible to improve the quality of moving images."

De Haan converted pieces of film into computer images and calculated the best intermediate images. Then he went to the appropriate product division within Philips, which in this case was 'Sound & Vision' and the specific business group for TV (BGTV).

"I had to convince them that my idea would work," recalls De Haan. "There was quite a lot of skepticism. Motion estimation was a rather academic subject that appealed to those with a very powerful computer who were prepared to spend the entire night doing calculations on a short piece of film. Nobody believed that I could do it in real time for a consumer price. I showed my simulations but they weren't convincing enough. They wanted to see it work with real-time television signals received via the antenna, before judging whether or not it was of interest to them."

De Haan's prototype was ready in 1992. When he demonstrated it, everyone saw immediately that it worked and they all became highly enthusiastic.

"Then it was decided to convert the idea into a product as soon as possible," says De Haan. "Fate was on our side because a group within the research laboratories had the means of integrating highly complex circuits onto a single chip. This was the 'Digital VLSI Design' group. VLSI stands for 'Very Large Scale Integration'. While this group was designing the chip, the Business Group TV began making the chassis into which it all would fit, as well as all of the software. Both groups were ready at about the same time, and within 18 months we had a product!"

The result was a product to make your mouth water. A large, clear, flicker-free and color-true television, on which motion pictures looked better than on any other television, and even better than in the cinema. This is because judder affects moving images in motion pictures shown in the cinema, just as much as it does if they are shown on television. It is caused by the repeated projection of frames.

"It is a nice argument for sales personnel, but you have to witness it for yourself as a consumer," laughed De Haan. "The only thing I want to stress, is that although it is a consumer product, it contains electronics which professionals would be pleased to own. One manufacturer that makes professional equipment for use in studios has already been in touch with me."

"He said: 'Your motion estimator is so good, it is better than the one we have in our professional kit which costs 200,000 dollars. You appear to make yours for a few dollars. How do you do it?' I must admit that our motion estimator is unique - not only in the field of consumer electronics but also when compared with professional equipment. This was made possible by the successful cooperation between Philips Research, Philips Sound & Vision and Philips Semiconductors."