IEEE awards Honeywell software team 2012 Control Systems Technology Award for OnRAMP Design Suite for Powertrain Control
PSA Peugeot Citroën and General Motors to co-develop three common vehicle platform projects; launch of vehicles expected in 2016

VisLab: 3D computer vision for autonomous driving

by Bill Cooke

VisLab 3D modeled view (foreground), bird’s-eye view (right) and photograph of scenario (back). Click to enlarge.

As inventors across the world work to make autonomous driving a reality, one of the most basic problems is for the vehicle to perceive its surroundings. Google’s vehicles rely on a lidar (laser radar) system. Dr. Alberto Broggi of the University of Parma and a spinoff company, VisLab, in Northern Italy believes 3D computer vision is an affordable and aesthetically acceptable way to capture much of the same data.

A pioneer of machine vision applied to driverless cars and unmanned vehicles, Dr. Broggi is the principal investigator of multiple projects involving autonomous vehicles, such as the ARGO prototype vehicle, the TerraMax entry at the DARPA Grand Challenge and Urban Challenge, and BRAiVE. Under his leadership VisLab organized the first intercontinental driverless trip in history: VIAC - VisLab Intercontinental Autonomous Challenge. VisLab is involved in basic and applied research, developing machine vision algorithms and intelligent systems for different applications, primarily for the automotive field.

The DARPA Grand Challenge for autonomous vehicles in 2005 dealt with an extreme environment (desert of the American Southwest) but only static objects were encountered. When two vehicles interacted, the slower vehicle was stopped so it could be treated as a static object.

The DARPA Urban Challenge of 2007 dealt with on-road scenarios but the autonomous vehicles had to interact with moving traffic. Traffic was simulated by 50 professional drivers who behaved rationally and followed the appropriate rules.

Their behavior allowed testing driverless vehicles in conjunction with vehicles whose trajectory was predictable by common sense rules, Dr. Broggi said, noting that over the recent history, the field of autonomous driving has come up with three representative cases:

  • Case 1: Fast autonomous driving in a static and predictable environment. An example would be a vehicle doing a timed lap on an empty racetrack.

  • Case 2: Autonomous driving in a complex known environment. Vehicles are programmed to drive a specific set of routes and are equipped with detailed maps, GPS and inertial systems. “The Google system has really precise maps,” he notes. Google’s self driving cars are an example.

  • Case 3: Autonomous driving in extreme and unknown environments. In 2010 VisLab undertook a drive from Parma, Italy to Shanghai, China using four autonomous test vehicles and a caravan of support vehicles. “It focused on perception, control of the vehicle, trajectory planning and route planning. We continuously improved the system during the trip,” Broggi said. The trip took 3 months and resulted in 25 Terabytes of data.

VisLab has a more affordable solution for 3D that is more easily integrated into the vehicle’s design (Parma is in Northern Italy and they take aesthetics seriously). “Our approach is based upon low cost and highly integrated sensors,” according to Dr. Broggi. He does admit the VISLAB solution achieves lower performance than the lidar system but they expect performance to increase as the image processing techniques that they are developing progress.

Broggi notes that “A lidar system (Velodyne HDL-64E S2) spinning at 10 Hz can process 2.5 million distance estimations per second...the resolution is very good up to 60m and still effective up to 120m...the sensor needs to be mounted on the roof of the car, it is not something that can be avoided...At this point, the cost is very expensive, perhaps twice the cost of the car.

In June 2012 USA Today reported that a prototype lidar system such as Google’s cost $70,000 but German supplier Ibeo was planning on offering systems for $250/vehicle in 2014 at automotive volumes. The $250 system only scans around 100 degrees, while the Velodyne scans 360 degrees all around the vehicle.

On the other hand, two stereo cameras (1024 x 768 pixels) can process 5 million distance estimations per second at 10 Hz. “As an indirect system, the stereo cameras have more noise and a lower accuracy at long distances...For almost every single pixel you have color and distance...if you compare the two streams, one with color and another with distance, you can get the 3D image... the system provides very good processing up to 50m away,” according to Broggi.

The images can be used for the two main tasks. They can be used to calculate the position and velocity of each point and the stereo images are easier to classify than lidar readings and patterns are easier to recognize as well.

Once we create our 3D map, we label each point. If the point belongs to the road we treat it differently than if it belongs to an obstacle. The next step is to take all of the obstacles together and create objects and then track the objects. You can even track very thin objects like poles on the side of the road. Detection, labeling and then tracking. …All of the processing is being done right now on a top of the line PC but we’re moving the processing to an FPGA (Field Programmable Gate Array) and then the PC will be just providing the high level control.

Challenges for 3D vision include night time driving with difficult lighting situations. The field of view and measuring precision are lower for a stereo system vs. LIDAR but can create resolution challenges but results are promising so far. Advantages include a lower cost, a cleaner integration aesthetically, improved robustness because there are no moving parts. There is also no interference with many systems working closely together.

—Alberto Broggi




You need both active (lidar and/or radar) and passive colour cameras, / stereo camera imaging systems.

Passive sounds very nice, all that high resolution image data, but you have to process it an deal with ambiguities. Stereo is very easy to talk about and demonstrate to a human, but much harder to get working in a machine.

There are lots of times when a passive system will get fooled, especially if you have low sun and lots of shadows across the road etc.

Hence you need active (Lidar / radar) which can see through this, and see in the dark / fog.

People are good at driving with uncertainty - they can drive in fog / snow / heavy rain (when they probably shouldn't), they can drive while dazzled, and usually don;t crash.

Programming a machine to work in increasingly uncertain conditions is very difficult, so you have to reduce the uncertainty, hence the active vision.

Nonetheless, the fact that humans can drive with stereo colour vision, and that you can buy the cameras + lenses for a couple of hundred dollars is very tempting. There is just the small matter of programming it.


"There is just the small matter of programming it."

LMAO. I'm currently working on image processing for recognizing the weft yarn on a loom, something seemingly simple. Small matter indeed. There's probably a reason why Google's driverless cars got a pass in California, no snow and lots of good weather.


Both technologies, when used simultaneously could eventually do a much better job than 99.9% of current humans drivers.

Removal the major cause for 85% of all road accidents (the human drivers) would be a great step forward for road safety.

The 'fun' drivers could practice their sport on race tracks and pay for all the consequences.


Yes, "You need both active (lidar and/or radar) and passive colour cameras, / stereo camera imaging systems.


You need many or most of the following:
* multiple passive, binocular cameras,
* sonar, ladar, radar
* active response (IFF, with speed and direction data)

to equal and better a human driver.

You need both/all and then capitalize on the multi-thread high accuracy, high data rate, massive/accurate memory of the computerized system.

A human has high resolution vision of the world outside the car by mostly ONE sense - binocular vision (with a little help from sound).

Human vision has very high fidelity/accuracy and good data rate only where the eyes are focussed and only when the view is clear.

The human "constructs" the world around the car in his mind using ancient instincts that are based on assumptions developed for 3 mph movement without being enclosed, in an unmechaized world.


TT...human drivers (failures) are responsible for 80% to 85% of car accidents, $$$B/year damages, 500,000+ injuries/year, 100,000+ fatalities etc.

This situation cannot be ignored forever and driver less vehicles is a logical solution.

'Fun' drivers could use out of town isolated race tracks to practice their sport and pay for all consequences but should not be allowed to drive on pubic roads and continue to injure and kill others.


Hd...such simple minded, unused and unworkable concepts have been around for a long time.

The new technologies referred to in this and the 20 Dec articles by Toyota and Continental offer a constructive, synergistic solution through new technology.

Technologies that HAVE, I hasten to point out, only recently become suitable.


TT...driver assistance, automated driver less vehicles and electrified vehicles will probably be three of the top technologies of the current century or the first half.

All three technologies will be available and mass produced, at an affordable price, by 2020/2025 or so.

The use of those three (very normal evolution) technologies may even spread as much as smart phones and tablets. They will certainly save more lives, injuries and material damages than cruel games on smart phones and tablets.

Accepting normal technical evolution is not so difficult if one has an open mind?


I know I should just ignore them and I know I will regret posting this but your inane posts with glib predictions of the future are like Chinese water torture.

HarveyD day you may have to accept that the world is and will be changing at a faster pace. Evolution is not always linear.

Automation is still in its early days but will find more and more applications by 2025/2050/2100 or so. Each generation will witness surprising applications.

The days when a few (Royalty, Religion and $$$ bags) could block evolution are numbered. Many (like Apple, Tesla and new Chinese firms) will come out from nowhere and mass produce new products that control groups will not be able to stop.

Look at what is happening to the information world. Printed news, information and data is fading away like horses and buggies did. ICEVs will follow during the next 2 or 3 decades. Too bad that lethal guns may not follow the same trend and be replaced by non-lethal units.


Taking lethal weapons and human driven over powered ICEVs out of service in USA could save about 68,000 lives/year, avoid over 500,000 injuries/year, save $$$B/year in repairs and property damages, save $$$B/year in health care and lost working days etc.

What is surprising is that many want to put more lethal weapons and more muscle cars and light trucks in circulation. We may be much closer to 'cowboy' pre-industrial days than we think.

The comments to this entry are closed.