When lidar companies market the performance of their products, they often use a list of key specifications, the most common being Maximum Detection Range, Angular Resolution, Field of View and Points Per Second. Other important things they publish include Wavelength, Frame Rate, Power Consumption and Range Accuracy.
Companies would often claim leadership with one or two very impressive numbers, such as a detection range of 600m, or over 10 million points per second. Some would promote the idea that 1550nm wavelength is better than 905nm, as it allows the lidar to “see further without eye damage". This can sometimes be misleading, as lidar specifications are interrelated, and performance is not determined by a single metric or two.
Without enough information about the targets and conditions in which some of the numbers are measured or calculated, it can be difficult to draw solid conclusions about a lidar’s performance and usability based on its published specifications, especially when they are read independently.
One might be surprised, for instance, that there is no industry-wide standard for how Maximum Detection Range is measured; how accurately total Points Per Second represent a lidar’s efficiency depends on many things; and a seemingly superior imaging or detection technology has a series of tradeoffs that limit the lidar’s scalability for mass-market adoption.
So how do we know if a lidar delivers the performance needed for its intended use case?
That’s why we focus on data – real data, real results. Using data published by a fellow lidar company, we compared Cepton’s long-range Ultra lidar’s black tire detection capabilities with some of the leading products in the industry, based on actual point clouds collected from our recent Ultra sample:
For visual animals out there –
These results illustrate the varying capabilities of different lidars in detecting a black tire at distances ranging from 25m to 125m, based on the points collected in a single frame of lidar point cloud data.
At a distance of 50m from the black tire, two competitor products can barely detect it, whereas the point clouds from Luminar Iris and Cepton Ultra clearly outline the tire's shape. Cepton Ultra captured 67% more points than Luminar Iris, resulting in not only denser point clouds but also significantly better result for perception. Different companies use different perception algorithms, but one thing is universal – the more points from the lidar to work with, the better.
By nature of how lidar (or most sensor technologies) works, as the range increases, the number of points collected from the target decreases. At 75m and 100m, Cepton Ultra collected almost double the points from the target compared to what Luminar Iris did. At 125m, Cepton Ultra appears to be the only product that was still able to get returned signals from the target.
Luminar Iris uses infrared light at 1550nm wavelength, which is often associated with better performance and a higher eye-safety power threshold. However, these graphics demonstrate that 1550nm laser does not necessarily guarantee better results than the 905nm laser that Cepton uses for Ultra.
In fact, lidar performance depends on three building blocks: transmission, detection and imaging. While wavelength only indicates the type of laser used as part of the transmission method, detection and, more crucially, imaging technology make a huge difference in how efficiently lidars work. Read more on the subject here.
Therefore, lidar performance needs to be evaluated using actual point cloud data. This can be done in many different ways, as it is usually highly customer and use case-specific, based on our experience working with all of the global top 10 automakers.
Black tire detection is commonly adopted because the color and material of a tire make it a good representative of common low-reflectivity (or low-visibility) objects on the road. If a lidar can effectively detect a black tire at 50m, it will likely do well with a pedestrian and a car at the same distance.
Now, even the comparison we use in this article could vary if some of the parameters changed. Say, if the Luminar Iris cited operated at 5Hz or 20Hz instead of 10Hz – a default frame rate used in product marketing, then the results would have been different. While we cannot confirm the sizes of tires used by other competitors, our choice of a smaller tire (185/60/15) for Cepton Ultra supports the credibility of our results.
That leads us back to our initial argument – lidar specifications individually can’t indicate whether a product performs well or not. Therefore, it would be helpful to know what lidar manufacturers are talking about when they tout their impressive numbers. In our next blog post, we will take a closer look at commonly marketed lidar specifications and unravel what they do and don’t tell you.
