As we enter the era of Physical AI, autonomous driving and embodied intelligence are creating growing demand for high-quality, high-confidence ground-truth data. In response, 6D full-color lidar is emerging as a next-generation ultimate sensor.
In April this year, Hesai unveiled Picasso, the world’s first 6D full-color ultra-sensitive lidar ASIC platform, achieving native pixel-level alignment of high-precision 3D spatial information and high-quality RGB color data at the chip-level, producing inherently consistent, estimation-free ground-truth data. This not only eliminates the decision-making hesitation caused by sensor conflicts in autonomous driving systems, but also positions native RGB-D data as critical infrastructure for perception and data in the Physical AI era.
Reducing Fusion Errors and Information Conflicts for More Reliable Autonomous Driving Decisions
In autonomous driving solutions, cameras and lidar both serve as the “eyes” of Physical AI. Cameras capture color and semantic information, while lidar provides precise distance and spatial information. Traditionally, these separate data streams must be synchronized, calibrated, and fused before they can be interpreted by the system. Any discrepancy between the two can increase fusion complexity and affect the consistency of object localization and decision-making. Research has also shown that lidar-camera fusion relies on precise temporal synchronization and spatial calibration, with synchronization offsets, object motion, and transmission latency potentially introducing mapping errors.
Hesai’s 6D full-color lidar takes a fundamentally different approach. Instead of relying on conventional system-level or optical-level fusion, it brings fusion directly to the chip level, natively integrating RGB and depth information into a single data source. Each pixel carries inherently aligned color, position, and distance information, reducing cross-sensor conflicts and providing Physical AI with more unified and consistent perception data.
The benefits extend across real-world driving scenarios. At traffic lights, color and spatial information are inherently aligned, enabling the system to identify signal states without the need to align and fuse separate camera and lidar outputs. In rain and fog, color information can help filter noise more efficiently. For ambiguous objects such as a white plastic bag blown across the road, visual and spatial information no longer needs to be repeatedly cross-checked between sensors. And for distant objects occupying only a handful of pixels, from traffic cones to children’s bicycles and pets, chip-level alignment keeps color and distance precisely matched, reducing localization errors caused by image-to-point-cloud misalignment.
From fusing separate data streams to capturing one unified source of perception data, 6D lidar delivers high-confidence perception input for safer, more reliable autonomous driving.
Native High-Quality RGB-D Data Accelerating Physical AI Model Training
As Physical AI is accelerating into the real world, it needs develop the ability to perceive, understand, and act in real environments, making the acquisition and scalable production of high-quality data a critical challenge for model development. NVIDIA has also highlighted the limited availability, high collection costs, and difficulty of scaling real-world robotics data. Whether for autonomous driving, robotics, or world models, AI needs to simultaneously understand not only what an object is, but also where it is and how it moves. By combining visual semantics with 3D spatial information, RGB-D data provides a critical foundation for machines to develop a richer spatial understanding of the real world. Research also shows that combining 2D imagery with 3D spatial data can enhance model performance in tasks such as spatial relationship understanding, affordance prediction, and robotic manipulation.
Traditional RGB-D data typically requires separate collection of images and point clouds, followed by time synchronization, extrinsic calibration, data registration, and 3D annotation. This multi-source data processing pipeline is long, costly, and prone to introducing errors during conversion and registration. As foundation models and world models demand rapidly growing volumes of real-world training data, this inefficient data production approach will become a bottleneck for model training at scale.
Hesai’s Picasso SPAD-SoC integrates RGB sensing and time-of-flight (ToF) ranging at the chip-level, ensuring that every point output by the lidar natively carries both RGB and XYZ information with chip-level spatiotemporal alignment. No stitching or reverse inference needed, as visual and spatial information are unified from the moment of data generation.
For Physical AI, this means direct access to native RGB-D data that turns uncertainty into clear, measurable information. It enables automated annotation, shortens the pipeline from data collection to model training, improves data production and training efficiency, reduces the cost of training at scale, and ultimately accelerates model iteration.
One Ultimate Sensor Redefining Perception for Physical AI
Traditionally, enhancing machine perception meant stacking more sensors. But adding cameras, lidars, and other sensors also brings more hardware, wiring harnesses, interfaces, and complexity in time synchronization, calibration, data registration, and fusion computation. The growing trade-off between perception capabilities and system complexity and cost is becoming a new challenge for the industry.
Hesai’s 6D full-color lidar integrates high-precision 3D perception and RGB information into a single sensor, capturing distance, position, color, and other multi-dimensional information in one pass. As the perception architecture evolves, the 6D lidar will take on some of the visual information previously provided by standalone cameras, reducing the number of sensors and shortening the cross-sensor fusion pipeline, thereby simplifying system architecture and lowering hardware, compute, and development costs.
The value of 6D lidar goes beyond simply eliminating one sensor. By capturing richer, multidimensional information within a single sensor, it marks a shift from multi-sensor fusion toward a more integrated perception architecture, pushing beyond the boundaries of conventional perception and emerging as a super sensor for the Physical AI era.
