Timing beats resolution
Teams reaching for better data usually reach for better sensors: higher resolution cameras, denser LiDAR. In multi-sensor capture, the more common failure is quieter. The sensors disagree about when something happened.
The arithmetic is unforgiving. A vehicle moving at 10 meters per second travels 10 centimeters in 10 milliseconds. If your camera and LiDAR timestamps differ by that much, a box drawn on one sensor sits 10 centimeters off on the other. At highway speeds, or on a fast manipulator, the error is larger than the object features you are trying to localize.
Where the offset comes from
- Cameras timestamp at the start of exposure, the end of readout, or when the driver receives the frame. These can differ by tens of milliseconds.
- A spinning LiDAR takes a full rotation, often 100 ms, to build a scan. Points at the start and end of the sweep were measured at different times and from different poses.
- Host clocks drift. Two machines logging the same event will disagree unless they share a time source.
What we do about it
Hardware triggering where the sensors allow it, and a shared clock over PTP where they do not. We log both the sensor timestamp and the host receive time, so any gap can be inspected later. LiDAR scans are motion compensated using the platform pose.
A label that is geometrically wrong by 10 centimeters is not a small error. It is a confident lesson in the wrong thing.
How to check your own data
- Compare IMU angular velocity against rotation estimated from consecutive camera frames. A consistent lag shows up as a shifted peak.
- Project LiDAR points into the image during fast motion and look at the edges of moving objects.
- Track reprojection error per session. A rising trend usually means calibration has drifted.
Resolution is easy to buy and easy to see. Timing is neither, which is why it is worth measuring before you collect more.