Multimodal Sensor Data for Robot Learning
Vision, depth, proprioception, force-torque, tactile, IMU and audio aligned to a measured synchronisation error with documented calibration and per-episode lineage.
Scope a Capture Programme How We Verify Synchronisation
Multimodal Does Not Mean Several Files from the Same Session
A video folder, a state CSV and a force log recorded during roughly the same session are not verified multimodal evidence. The deliverable is proof that streams agree about time and space.
What We Capture
| Modality | Training signal | Typical rate |
|---|---|---|
| RGB video | Appearance and visual feedback | Tens of fps |
| Depth and point cloud | Metric geometry and surface structure | Tens of fps |
| Proprioception | Joint, pose and gripper state | Hundreds–thousands of Hz |
| Force and torque | Contact and insertion feedback | Hundreds–thousands of Hz |
| Tactile | Pressure, slip and deformation | Sensor-dependent |
| IMU | Acceleration and rotation | Hundreds–thousands of Hz |
| Audio | Contact and failure signatures | Audio rates |
The Rate Mismatch
| Approach | What it does | Cost |
|---|---|---|
| Nearest-neighbour | Closest high-rate sample | Can introduce half a frame of error |
| Zero-order hold | Carry last value forward | Wrong for fast continuous signals |
| Linear interpolation | Interpolate to frame time | Can erase transients |
| Aggregation | Min, max or mean in window | Loses event shape |
| Native rate | Keep the original stream | Requires a capable loader |
Synchronisation, Measured Rather Than Assumed
| Method | Realistic alignment |
|---|---|
| Software timestamping | Tens of milliseconds, drifting |
| Network time protocol | Sub-millisecond with PTP |
| Hardware triggering | Best available alignment |
How We Verify Before Delivery
| Check | Evidence |
|---|---|
| Alignment verification | Cross-stream offset measured at start and end |
| Calibration residual | Reprojection and transform residual retained |
| Gap and dropout scan | Missing samples and resets logged |
| Cross-modal consistency | Visual and proprioceptive motion compared |
| Loadability test | Pilot loaded through the client pipeline |
Where Multimodal Capture Goes Wrong
| Failure | Consequence | Control |
|---|---|---|
| Synchronisation never measured | Policy learns lag | Measure per episode |
| Interpolation undocumented | Transients disappear | Record method per stream |
| Extrinsic inverted | Spatial fusion is wrong | State and verify direction |
| Force uncompensated | Gripper weight becomes contact | Gravity and CoM calibration |
| Calibration drift | Unknown affected range | Version and validity interval |
Robotics Service Map
| Page | Deliverable |
|---|---|
| Human Demonstrations | We create the episodes |
| Multimodal Sensor Data | We capture and align the signals |
| 3D & Spatial Annotation | We label the world |
| VLA Evaluation | We score the model |
| Deployment Validation | We verify it in place |
How a Programme Runs
- Sensor scope and specification
- Rig instrumentation
- Calibration
- Alignment verification
- Pilot batch
- Volume capture
- Ongoing verification
Formats and Delivery
LeRobot, RLDS, HDF5, ROS bag, MCAP or raw synchronised streams with per-stream manifest, measured sync error, calibration package and lineage record.
Related Services
Human Demonstrations 3D Point Cloud & LiDAR Annotation Data Validation & Verification
Multimodal Sensor Data FAQs
What is multimodal sensor data in robotics?
Synchronised streams captured during a robot episode with a common time reference and known spatial relationships. Several files from roughly the same session are not a multimodal dataset.
How accurate does synchronisation need to be?
The tolerance depends on the task and is measured per episode.
What is the difference between hardware triggering and software timestamps?
Software timestamps can drift; PTP disciplines clocks; a common hardware trigger gives the best available alignment.
Why does the rate mismatch matter?
Video and high-rate force or state require a documented alignment decision that changes what the model learns.
What is force-torque gravity compensation?
It separates gripper and payload weight from genuine contact force across robot orientations.
Do we need tactile and force sensing?
Contact-rich tasks usually do; adding it later can require recollection.
What metadata comes with the data?
Identifiers, checksums, clocks, offsets, drift, gaps, calibration residuals, versions, QA and withdrawal history.
Can you audit an existing dataset?
Yes. We assess sync, calibration and metadata integrity and separate recoverable defects from required recollection.
Which formats do you deliver?
LeRobot, RLDS, HDF5, ROS bag, MCAP or raw streams with a manifest.
How is this different from human demonstration collection?
Human Demonstrations creates episodes; Multimodal Sensor Data aligns and documents the signals beside them.