Understanding data bottlenecks in Physical AI
Physical AI systems such as humanoid robots, autonomous manipulators, and embodied agents require fundamentally different data than their purely digital counterparts. A language model trains on text from the internet, which exists in virtually unlimited quantity. A vision model trains on labeled image datasets containing billions of examples. But a robot learning to assemble a circuit board needs demonstrations of human hands performing precise manipulation tasks, and this data is scarce.
This scarcity is the central data bottleneck in Physical AI. The largest publicly available robotics datasets contain roughly one million trajectories. By comparison, GPT-3 trained on 300 billion tokens. Closing this gap requires not just more data, but entirely new approaches to data collection, processing, and infrastructure.
Three interconnected bottlenecks define the current landscape:
- Data volume. Factories generate billions of hours of video annually, but converting unstructured footage into structured training data requires sophisticated processing pipelines that most organizations lack.
- Data quality. Raw video contains occlusion, variable lighting, and camera noise. Extracting clean 3D trajectories and action labels from imperfect footage is a hard computer vision problem.
- Data format. Robot learning frameworks expect specific input formats: paired visual-action sequences, 3D joint trajectories, language annotations. Most industrial video was captured for entirely different purposes.
The scale mismatch
The gap between available data and data requirements is not linear, because it widens as models grow more capable. Published work on large robot-learning corpora shows that each order of magnitude increase in training data produces qualitative improvements in generalisation. A model trained on 100 demonstrations can perform a single task reliably. A model trained on 100,000 demonstrations begins to generalise across objects, environments and task variations.
Traditional data collection via teleoperation or kinesthetic teaching simply cannot reach these volumes. A human operator can produce roughly 50-100 valid demonstrations per day in a well-equipped lab. Reaching 100,000 demonstrations would take 3-5 years with a dedicated team. This is why the industry is turning to alternative sources of training data, which explains why data infrastructure has become the critical competitive differentiator in Physical AI.
Best practices for structured data creation
Building effective data infrastructure for Physical AI requires more than just capturing video. The pipeline must convert raw observations into structured representations that robot learning algorithms can consume. Based on our experience building production data pipelines for humanoid robot training, here are the essential best practices.
Action segmentation at the right granularity
Complex factory tasks decompose into hierarchical action primitives: reach, grasp, transport, align, insert, release. Your pipeline must segment continuous video at both coarse (task-level) and fine (primitive-level) granularities. Sub-frame temporal accuracy is critical because misaligned action boundaries introduce noise that propagates through the entire training pipeline. Validation against human-labeled ground truth should target < 0.5 second boundary error.
3D kinematics from standard 2D video
Depth sensors are rarely available in existing factory installations. Your pipeline must estimate 3D motion from monocular video using a combination of learned depth estimation, multi-frame motion analysis, and workspace geometry. Target accuracy: < 2cm position error for end-effector trajectories. This is achievable with modern pose estimation models fine-tuned on domain-specific industrial imagery.
Dataset diversity across workers and conditions
A common failure mode is training on demonstrations from a single worker or shift. Human workers vary in technique, speed, and anthropometry. Your dataset must capture this variation to produce robust policies. Collect demonstrations across multiple workers, shifts, lighting conditions, and product variants. Monitor diversity metrics (action duration variance, trajectory dispersion, object interaction frequency) as part of your data quality dashboard.
Multi-modal annotation for VLA compatibility
Vision-Language-Action models require synchronized visual observations, action sequences, and language annotations. Each demonstration needs a natural language task description, object identities, and spatial relationships. While automated annotation handles the bulk of labeling, maintain a human-in-the-loop review process for edge cases: ambiguous actions, partially occluded objects, and rare failure modes.
Quality metrics and continuous validation
Treat your data pipeline as a production system with monitoring. Define and track key quality metrics: action segmentation accuracy, 3D pose estimation error, annotation consistency across workers, and dataset diversity indices. Run periodic validation by training a small policy on a sample of your dataset and measuring held-out demonstration performance. Degradation in these metrics triggers pipeline review.
Patterns: data infrastructure in practice
Teams building Physical AI systems keep running into the same three shapes of problem. Recognising which one you are in tells you what your data infrastructure actually has to do.
Precision assembly
- Situation. A team needs large volumes of demonstrations of precision assembly work: fastener insertion, cable routing, sealant application. Teleoperation can produce clean examples, but not at the volume or variety the policy needs.
- What the infrastructure has to provide. Action boundaries accurate enough that a segment is a usable training example on its own, labels that survive a change of plant, and enough variety inside one task that the policy is not learning a single line habits.
- What changes. The bottleneck moves from collecting demonstrations to deciding which skills to target next, because breadth inside a task is what drives generalisation.
Unstructured handling
- Situation. A logistics operator wants to automate picking and sorting, but the objects change weekly and no two totes look alike. Staged demonstrations cannot cover that distribution.
- What the infrastructure has to provide. Data captured against the real object mix, with grasp and contact events labelled, so the policy sees uncertainty and recovery rather than only clean successes.
- What changes. Generalisation to unseen products becomes something you can measure instead of something you hope for, and new stock folds into the next collection round.
Archived footage
- Situation. A manufacturer already holds hundreds of hours of inspection and assembly footage, recorded for entirely different reasons, and has no route from that archive to a training set.
- What the infrastructure has to provide. A path from unstructured archive to structured dataset that does not require re-recording the work, plus quality scoring so the unusable parts of the archive are filtered rather than shipped.
- What changes. Footage that was a storage cost becomes a training asset, and the cost of trying a new task drops to the cost of asking a question about data you already own.
Tools and platforms for Physical AI data
The tools you choose shape the scale, quality and velocity of your Physical AI data work. What follows surveys the open ecosystem you can build with, alongside what Khenda delivers. It is a map of the field, not a description of anyone's internal stack.
| Approach | Scale | Fidelity | Consent and provenance | Getting started |
|---|---|---|---|---|
| Production video, structured | A plant network, plus archives you already hold | Real-world native | Explicit operator consent and documented provenance | Footage you hold, or a task brief |
| Teleoperation | Bounded by operators and rigs | Partial | Operator agreements, varies by provider | Build the rig, recruit and train operators |
| Simulation | Bounded by compute and authoring | Low, with a sim-to-real gap | Not applicable, no human subjects | Author scenes, assets and physics |
| Manual annotation | Bounded by annotator hours | Only as good as the source footage | Inherited from whoever recorded the footage | Recruit annotators, write guidelines |
Khenda
- Task-directed manipulation data from live production plants
- Or structuring of operator footage you already hold
- Action segments, labels, masks, contact events, trajectories
- Delivered in HDF5, LeRobot or RLDS, or your own schema
- SOC 2 Type II, ISO 27001, GDPR and KVKK compatible
Open-source ecosystem
- LeRobot: community dataset repository
- Open X-Embodiment: cross-platform benchmarks
- MuJoCo / Isaac Sim: simulation augmentation
- ROS 2 / ROSbag: standard robotics I/O
- Weights & Biases / MLflow: experiment tracking
Computer vision stack
- MediaPipe / OpenPose: hand and body pose
- DETR / SAM: object detection and segmentation
- FoundationPose: object pose estimation
- AnyGrasp / Contact-GraspNet: grasp synthesis
- OpenCV / FFmpeg: video processing pipeline
Training frameworks
- RT-2 / RT-X: published VLA model architectures
- OpenVLA: open-source VLA training framework
- Octo / Diffusion Policy: action sequence modeling
- Implicit / explicit BC: behavior cloning backends
- RLlib / Stable-Baselines3: reinforcement learning
Frequently asked questions
What makes data infrastructure for Physical AI different from traditional ML data pipelines?
Physical AI data requires capturing 3D spatial information, temporal sequences of physical actions, and contact dynamics rather than just text or images. The infrastructure must handle video ingestion, action segmentation, 3D pose estimation, and output structured datasets compatible with robot learning frameworks.
Can I use existing factory camera footage as training data?
Often, yes. The practical requirements are adequate resolution (720p or better), stable framing of the work area, and enough duration to cover complete task sequences. Variation in lighting, camera angle and occlusion is normal in real footage and has to be handled rather than avoided.
How much data do I need to train a robot policy?
Requirements vary by task complexity. Simple pick-and-place policies may need hundreds of demonstrations, while complex assembly tasks benefit from thousands. The relationship between data volume and policy performance follows a power law: each 10x increase in data yields measurable improvements in generalization and robustness.
What format should training data be in for VLA models?
Most VLA models expect paired visual observations and action sequences, with optional language annotations. Common formats are RLDS, HDF5 and LeRobot. Khenda delivers in HDF5, LeRobot or RLDS, or in a schema your team defines.
How do I validate whether my data infrastructure is producing high-quality training data?
Key metrics include action segmentation accuracy, 3D pose estimation error (target < 2cm), dataset diversity across workers and conditions, and downstream policy performance. Regular validation against held-out demonstrations helps maintain data quality at scale.