Onticlabs · data pipelines

Data work

We make point clouds from real hand-object datasets and labelled synthetic scenes from GraspXL grasps. The data serves world models, 3D tracking and robot policies.

Overview

Two kinds of data, three uses

dataset_procThe CLI. It has one plugin per dataset, one canonical point-cloud NPZ, Blender renders, releases with cards and one PyTorch loader.

Real data

Captured hand-object datasets. The CLI converts them into the canonical format.

HOT3D, ARCTIC: releases with 2.5 M image frames
ACE, TACO, GigaHands, BEHAVE, EgoSuite: samples and checks

Synthetic data

Labelled pick-and-place scenes. The generator makes them from GraspXL grasps.

Table, targets, simulation, text label, forces
SMPL-X body, iTHOR rooms, cluster pilot (98 scenes)

World model trainingOne loader serves point clouds, renders and text labels.

3D trackingScenes with a body, clutter and a room. Each camera gives RGB, depth and segmentation.

Robot policy trainingGraspXL policy grasps exist today. The exports hold hand joints, object poses and labels.

Part 1

dataset_proc

The CLI converts real hand-object datasets into one canonical point-cloud format.

dataset_proc · architecture

One CLI, one plugin per dataset, one NPZ schema

IngestLean downloads: manifests, Hugging Face, HTTP-range zip members. The CLI never changes the raw data.

→

Scene builderEach dataset has one build_scene(). It gives rigid actors and hand actors in the dataset's own axes.

→

CleanThe CLI infers support surfaces or pads. It cuts the bad frames and keeps the segments.

→

Canonical NPZ 1.1The CLI samples surfaces at 4 mm and 15 fps. It stores each rigid actor once, with (F,7) poses. Each hand has 21 joints.

→

RenderBlender Cycles renders 4 cameras, fitted to the interaction, as 512x256 RGBA webp.

→

Release + loaderA release has h5 frames and a card. PointClouds(...) serves points, joints, images, K/R/t and text.

The same scene builder feeds the NPZ export and the on-the-fly PyTorch loader.
Points keep their identity across frames. Each point has a normal and a colour.
A cluster recipe renders on HTCondor.

ARCTIC frame sampled with seed 0 and seed 1, with the left hand on the capsule machine

ARCTIC: the same frame, sampled with seed 0 and seed 1 at 4 mm. Both seeds cover the same hand and object surfaces with different points.

dataset_proc · HOT3D

HOT3D: ground truth, UmeTrack hands and support surfaces

We use the ground truth, the hands and the object assets.
We fixed the axes: Aria is Z-up, Quest is Y-up.
The CLI infers the support surfaces under the objects.
The hands come from UmeTrack, with 21 joints per hand.

294sequences

33object meshes

Sampled points (left) and the source meshes (right) for 5 sequences.

dataset_proc · HOT3D

Hands passed through the inferred surfaces, so we cut the bad frames

We refined the surfaces per sequence.
The CLI cuts the frames with a hand inside a surface. Each sequence becomes 1 to 10 segments. 206 of 294 sequences stay whole.

96.3 %frames kept (497,675 of 517,004)

552segment NPZs from 294 sequences

9.2 hof motion kept

The worst-hit sequence (P0010) after the cut.

dataset_proc · HOT3D

The cluster renders four cameras fitted to every interaction

Blender Cycles writes 512x256 RGBA webp frames and an OpenCV cameras.json.
The cluster rendered 214 sequences.

25 minon the cluster

21.5 hestimate without the cluster

2.07 Mreleased image frames, 14.6 GB webp

dataset_proc · HOT3D

We sampled every data type, rectified the fisheye video and verified the calibration

Ground truth projected on the RGB camera (median overlay). The epipolar test passed.

The Aria fisheye RGB, rectified to a pinhole camera.

dataset_proc · ARCTIC

ARCTIC: MANO hands and articulated objects on a 10 cm pad

We export the hands only. A full-body export would be about 610 GB. Hands, object and pad are about 2.6 GB.
Object posing reproduces the authors' matrices to 6e-5 mm.
Each frame has two object parts and two MANO hands.

301sequences

436 kreleased image frames, 1.7 GB webp

Top: datasheet example. Bottom: the 10x10 cm support pads.

dataset_proc · loader

One PyTorch loader over every release

PointClouds(...) serves points, joints, images, K/R/t, episode tables and text.
The clip id selects the points, so a clip always returns the same points.
A full sweep read 853 NPZ files and 606,587 frames with 0 problems.

540 /sARCTIC items without images

130 /sARCTIC items with images

A loader item next to its NPZ, with the point trajectories over time.

dataset_proc · releases

Each release ships with a card. HOT3D also has a sample CSV and a datasheet

Release	Seq	Image frames	Size
point-clouds-hot3d	294 (552 NPZ)	2,068,016	30 GB
point-clouds-arctic	301	435,648	4.5 GB
mesh2gs-16k	19,619 obj	—	9.0 GB
mesh2gs-05k-refit	19,619 obj	—	0.6 GB

HOT3D datasheet: the render strip over the projected points, and how often each object appears.

dataset_proc · not converted yet

Real datasets we downloaded, not yet in the canonical format

ACE-Data-0

WhatFull body (SMPL-X) and rigid objects, 453 takes.

We haveDownload, contact and glitch checks, 452 render clips, a ranking.

IssueOdd body poses, object interpenetration, missing contact.

TACO

WhatTwo hands using tools, 2,317 sequences. CC BY 4.0.

We haveGround truth of all sequences (26 GB), video for 5, a hand overlay.

GigaHands

WhatTwo-hand activities, 51-camera capture.

We haveThe 5-sequence demo (779 MB), a verified hand overlay.

BEHAVE

WhatFull body (SMPL-H) and large objects.

We have299 sequences (795 MB), contact sheets.

Humoto

WhatFull body and rigid objects. The full set needs approval.

We haveA 70-sequence sample, SMPL-X fit tests, Blender renders.

NymeriaPlus

WhatEgocentric Aria capture with a body (MHR, SMPL), object boxes and meshes.

We have3 sequences: body, SLAM and narration (12 GB).

No visual yet

EgoSuite

WhatEgocentric video with 3D body and hand joints.

We havePoses only, 50 episodes (382 MB).

No visual yet

RHINO

WhatIn progress.

We haveNo local data yet.

dataset_proc · candidates

Candidate datasets and generators, not used yet

Hand-object capture

OakInk2Two hands, 4.01 M frames, 75 objects, some articulated.

GRABMarker capture of whole-body grasps, 51 objects. No RGB video.

HIMONot surveyed yet.

EPIC-contactThe data quality needs to improve before we can use it.

PAMINot surveyed yet.

SchleppNot available yet.

Egocentric and sensors

Eidon AI tracker-povEgocentric video with hand, arm and shoulder IMU sensors.

Commercial

BonesA commercial dataset, probably 100 k to 1 M. For a later phase that needs higher quality and commercial use.

ProtegeCommercial data, maybe only a capture service. Probably 100 k to 1 M.

Synthetic and point tracking

Syn4DVideo with 3D point tracks. 1 of 29 scenes downloaded.

IGGT4DVideo and point clouds of Infinigen interiors. The dataset is not released.

EmbodiedGenGenerates environments and objects, with labels for where to grasp. Sample assets downloaded.

SuperDexPhysics simulator. We made scripted grasp scenes as NPZ. Data and Quest 3 / Aria teleoperation in Q4 2026.

Part 2

GraspXL synthetic scenes

The generator makes labelled pick-and-place scenes from released grasps.

GraspXL scenes · pipeline

From one released grasp to a labelled scene

Source graspThe source is one GraspXL tabletop sequence (100 frames) of an LVIS-labelled Objaverse object.

→

ApproachThe generator adds 0.5 to 1.0 s before the grasp, along the approach direction.

→

SceneThe generator draws a random table and texture, free objects, raised surfaces, a relation and a target.

→

Carry, releaseThe hand lifts the object, carries it with a wrist turn and releases it. MuJoCo simulates the object after the release.

→

Checks, render, forcesA failed check drops the scene. Each kept scene gets an EEVEE video and force estimates.

The label is one sentence, for example "drop the strawberry onto the orange ball".
There are five relations: next to, drop on, place on, place on surface, drop on surface.
The object pool has 15,813 objects in 1,018 LVIS categories. Each object has a material class and a mass estimate.

A review clip with the phase labels.

GraspXL scenes · samples

Six of the 50 public samples

pages.onticlabs.io/graspxl-synthetic-data-generation640x360 copies, EEVEE, tracking camera

GraspXL scenes · batch

Twenty scenes of one batch, with rotated target colours

Target colours: free targets rotate through red, green, yellow, purple, orange and white. Surfaces rotate through wooden, white, gray and black.

GraspXL scenes · quality gates

Two gates: one for the source grasp, one for the finished scene

Source gate, before the scene is built

Grasp: the thumb opposes a finger or the palm (opposition angle of 115 deg or more).
Joins: the added motion joins the released motion without a speed or angle jump.
Approach: the hand stays 1 cm above the table and touches the object no earlier than in the source.

Scene checks, after the simulation. All six must pass

Label: the outcome matches the label. The generator redraws the scene up to 80 times until it does.
Release: the hand lets go without a speed or angle jump.
Hand: the hand never goes more than 5 mm under the table top.
Penetration: the release adds at most 3 mm of hand-object penetration.
Rest: the object comes to rest.
Settle: a placed object turns 10 deg or less. Drops are exempt.

Glued holds stay, with a flag. In a glued hold the contacts cannot carry the weight.

56 %sources kept in the first batches (56 of 100)

49 %sources kept in the cluster pilot (98 of 200)

Review page: six phases per scene. We checked the label and the target colour by eye.

GraspXL scenes · scale-up

One cluster job runs the whole chain for 20 sources

Pilot	Sources	Gate	Scenes	Kept	Glued
10 jobs	200	164	158	98	20

Each job took 16 to 24 min on one GPU. Nine jobs in parallel took 20 min.
Kept scenes per relation: drop on 30, drop on surface 25, place on 18, place on surface 15, next to 10.

A review page of the cluster pilot, rendered with EEVEE.

GraspXL scenes · forces

Contact forces and joint torques per frame

Contact force arrows on the hand during the hold.

One scene: finger contact forces, twists and joint torques over the clip.

The check flags a hold as glued when the measured contacts cannot carry the object mass for more than 10 frames.

GraspXL scenes · full body

An SMPL-X body fitted to every clip with the FUSION motion prior

FUSION runs diffusion noise optimisation on 120-frame windows. The MANO hand stays exactly as generated.
The loss has table, floor and object penetration terms. IK locks the wrist exactly.
Review of 20 clips: 5 good, 15 minor, 0 bad. One clip takes 4 to 7 min.
Objaverse policy set: 63 clips with a body. The newest 44: 12 good, 28 minor, 4 bad.

20 clips with the fitted body.

GraspXL scenes · rooms

We place each clip in a MolmoSpaces iTHOR room

20 clips in 20 rooms: 8 kitchens, 7 living rooms, 3 bedrooms, 2 bathrooms. The room is a rendered backdrop only.
Simulation, labels and checks stay the same.
A contact check passes all 20 clips. The hand and the objects stay more than 10 cm from every room surface.

iTHOR FloorPlan1 backdrop.

20 clips in iTHOR kitchens, living rooms, bedrooms and bathrooms.

GraspXL scenes · earlier sets

Before the scenes: longer samples and pick-and-place on a plain table

Long samples v2: 93 curated clips with a kinematic extension of the released motion.

Pick-and-place: 88 curated clips, with an automatic grasp check and a tuned release speed.

GraspXL · the grasp policy

The GraspXL RaiSim policy on 64 Objaverse objects

We ran the pretrained policy on 64 new Objaverse objects with URDF packages.
Finding: the pinky splays in all policy output. Fix: the pinky follows the ring finger.
An automatic grasp check and a finger-finger intersection exclusion give 45 curated sequences.

This is the policy side of the data. A learned grasp policy is the source of the motions. The exports carry its hand joints, object poses and labels.

Top: 64 policy grasps. Bottom: pinky before and after the fix.

Part 3

Applications

World model training, 3D tracking, robot policy training.

Applications · World model training

Points, renders and the instruction through one loader

GraspXL scenes are a dataset_proc source. A scene is a scene.json plus a scene.npz. Every loader item carries the text label.
The default output has the hands only (about 2 MB per scene). --human body adds the SMPL-X body (46 to 50 MB).
The 4-camera rig keeps 0.25 m clear of a body. Other exports stay byte-identical (tested on ARCTIC).
The orbit camera turns at 20 deg/s. The generator draws the direction per scene.

Top: canonical NPZ point cloud with the two hands. Bottom: the four fitted cameras, five phases, hands only.

Applications · 3D tracking

A textured body, clutter and a room, with RGB, depth and segmentation

The tracking output adds an SMPL-X body, PartNet clutter (76 objects) and a MolmoSpaces room (120 rooms, referenced).
Camera setups: orbit (20 deg/s, random direction), dolly, pan, tracking, egocentric, and an 8-camera multiview.
New: BEDLAM skin and clothing textures, drawn per scene. Depth and segmentation do not change.
EEVEE renders 640x360 at 15 fps. We tested it on 2 scenes so far.

Orbit camera: RGB, depth (1.1 to 6.8 m), class segmentation (static, dynamic, human).

One scene through every camera setup, plus the world-model rig without a room.

Applications · 3D tracking

A second scene: "place the motorcycle on top of the yellow box"

Every camera of one scene: 13 tracking views (orbit, dolly, pan, tracking, egocentric, 8 static multiview) and the 4 world-model views, hands only. A living room, another BEDLAM skin and outfit, and an orbit that turns the other way.

Applications · 3D tracking

How one tracking scene is built, layer by layer

Twelve steps, one scene: source grasp, extended motion, table, texture, target and label, clutter, fitted body, skin, clothing, room, camera setups, outputs. Steps 1 to 10 come from the tracking renderer itself.

Applications · Robot policy training

What the data gives a policy learner

Each frame has 21 hand joints per hand, the object 6-DoF and a text instruction. Contact forces and joint torques are estimates in a separate file.
Kept scenes carry a verified outcome: the label matches the simulation.
The GraspXL RaiSim policy already produced 64 grasps on new objects.
The grasp forces are still work in progress.

Labelled grasp-to-drop phases, four scenes.

Status

Where it stands and what comes next

Done

HOT3D and ARCTIC releases with cards. We verified the loader on 853 files.
GraspXL scene generator, cluster pilot, forces, body fit, rooms. Public samples page.
GraspXL as a dataset_proc source with world-model and tracking outputs.

In progress

2,000 synthetic scenes are rendering now, for 1 to 1.5 M frames in total.

Next iterations

Real data

Add more datasets.
Improve the object contacts and/or the video correspondence with optimization.

Synthetic data

Add more object variety: Kubric, Objaverse.
Make the human walk and move objects from one surface to another.
Add failure cases, for example failed grasps.
Implement object handover from one hand to the other.
Add different types of grippers.
Enhance realism with diffusion, if necessary.