# Data > Publish and download robotics datasets with the Qualia SDK The `data` resource publishes robot data to Qualia and downloads exact published trees. Uploaded bytes go straight to storage; Qualia verifies the manifests and publishes them together with the data, never handling the bulk bytes itself. ## Install ```bash pip install 'qualia-sdk==0.8.0rc14' # or with uv uv pip install 'qualia-sdk==0.8.0rc14' ``` This deployment runs ahead of the public release, so it needs the matching pre-release wheel — `0.8.0rc14`. Pin it rather than upgrading: the upload protocol is negotiated exactly, so the client and the endpoint it talks to have to be the same generation. A plain `pip install qualia-sdk` resolves to the current public release and will be turned away by this deployment. The native transfer engine that powers upload/download is bundled in the wheel — there is no separate step. ## Upload a dataset Stage a local dataset directory and atomically publish its validated source, Git history, and lakehouse rows in one call. Episodes and frames become visible only after the paired publication succeeds. ```python from qualia import Qualia client = Qualia() result = client.data.upload("/data/gongsta", name="gongsta") print(result["dataset_id"], result["episode_count"], result["total_frames"]) ``` **Response** ```json { "repo_path": "acct-01234567-89ab-4cde-8fab-0123456789ab/gongsta", "name": "gongsta", "file_count": 128, "committed_files": 128, "dataset_id": "…", "dataset_version_id": "…", "episode_count": 42, "total_frames": 51840 } ``` ### Parameters | Parameter | Default | Description | | ---------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `local_dir` | — | Directory of dataset files (or an MCAP session) to upload. | | `name` | — | Dataset name within your account (a single path segment). | | `source_format` | `None` | One of `lerobot_v2_1`, `lerobot_v3`, or `mcap_ros2`. `None` reads it from the directory: `codebase_version` in `meta/info.json` for LeRobot, any `.mcap` for a session. | | `max_workers` | `None` | Files read and chunked at once; `None` means 4. Bounds CPU, RAM and disk so a large upload can't saturate your machine. | | `num_threads` | `None` | Native chunk/hash thread-pool size; `None` means about half your cores. | | `quiet` | `None` | `True` silences the live per-phase progress on stderr. `QUALIA_QUIET=1` does the same for a whole process. | | `on_progress` | `None` | Called with one dict per progress state (`source`, `message`, `elapsed_s`, and more), so you can render progress yourself. Read it tolerantly: new keys ship without an SDK release. | | `allow_underivable` | `False` | Upload an MCAP session even when no bag in it can be derived. It cannot lift a LeRobot structure refusal. | | `raise_on_underivable` | `False` | Raise `UploadUnderivableError` (carrying the receipt) when no episode could be derived, instead of returning the receipt with `ingest_verdict="underivable"`. | | `merge_target` | `None` | Existing merged dataset to append this upload to, as a full resource name (`accounts/{account}/datasets/{dataset}`); a bare id is refused. `None` uses the destination saved by `qualia dataset destination set`, but only under the API host, account and workspace it was saved for. | | `standalone` | `False` | Keep this upload its own dataset even when a saved destination matches. Cannot be combined with `merge_target`. | | `rig` | `None` | UMI rig to convert a handheld (UMI) recording at as it lands: its resource name, id or display name (or the old name `"umi-handheld"`, the rig that holds that key, else the workspace's one UMI rig). The workspace must have it: make one from a gripper template first. A recording whose `meta/ingest.toml` declares `[rig] kind = "umi-handheld"` and names none takes the workspace's one UMI rig; none or several refuses it. See [Handheld (UMI) recordings](https://junction-docs.qualiastudios.dev/global/guides/data-collection.md#handheld-umi-recordings). | Re-uploading a dataset only sends the chunks that changed — unchanged files are skipped client-side, and edited files upload just their changed regions. ### Chunk reuse across client restarts Until now an ordinary `upload()` kept no record of how the previous version of a file was cut into chunks, so an edited file went through the ordinary cold upload path every time. `upload()` now keeps a small on-disk cache of each file's chunk boundaries (chunk IDs plus offsets and sizes; never bytes, credentials, URLs or manifest IDs) that survives client restarts. On the next upload of a changed file the SDK re-chunks the current file as usual, compares the unchanged prefix and suffix against the cached boundaries, and asks the server — under your current authorization — whether those chunks still exist. Only chunks the server confirms are kept; everything else uploads normally. A missing, damaged or unreadable cache is silently treated as a miss. What this saves is bytes stored, not time. On the harness's 36.3 MB changed-video fixture, re-uploading the edited file after a client restart stored 2,922,261 bytes without the cache and 26,741 bytes with it. No upload-time speedup is claimed: the file is still read and chunked in full. Uploading a directory writes the cache once at the end, however many files it holds, so the database work does not grow with the file count. The work of preparing each eligible file's hint does grow, though it stays small: on one development machine, offering 1,009 multi-chunk files took about 97 ms and 10,009 took about 301 ms. The cache applies to changed files of at most 64 MiB in ordinary directory uploads. See [Configuration](https://junction-docs.qualiastudios.dev/sdk/configuration.md#upload-boundary-cache) for where it lives, how to disable it, and how it is bounded. ## Import an exact Hugging Face dataset The SDK sends only an exact lowercase 40-hex commit and optional token to the service. The import runs entirely server-side, streaming the pinned Hugging Face tree straight into storage; dataset bodies and scratch files never touch the SDK or any Qualia service: ```python from qualia import Qualia client = Qualia() repo = "qualia-robotics/openarm-cube-in-box-v1" revision = "17c29c487d886878ddcb3c5649101b2860fbc047" result = client.data.import_huggingface( repo, revision, "cube-in-box", source_format="lerobot_v3", ) print(result["dataset_id"], result["dataset_version_id"]) ``` A private or gated repository is read with the workspace's Hugging Face key from Settings → Integrations (or [`client.integrations`](https://junction-docs.qualiastudios.dev/sdk/resources/integrations.md)). To use another key for one import, pass `hf_token=os.environ["HF_TOKEN"]`. With no key that can read the repository, the import fails and says which key to add or give access. The token appears only in the initial submission and is never returned in poll results or progress events. The worker authenticates the exact remote file identities and publishes an immutable DatasetVersion. An import always creates a new dataset. The service checks the name before anything runs: - If a dataset already has the name, the import fails at once with a 409 (`dataset_name_taken`). Pick another name. - If another import into the name is still running, the import fails with a 409 (`dataset_name_importing`), unless it is the same import: same repository, revision, format and name. Then your call joins the running import and returns its receipt, so a second start never imports twice. - Calling again with the same `idempotency_key` rejoins the import while it runs, and returns its final receipt once it has finished. If the earlier wait timed out, this is how you pick the import up again. Reusing a key for different coordinates fails with a 409 (`idempotency_key_reused`). Qualia intentionally rejects dot-prefixed Git-control paths. The importer permits only the exact root names `.gitattributes` and `.gitignore`: it downloads and verifies them, commits their path, size, Git blob OID, and content identity under `excluded_repository_metadata`, and removes them before publication. Any other hidden path, `.lock` path, or `meta/qds/**` attempt fails preflight. The provenance sidecar itself is published at the non-authoritative, `meta/qualia/imports/huggingface-dataset.json` path. The pinned medium validation fixture `TobiBrtnr/openarm_shirt_folding_new_converted@c34057e9cb7760f939dbd5759679c12818fa896b` is LeRobot 2.1 and about 2.08 GB. Its pinned media inventory reports AV1 even though legacy metadata may label it `avc1`; Qualia treats validated media bytes as authoritative. The public metadata does not establish collection hardware, so do not describe it as real-robot data without separate provenance evidence. ## Upload a robot MCAP session Recorded robot sessions upload through the same call with `source_format="mcap_ros2"`. Lay the session out as one subdirectory per recording, each holding its `.mcap` bags. Every `.mcap` is ingested on its own, so rosbag2 split files (`bag_0.mcap`, `bag_1.mcap`, …) are accepted; a take that straddles a split becomes two episodes. An `episode.json` sidecar may provide normalized `task` and `mode` fallbacks when the bag does not. An optional `meta/ingest.toml` at the session root declares state and action topics, the clock and episode boundaries; see [Declaring state and action channels](https://junction-docs.qualiastudios.dev/global/guides/data-collection.md#declaring-state-and-action-channels): ``` session-2026-07-20/ meta/ ingest.toml # optional episode-000/ episode-000.mcap episode.json # { "task": "fold the towel", "mode": "teleop" } episode-001/ episode-001_0.mcap # rosbag2 split files episode-001_1.mcap episode.json ``` ```python result = client.data.upload( "/data/session-2026-07-20", name="dk1-kitchen", source_format="mcap_ros2", ) print(result["dataset_id"], result["episode_count"], result["total_frames"]) ``` A handheld (UMI) recording uploads the same way with `rig=` naming the UMI rig it was recorded on. A workspace has the UMI rigs someone made and none otherwise: nothing makes one for you, so make one first from a gripper template (New rig, UMI in the dashboard, `qualia gripper umi-rig`, or `client.robots.make_umi_rig`), calibrate it, then name it. The server writes the rig's kind and TCP calibration into the upload's `meta/ingest.toml`, so the pair converts to gripper poses as it lands; your local file is never written. Naming a rig the workspace does not have is refused before anything is stored (`no UMI rig in this workspace`, with how to make one). A recording that says it is a UMI one in its bytes (a `[rig]` kind in its `meta/ingest.toml`, a LeRobot `robot_type`, a sidecar `rig_id`) is checked at ingest: with no UMI rig in the workspace the upload ends failed with `no UMI rig in this workspace` and nothing is published, and it is not retried. So is one from which a complete hand pair settles by its channel names. A recording whose channel names only loosely look UMI-like (per-side x/y/z positions and no arm joint) and from which no complete pair settles (for example jawless, one hand, `infer_handheld = false`, or positions without rotations) is not refused: with no UMI rig in the workspace it publishes raw, as recorded and with no UMI conversion, with a `handheld_conversion_skipped` finding saying so. If it is a UMI recording, record both hands' position, rotation and jaw, make a rig and upload again naming it. With several UMI rigs either kind of recording asks which one: pass `rig=` so the upload names it. Only an upload that named a rig (or the one rig a merged dataset lends) counts as having one; a `[rig]` written into your own `meta/ingest.toml` does not name a rig, with its TCPs or without, and is refused the same way: ```python # once per rig: copy a gripper template, then import its calibration rig = client.robots.make_umi_rig("KIWI pair", "Kitchen UMI") client.robots.import_gripper_calibration(rig.name, "tcp_calibration.json") result = client.data.upload( "/data/umi-session", name="umi-kitchen", source_format="mcap_ros2", rig="Kitchen UMI", ) ``` A rig is named by its name or id; `"umi-handheld"` is the rig that holds that key (a workspace's first one, from before a workspace could have several), else the workspace's one UMI rig, so an upload that already names it keeps working. A recording that declares the UMI kind in its `meta/ingest.toml` and names no rig takes the workspace's one UMI rig the same way: with none it is refused ("no UMI rig in this workspace"), with several it asks which, both before a byte moves. `client.robots.umi_rigs()` lists the workspace's rigs. Then pick its robot and resolve its poses at the robot's TCP; see [Set up a handheld (UMI) dataset](#set-up-a-handheld-umi-dataset). The upload checks every bag before a byte moves. A damaged or empty bag is listed and uploads as recorded; the rest of the session publishes. Only a session with no `.mcap` at all, or with every bag underivable, raises `UploadBlockedError` and uploads nothing; `allow_underivable=True` uploads it anyway. A missing sidecar is valid; Qualia derives task and recording purpose from authenticated bag contents when available. ## Read the ingest report Ingest degrades instead of failing: a bag missing a camera drops that camera from the whole dataset, and an episode that never produced video for a camera publishes without it. Every such decision is in the source's ingest report, which is the answer to "which cameras does this dataset actually have": ```python report = client.data.ingest_report(dataset.path) # acct-/ for camera in report.camera_summary(): print(camera.camera, camera.recorded_sources, camera.dropped, camera.detail) for finding in report.findings: if finding.lost: print(finding.code, finding.camera, finding.message) ``` `report.sources` holds one entry per ingested bag with its topics, cameras and findings. The model keeps unknown fields, because the ingest adds findings without an SDK release. A dataset without a report (an imported LeRobot tree) raises `QualiaAPIError` with status 404. [`client.datasets.ingest_report`](https://junction-docs.qualiastudios.dev/sdk/resources/datasets.md#read-the-ingest-report) reads the same report by dataset id. ## Download a dataset Download a dataset's version into a folder. Name the dataset alone for its newest version, or add a version after a colon. `format="original"` downloads every file uploaded to the dataset instead, byte for byte: ```python result = client.data.download("my-dataset") # newest version, into ./my-dataset- result = client.data.download("my-dataset:1.0", "/data/robot-run", format="mcap_ros2") result = client.data.download("my-dataset", format="original") # ./my-dataset-original print(result.destination, result.file_count, result.byte_count) ``` The dataset is its slug, its id or its full name. The version is a tag, `stable` (the newest tag), `v0` (an upload as it arrived), a DatasetVersion id such as the upload's `dataset_version_id`, or a prefix of a digest or id. With no version named, a dataset with no tag yet gives its newest, `v0` for a fresh upload, and `on_progress` gets `{"step": "newest", "version": "my-dataset:v0"}`. The head, what a dataset yields right now, has no download: [tag it](https://junction-docs.qualiastudios.dev/sdk/resources/versions.md#tag-a-version) first. A dataset with no version at all says so. | Parameter | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------- | | `dataset` | `""` or `":"` | | `dest` | New folder; must not exist. Default `-` here, plus `-v21`, `-mcap`, `-pi` or `-dreamlabs` for those formats | | `format` | `lerobot_v3` (default), `lerobot_v2_1` (for loaders that predate v3), `mcap_ros2`, `pi_mcap_v1`, `dreamlabs_v2_1` or `original` | | `on_progress` | Called with progress steps (below) | | `timeout` | Seconds to wait for the server to build the version (default 30 minutes) | | `parallel` | Files fetched at once (default 4) | `pi_mcap_v1` and `dreamlabs_v2_1` are files to upload to Physical Intelligence or Dream Labs with your own account; see [`client.versions.download`](https://junction-docs.qualiastudios.dev/sdk/resources/versions.md#download-a-version). `format="original"` is every file uploaded to the dataset, byte for byte. It also holds what ingest stored beside those files: the annotations under `meta/qds`, `meta/ingest_report.json` and, for an MCAP upload, the LeRobot data and videos ingest made from the recordings. Trims and splits are not applied: the takes are whole, and a split's pieces are only annotations under `meta/qds`. When the dataset has any, the call warns with `UncuratedDownloadWarning`, saying how many and how to download the curated episodes (tag a version, then download it). It takes no version, and only an upload has files of its own: a merged dataset or mixture is refused. The folder defaults to `-original`, and the call returns an `OriginalDownload` (`destination`, `dataset`, `file_count`, `byte_count`, `downloaded_file_count`, `reused_file_count`). For now it goes over the older `/v1/datasets/download` routes, which answer organization keys only. Uploads accept and flag video rates that differ from the state and action row rate, but downloading a version and fine-tuning require re-exporting the video at the row rate and uploading again. Progress arrives as dictionaries, one at a time: ```python def show(event): if event["step"] == "preparing": print("The server is building", event["version"], event["state"]) elif event["step"] == "listing": # format="original" only print("Listing the files") else: print(f'{event["files_done"]}/{event["files_total"]} files') client.data.download("my-dataset:1.0", on_progress=show) ``` The first time a version is downloaded in a format, the server builds it, and that needs a key that can write to the dataset (`data:write`). After that, any key that can read the dataset downloads it. Bytes come straight from object storage. The SDK checks every file against its sha256 in a private folder next to `dest` (`..qualia-version-partial`) and creates `dest` only when all files match, so a failed download never leaves a partial `dest`. If a download stops, run the same call again: the files already checked are kept. The call returns a `VersionDownload` (see [Download a version](https://junction-docs.qualiastudios.dev/sdk/resources/versions.md#download-a-version)). `format="original"` checks every file against its digest the same way, staged in `..qualia-download-partial`; a large file's plan is asked for in pages. If it stops, that folder is kept and the SDK prints where it is; run the same call again to resume, a partly downloaded large file included. To discard the progress instead, delete that folder and the `..qualia-download-state` folder beside it. A version that contains a blocked episode (a NaN, an infinity, a missing value or a wrong width in its state or action rows) is refused, because a download carries every episode. Publish a curated version without it first, with `apply_curation(evidence, idempotency_key=..., name="")`, and download that. If every episode is blocked, fix the values at the source. `format="original"` is not refused: it renders nothing, so it downloads the upload's files, blocked episodes included. ## Explore the episode lake Filter episodes across every source dataset in the authenticated account with one bounded query. The result includes one offset page, totals over the complete filtered set, and task, tier, and embodiment facets. Each facet axis excludes its own filter while retaining every other filter. ```python from qualia import LakeExploreQuery, LakeNumericPredicate query = LakeExploreQuery( tasks=["fold shirt"], tiers=["retrainable"], tags=["reviewed"], numeric_predicates=[ LakeNumericPredicate(field="jerk_mean", op="lt", value=0.4) ], limit=50, ) page = client.data.explore(query) print(page.total, page.total_frames, page.facets.by_task) for episode in page.episodes: print(episode.dataset_name, episode.episode_index, episode.metrics) ``` Numeric fields are the lake columns `duration_s`, `frame_count`, and `quality_score`, plus Qualia's typed computed metrics. Channel predicates support `action`, `state`, and `telemetry` streams with `min`, `max`, `mean`, or `std`. Every matching episode carries `channel_predicate_evidence`: the exact native channel, index, aggregate observation, operator, and threshold that satisfied each predicate. The SDK binds that evidence back to the ordered request and rejects missing, changed, non-finite, or non-satisfying observations. Search is a literal, case-insensitive substring match over task and dataset name; `%` and `_` have no wildcard meaning. A facet axis returns at most 512 values, while `by_task_total_values`, `by_tier_total_values`, and `by_interface_hash_total_values` report the complete cardinality. Active selections are retained when an axis is truncated. Each response's page, totals, and all facet axes come from one database statement and therefore one live MVCC snapshot. `page_limit` and `page_offset` bind the response to the request. Offset pagination is not mutation-stable between calls: after publishing data or changing review state, restart at offset zero. Use exploration for analysis and review, then publish a pinned DatasetVersion for reproducible training. The SDK rejects malformed rows, duplicate identities, inconsistent totals or facets, altered evidence, and truncated or mismatched pages. ## Split episodes at their recorder markers A continuous teleop session is uploaded as pause/resume-delimited sequences. The presses inside one — a foot pedal calling a take a success or an issue — are recorded as markers. Splitting draws the windows to keep on each take's draft. The recording and dataset head stay unchanged; publishing a version from a Publish preview creates the pieces that version contains. These calls are on `client.episodes` (qualia.v1, since 0.8.0rc1). They need your account, and they name an episode through the upload it was recorded in: a bare id with `dataset=`, or its full name. ```python episode = client.episodes.get(episode_id, dataset=upload) for marker in episode.markers: print(marker.time_seconds, marker.label, marker.role) result = client.episodes.split_at_markers( upload, [episode_id], labels=["success", "issue"] ) for split in result.splits: print(split.episode, split.piece_count, split.draft.etag) for skipped in result.skipped: print(skipped.episode, skipped.reason) ``` `labels` names marker labels as the recording wrote them, never the role the ingest assigned. Each cut closes the span before it, and the draft piece keeps that cut's labels as tags. To cut at instants of your choosing, use `client.episodes.split(upload, {episode_id: [12.0, 32.8]})`, seconds from the start of the recording. `start_labels` and `end_labels` narrow each episode first, to ITS first and last marker so labelled; given without `labels`, each draft keeps one piece, the window. `dropped_lead_frame_count`/`dropped_tail_frame_count` say what was left out. Pass a list of episodes or one `filter=` to cut a whole session; an episode the server cannot cut that way (no marker matched, or a split is already drawn) is listed in `skipped` with its reason and the rest go ahead. Takes another person is editing are listed in `locked` with their holders. A split is planned inside the draft's kept window, or the take's existing trim when it has no draft. `dropped_lead_frame_count` and `dropped_tail_frame_count` include the frames that trim leaves out. A saved split returns its `draft` and `piece_count`; `children` is empty. Pass `validate_only=True` to see the plan without saving it. `client.episodes.unsplit` discards a split draft, and `client.episodes.discard_draft` discards any drawing. Existing published versions keep their pieces and pins while the take's draft remains editable. Qualia privileges no marker vocabulary — the mapping from recorded values to roles is declared per upload. ## Trim, name, and undo episode edits Trim and subtask edits save a draft, just as studio edits do. Task and tag labels still publish a new revision on head. An SDK update combining both sends the drawing first and the labels second; those requests are not atomic. ```python trimmed = client.episodes.trim_to_markers( upload, episode_ids, start_labels=["ready"], end_labels=["done"] ) restored = client.episodes.clear_trims(upload, episode_ids) for skipped in trimmed.skipped: print(skipped.episode, skipped.reason) ``` Each episode is narrowed to start at ITS first marker so labelled and end at ITS last, so one call fits a whole selection whose presses landed at different instants. The recording is never overwritten: `clear_trims` draws the whole recording again. An episode with no such marker, or one outside its frames, lands in `skipped` with a reason. `name_subtasks` uses the same boundary rule to label a span instead of carving it — useful when you want the structure recorded but the episode left whole: ```python named = client.episodes.name_subtasks( upload, episode_ids, labels=["grasp", "place"], boundary="ends_at", # or "starts_at" to have each marker OPEN its span ) ``` Each drawing helper works from the effective draft, preserving its other edits, and sends `expected_draft_etag` so a concurrent drawing returns `QualiaAPIError` with code `"aborted"`. A trim over a drafted split is refused; a subtasks edit preserves the split's pieces. For one episode, `client.episodes.update` edits trim, subtasks, task, tags and the other editable fields: ```python episode = client.episodes.get(episode_id, dataset=upload) client.episodes.update(episode.name, tags=["reviewed"], etag=episode.etag) ``` `client.data.list_episode_revisions(episode_id)` still reads the episode's append-only revision chain, the audit trail of who changed it and when. ## Analyze an immutable DatasetVersion Run native action/state health analysis against one exact published version: ```python report = client.data.health_for_version( "44444444-4444-4444-8444-444444444444" ) print(report.health_score, report.grade) for flagged in report.flagged_episodes: print(flagged.member_ordinal, flagged.episode_revision_id, flagged.reasons) ``` Qualia re-proves the complete ordered membership and dataset-snapshot identity. It then fully proves and reads exact Parquet ranges for an evenly spaced, deterministic sample of at most 120 members. Every flag is bound to an immutable episode revision and is a review recommendation, not an automatic deletion. The sample is explicit in `report.sampling`. This synchronous analysis is not a byte scrub of unsampled artifacts or video, and it does not claim model-based visual or behavioral metrics without a pinned model revision and output artifact; those fields are marked unavailable in `evidence_completeness`. ## Review and apply bad-data suggestions Analyze one or more immutable DatasetVersions without changing them. Every recommendation includes a risk score, machine-readable reasons, and supporting evidence. The response binds the complete recommendation set to an evidence digest and the exact paired Git/lakehouse head. ```python evidence = client.data.suggest_curation( ["44444444-4444-4444-8444-444444444444"] ) for suggestion in evidence.suggestions: print(suggestion.episode_index, suggestion.risk_score, suggestion.reasons) ``` After review, publish the analysis's sealed recommendation set as a new immutable DatasetVersion. `apply_curation` starts a durable publication and waits for its receipt; reuse the same idempotency key after a lost response or a timeout to resume it. The source version, source bytes, and Git history remain available. ```python from uuid import uuid4 receipt = client.data.apply_curation( evidence, idempotency_key=uuid4(), definition_id="55555555-5555-4555-8555-555555555555", note="Reviewed automated bad-data evidence", ) print(receipt.version_id, receipt.membership_hash, receipt.git_oid) ``` Application fails closed if the analysis selection, evidence digest, repository head, target definition, membership counts, snapshot, or Git receipt changes between review and publication. Resuming with the same idempotency key inside the same process — passing the same `evidence` object again — replays cleanly. A later process that no longer has that `evidence` (a retried CLI invocation, say) should call `resume_curation_publication(idempotency_key)` first. It follows a publication that is queued, running or published to its receipt, and raises when the publication failed: a failed key stays failed, so publish again under a new key. It returns `None` when no publication exists for the key, or when its lease is `retryable`; then suggest fresh evidence and publish with the same key. ```python receipt = client.data.resume_curation_publication(idempotency_key) if receipt is None: evidence = client.data.suggest_curation([version_id]) receipt = client.data.apply_curation( evidence, idempotency_key=idempotency_key, name="curated" ) ``` ## Run a durable curation analysis `health_for_version` reads one version's health directly. When you want the analysis to survive a disconnect — or to span several pins at once — start a durable one and read its progress back: ```python import time analysis = client.data.start_curation_analysis( [version_id], idempotency_key=operation_uuid, ) while analysis.status not in ("ready", "failed"): time.sleep(2) analysis = client.data.get_curation_analysis(analysis.analysis_id) if analysis.status == "failed": raise RuntimeError(analysis.failure_detail or analysis.failure_code) page = client.data.list_curation_suggestions(analysis.analysis_id) for suggestion in page.suggestions: print(suggestion) ``` The analysis is keyed to exact DatasetVersion pins, so the same `idempotency_key` recovers the running analysis instead of starting a second one. Suggestions are only listable once `status` reaches `ready` — the server enforces that seal — and they page by opaque keyset — pass `page_token` to continue. Applying them is `apply_curation`, which publishes a new immutable DatasetVersion and deletes nothing. ## List datasets `list()` is the legacy list: it reads the older list route and returns `qualia.Dataset`. `client.datasets.list(filter="kind != MIXTURE")` replaces it, reads the qualia.v1 Datasets API and returns `qualia.v1.Dataset`; it needs your account (the API key's own). Uploads merged into one dataset (`client.datasets.merge`) are listed under it, as a dataset of kind `MERGED`; filter `kind = MERGED` for those. A merge refuses uploads that differ in what training freezes, and asks about what a write settles: preview it with `validate_only=True`, which returns the conflicts and writes nothing, then answer each with `resolutions`. A label or Task needs no answer: the merged dataset takes its uploads' commonest value, or the one you pick, as its own; each upload keeps its own. A whole merge cannot be undone: the merged dataset holds the uploads (their reads keep answering), and every write goes through the merged dataset. `remove_merged_member` takes one upload out again; it becomes a standalone upload with its data and review decisions, and the merged dataset's history records the removal. ```python uploads = ["week-a-id", "week-b-id"] conflicts = client.datasets.merge(uploads, display_name="Cable bench", validate_only=True) answers = [ {"field": c.field, "channel": c.channel, "value": c.sides[0].value} for c in conflicts if not c.blocking ] merged = client.datasets.merge(uploads, display_name="Cable bench", resolutions=answers) ``` ```python page = client.data.list() for d in page.items: print(d.artifact_id, d.path) ``` `client.datasets`, `client.episodes`, `client.versions` and `client.mixtures` cover the rest of the qualia.v1 API: mixtures, episode rulings and edits, and version tags. The SDK README has examples. ## Edit collection tags `client.datasets.update(..., tags=...)` edits collection classifications through the saved method, environment, origin and controller metadata. The recognized tags are `teleop`, `vr`, `umi`, `egocentric`, `policy-rollout`, `sim`, `real-world`, `human` and `gello`; you can also keep your own tags. The limit is 64 distinct tags. A changed, nonempty classification set replaces those metadata choices and the custom tags in one atomic update. Nonempty custom-only input, or unchanged classifications with edited customs, replaces only custom tags; undeclared or explicit Unknown methods and inference are preserved. When classifications are present, echo them to clear customs without changing methods. `tags=[]` explicitly makes every collection dimension Unknown and clears customs, preventing inference from refilling the set. ```python dataset = client.datasets.get("dataset-uuid") dataset = client.datasets.update( dataset.name, tags=["umi", "real-world", "human", "gello", "warehouse-a"], etag=dataset.etag, ) dataset = client.datasets.update(dataset.name, tags=[], etag=dataset.etag) # Custom-only replacement preserves the collection classifications and Unknown. dataset = client.datasets.update(dataset.name, tags=["warehouse-b"], etag=dataset.etag) ``` The SDK first reads the owner and uses its etag when none was supplied. A stale caller etag is refused rather than retried with a new one. These edits require a server that returns the independent collection metadata fields. Contradictory values on one axis, such as `vr` together with `gello`, are refused. A merged dataset owns this set: edits change C alone, and adding uploads preserves its tags, including Unknown. Mixture selection, collection filters and reach eligibility use the owner's tags. Recording methods on episodes, episode curation tags and immutable version tags retain their own meanings. The older `methods=` update changes only the six-name compatibility view, preserving other dimensions and free tags. Use `tags=` separately from those legacy collection arguments. ## Set up a handheld (UMI) dataset `client.datasets` does what the dashboard's Setup strip does: pick the robot a handheld upload plays on, set its placement, and move the poses to the robot's TCP. It acts in the API key's own account. A dataset has no solver of its own: it follows its robot's, which is set on the robot. `poses()` says where an upload's poses are: `root` (the gripper root's `.ee.*` poses, as ingest placed them), `tcp` (each hand's TCP too) or `recorded` (a raw recording, which a resolve converts). It also names the capture gripper, the workspace gripper whose `rig_key` is the upload's rig, and whether its calibration has changed since the poses were resolved. ```python poses = client.datasets.poses("dataset-uuid") print(poses.at, poses.rig, poses.current) setup = client.datasets.setup( "dataset-uuid", robot="robot-uuid", # an id or accounts/{a}/robots/{r} base=[0.0, 0.0, 0.0], # metres in the robot's base frame, within 5 m heading_deg=0.0, ) print(setup.changed, setup.resolve, setup.resolve_skipped) ``` Only what you pass is written; the rest of the reach settings are kept, and a call that changes nothing writes nothing, so it can be repeated. ### Resolving the poses at the TCP Picking a robot with `robot=` starts a resolve at the TCP by itself when all of this holds: - the dataset is an upload with handheld poses (not a mixture or a merged dataset); - every arm of the robot has a TCP; - the workspace has exactly one UMI rig of the upload's kind (`poses.rig`) and it has a tracker-to-TCP calibration, the capture gripper; - the poses are not at the TCP yet, or are there with another calibration. The resolve uses the capture gripper's calibration, not the robot's own gripper's: the robot only decides when. Otherwise nothing starts and `resolve_skipped` says why (`robot-without-tcp`, `no-capture-gripper`, `several-umi-rigs`, `no-calibration`, `up-to-date`, `not-an-upload`, `no-poses`). With several UMI rigs of one kind none is the upload's: `client.datasets.resolve_poses(dataset, gripper=rig)` names the one to resolve with. `resolve=False` never resolves; `resolve=True` resolves with the capture gripper whatever the robot. A resolve re-derives the whole upload, so it takes as long as ingesting it did: minutes to hours. `wait=True` stays for it, and `on_progress` sees each state. ```python setup = client.datasets.setup( "dataset-uuid", robot="robot-uuid", wait=True, on_progress=lambda r: print(r.state, r.progress), ) # Or move the poses by hand, and wait later started = client.datasets.resolve_poses("dataset-uuid", "tcp", wait=False) client.datasets.wait_resolve(started.operation) client.datasets.resolve_poses("dataset-uuid", "root") # back to the gripper root ``` A failed re-derive raises `QualiaError` with the server's reason (a ruling or a trim made while it ran is one cause: resolve again). A wait that runs out (`TimeoutError`, two hours by default) leaves the work running; wait for it with `wait_resolve`, and do not resolve again before the poses have moved. ### Run the solver Run solver is never started by itself. `setup(run_solver=True)` presses it once the poses have settled, after waiting for a resolve it started; `checks.run(dataset, "reach")` presses it without touching the setup. Follow and stop the run with the checks resource: ```python client.datasets.setup("dataset-uuid", run_solver=True) status = client.checks.reach_status("dataset-uuid") run = status.last_run print(status.settings.solver, run.state, run.done, run.total) client.checks.cancel_reach("dataset-uuid") # solved takes keep their results ``` `current_key` is empty when the server cannot key a run on the dataset's robot (none, no model, another workspace): a run would start nothing, and `setup` refuses to press it. `datasets.update(dataset, reach_settings={"base_m": [0.1, 0, 0], "home_heading_rad": 0.0})` writes the settings alone; it replaces both, so what you leave out follows the robot's model, and it starts no solve. A `solver` in `reach_settings` is refused. ## Watch in-flight uploads `upload()` narrates the server's side of a transfer on stderr, but only in the process running it. The account's pending feed, the one the dashboard's pending band renders, is also readable directly, so an upload running detached on a rig can be watched from anywhere: ```python for op in client.data.uploads(): print(op.name, op.phase, op.progress_line or op.message) ``` Rows in `uploading` carry the SDK's heartbeat (`transferred_bytes`, `staged_files`); rows in `preparing` carry the worker's live step in `progress`, refreshed with its lease every 30 s; a `failed` row survives 24 h with `message`, `failure_retryable` and, when ingest got that far, `ingest_report`. The model is tolerant on purpose: `phase` is a plain string and unknown fields are kept, because the platform adds steps without an SDK release. ## Dismiss or cancel an upload, follow an import ```python op = client.data.uploads()[0] client.data.cancel_upload(op.repo_id) # stops it from any machine client.data.dismiss_upload(op.repo_id, dismissed_through_unix=op.updated_at_unix) for row in client.data.list_huggingface_imports(): print(row["import_id"], row["status"], row["progress_percent"]) client.data.get_huggingface_import(import_id) client.data.cancel_huggingface_import(import_id) ``` Cancel refuses the uploading process's next step and stops server-side work on its next heartbeat; nothing is published, and it is refused once the upload has published. Dismiss hides a failed or finished entry from the feed and keeps the record of why it failed; a live upload is never hidden (`phase_now` names it). A key restricted to workspaces is refused on these routes. ## Browse DatasetVersions and their sources `list` returns canonical datasets. A _DatasetVersion_ is an immutable pin of one — what training actually consumes. Since 0.8.0rc1 versions are qualia.v1 tags on a dataset, and each tag carries the DatasetVersion id it pins: ```python for tag in client.versions.list(dataset).items: print(tag.tag, tag.dataset_version_id, tag.create_time) tag = client.versions.get("stable", dataset=dataset) # or a tag, or a digest prefix commits = client.versions.list_source_commits(tag.name) for commit in commits.items: print(commit.dataset, commit.repo_id, commit.git_oid) ``` `list_source_commits` returns the exact repository commits that contributed the version's members, which is how you answer "which upload did this training set actually come from". Lists page with `limit` / `cursor`. ## Saved lake queries A _dataset definition_ is a saved lake query plus the versions curation published into it. ```python for definition in client.data.list_definitions().defs: print(definition.id, definition.name) definition = client.data.get_definition(definition_id) consumers = client.data.list_definitions_by_source(source_dataset_id) ``` `list_definitions_by_source` answers the reverse question — which definitions consume a given source, directly or through an immutable pin — so you can see what a re-upload would affect before you publish it. ## Tag a dataset An upload registers a dataset. Training features, health, curation and materialization all take an immutable DatasetVersion, so a fresh upload needs a tag before anything downstream can run: ```python tag = client.versions.create(dataset, "1.0", note="first cut") print(tag.reference, tag.dataset_version_id, tag.episode_count) ``` A tag freezes exactly what the dataset yields now: an upload's kept episodes, a mixture's episodes it has not rejected. Dropping flagged work is `apply_curation`'s job. Tags never move, and a later tag of a changed dataset pins a new version. `client.versions.compare(dataset)` shows what changed since the last tag and suggests the next name. `client.versions.create` tags head and leaves saved trim, split and subtask drawings out. The returned version's `drafts_not_included` counts those drafts. Use the dashboard's Publish dialog to include them in a measured version; SDK and CLI helpers for publishing previews will follow separately. ## Define a saved query `create_definition` saves a lake query as a definition, a target `apply_curation` can publish versions into. `pin_version` freezes what the query selects now as the definition's next DatasetVersion: ```python from qualia.models import DatasetSavedQuery definition = client.data.create_definition( "kitchen-2026-09-16", query=DatasetSavedQuery(dataset_ids=[source.artifact_id]), description="one recorded session", ) version = client.data.pin_version(definition.id, note="first pin") ``` Pinning a definition stays on the `/v1` path: saved queries have no qualia.v1 equivalent. Tag an upload or a mixture with `client.versions.create` instead. `DatasetSavedQuery` mirrors ExploreLake's filters field for field: `dataset_ids`, `task`, `tier`, `min_duration_s` / `max_duration_s`, `tags`, `review_state`, `search`, `numeric_predicates`, `predicates` and an exact `episodes` selection. An empty query selects the whole lake. ## Get immutable training features Training accepts a pinned DatasetVersion, never a mutable repository name or external dataset identifier. Retrieve its exact camera descriptors: their `key` values are the cameras the dashboard's training launcher maps onto a model's camera slots. ```python features = client.data.get_training_features( "44444444-4444-4444-8444-444444444444" ) for camera in features.camera_features: print(camera.key, camera.stream, camera.timebase_numerator, camera.timebase_denominator, camera.frame_timestamp_mapping) ``` **Response** ```json { "dataset_version_id": "44444444-4444-4444-8444-444444444444", "training_adapter_revision_id": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", "camera_features": [ { "key": "observation.images.wrist", "stream": "video/wrist", "timebase_numerator": 1, "timebase_denominator": 1000000, "frame_timestamp_mapping": "time_start_plus_frame_over_fps" } ] } ```