# Use Qualia from an agent

> Curate a dataset before training with an API key, from the CLI, the SDK or plain HTTP

An agent uses Qualia with an API key, through the CLI or the SDK. A key acts as
the person who made it, inside one organization. It can do what that person's
role allows, and its scopes can only narrow that.

This page walks one job from start to end: curating a dataset before a training
run. You upload it, link your tasks, wait for the checks Qualia starts by
itself, read the results as one table, rule on the episodes, pin a version and
download it. Then it covers the cost, plain HTTP, a new batch every day, robots,
finding your way around, admin work, and what a key cannot do. Each step shows
the CLI first, then the same work in Python. The
[command reference](https://junction-docs.qualiastudios.dev/cli/commands.md) and the [SDK pages](https://junction-docs.qualiastudios.dev/sdk/overview.md) list
every option.

Pass `--json` for output a script can read. A command that fails exits 1 with
the reason on stderr. A command used wrongly exits 2.

## Install

The CLI is the `qualia-cli` package, and it installs the SDK (`qualia-sdk`)
with it. Install the version this deployment speaks: the exact command is under
[Installation](https://junction-docs.qualiastudios.dev/cli/overview.md#installation), and the one for the SDK alone on
the [SDK overview](https://junction-docs.qualiastudios.dev/sdk/overview.md#installation). Keep the pin: a client and a
deployment negotiate the upload protocol exactly, so they must be of the same
generation.

```bash
qualia version
```

It prints the CLI and SDK versions that are installed.

## Keys

Make a key in the dashboard: open **Settings**, then **Account** › **API
keys**, click **New key**, then **Create key**. Copy the key: it is shown once. Keys expire after 90 days by default. No key can make another
key.

The key acts in the workspace it was made in. `QUALIA_WORKSPACE` picks another
workspace you can reach. The reference pages call it an organization key. Every
step on this page works with it.

A key restricted to some workspaces (a workspace key) reaches less for now,
and cannot walk this path: it is refused on tasks, which belong to the whole
organization. It lists only its own workspaces, and it cannot change or delete
a workspace.

A key carries every scope except three admin scopes, so it can upload and curate
as soon as it exists. The three are `members:read`, `members:write` and
`integrations:write`. None is on by default. Open **More options** and tick
one under **Admin (opt-in)** when you make the key. A key with `*` does not get them. See
[Admin scopes](https://junction-docs.qualiastudios.dev/cli/overview.md#admin-scopes).

Give the key to the CLI once:

```bash
qualia auth login --base-url https://junction-api.qualiastudios.dev
qualia auth status
```

`qualia auth status` shows what is stored. It does not call the API.
`qualia auth whoami` does, so it proves the key works: it answers with the
account, the workspace, the role and the key's name and reach. Every command
acts in the key's own account, so none needs an account id
([Your account](https://junction-docs.qualiastudios.dev/cli/overview.md#3-your-account)).

An agent usually gets its key from the environment instead. Set `QUALIA_API_KEY`
and `QUALIA_BASE_URL` together, because the CLI ignores its config file once the
key is set. `QUALIA_WORKSPACE` picks the
workspace to act in, if it is not the key's own.
[Scripts and CI](https://junction-docs.qualiastudios.dev/cli/overview.md#scripts-and-ci) has the details.

The SDK reads the same variables:

```python
from qualia import Qualia

client = Qualia()
```

## The path

### 1. Upload

```bash
qualia dataset upload ./session-0412 session-0412 --standalone --json > receipt.json
DATASET=$(jq -r .dataset_id receipt.json)
qualia dataset ingest-report "$DATASET"
```

The later steps reuse `$DATASET`. The upload's exit code says what it came to: 0
published, possibly with flagged episodes; 1 it failed or was refused; 3 the
bytes are kept but no episode could be derived.
[`qualia dataset upload`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-upload) lists the
formats.

An episode keeps its place and its task. Its `index` is its `episode_index` in
a LeRobot folder, counted from 0, and its task label is the folder's task
string for it: the text training reads.

The upload is not the end of it. Where a deployment prepares uploads,
preparation and two checks start by themselves, and one of the checks costs
credits. Read [step 3](#3-know-what-starts-by-itself) before you start anything.

Exit 1 does not mean nothing is left. A transfer that fails partway stays in the
upload feed as `failed`, and the bytes it stored stay too. Read the feed before
you try again:
[`qualia dataset uploads`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-uploads) works from any
machine. [`cancel-upload`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-cancel-upload) stops an
upload in flight.
[`dismiss-upload`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-dismiss-upload) takes a failed
one off the feed and leaves its bytes.

`--standalone` keeps the upload on its own. Without it, a machine with a saved
[destination](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-destination) appends the upload to
that merged dataset. The upload is then part of that dataset, and the server
refuses the steps below on it: run them on the merged dataset
([A new batch every day](#a-new-batch-every-day)).

```python
receipt = client.data.upload("./session-0412", name="session-0412", standalone=True)
dataset = receipt["dataset_id"]
report = client.datasets.ingest_report(dataset)
```

See [data](https://junction-docs.qualiastudios.dev/sdk/resources/data.md) and [datasets](https://junction-docs.qualiastudios.dev/sdk/resources/datasets.md).

### 2. Set up the robot and gripper

Curation needs no robot: skip this step. A dataset needs one only to be played
on it, which the reach check and handheld (UMI) recordings do. The commands are
under [Robots and grippers](#robots-and-grippers).

### 3. Know what starts by itself

Where a deployment prepares uploads, an upload starts three things and asks
about none of them:

- **Preparation** reads the takes. Count on 3 to 8 minutes and allow for more:
  one run of 24 episodes took 17, and the platform's own deadline for a run is
  two hours. It sorts the takes into tasks and gives each one a Task: a
  published Task of the account that matches it, somebody else's included, or a
  new draft. It links the episodes to that Task, and it may split a take into
  pieces.
- **The free checks** (heuristic: code reads each episode's data, with no
  model) start when preparation ends.
- **The paid check** (VLM: a model reads each linked episode against its Task)
  starts then too. It reserves its whole estimate from the account's credits.
  No upload option caps it or turns it off.

```bash
qualia dataset preparation "$DATASET"    # its state, the tasks found, the Task of each
qualia check runs --dataset "$DATASET"   # the runs on the dataset, newest first
qualia check runs --dataset "$DATASET" --json \
  | jq -r '.checkRuns[] | [.name, .state, .creator.agent] | @tsv'   # the same, with whole names
RUN=$(qualia check runs --dataset "$DATASET" --json | jq -r '.checkRuns[0].name')
qualia check status "$RUN"               # one run, by its whole name
```

The table cuts a run's name short, so take names from `--json`. A run that
started by itself has `qds-scheduler` as its `creator.agent`.

Whether this deployment prepares uploads shows in `qualia dataset preparation`,
not in the upload's receipt. A state of `QUEUED`, `RUNNING`, `SUCCEEDED` or
`FAILED` means it does, and the checks follow. `No prepare run` (state `NONE`
with `--json`) means no preparation is queued: nothing starts by itself, and
you start the checks in step 5. Ask twice, some seconds apart, before you
believe `NONE`. Right after an upload the preparation may not be queued yet,
which is also why the receipt's `automatic_check` can be missing where uploads
are prepared.

The paid check judges each episode against the Task it links when the run
starts: yours if you linked it first, else the one preparation gave it. So if
you bring your own Tasks, do step 4 now, while preparation runs, and you pay
for one reading, not two.

To stop a run you do not want, this one included, cancel it by the last part of
its name: `qualia status cancel "${RUN##*/}"`
(`client.statuses.cancel(run.name.rsplit("/", 1)[-1])`). The run ends `FAILED`.
It pays for the calls it had made, and the rest of its reservation comes back.

See [preparation](https://junction-docs.qualiastudios.dev/sdk/resources/preparation.md) and
[`qualia dataset preparation`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-preparation).

### 4. Link your tasks

A Task says what the robot should do and how success shows. Make one for each
task string in the dataset, publish it, and link the episodes that carry the
string:

```bash
qualia dataset summary "$DATASET" --facet task    # the dataset's task strings, with counts
TASK=$(qualia task create pour.yaml --name "Pour coffee" --json | jq -r .name)
qualia task publish "$TASK" -m "First revision"
qualia dataset assign-task -d "$DATASET" --task "$TASK" \
  --label "pour coffee into the cup" --filter 'task = "pour coffee into the cup"'
```

`pour.yaml` holds your rules. A description and a success are enough to
publish:

```yaml
description: Pour coffee from the pot into the cup
success: The cup is full and no coffee is on the table
failure: Coffee is spilled, or the cup tips over
```

- [`assign-task`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-assign-task) links every
  episode the filter leaves, so a dataset with four tasks takes four calls.
  Repeat the string the episodes already carry as `--label`: without it, their
  task label is replaced by the Task's name.
- A Task's name is at most 32 characters, so a long instruction cannot be it.
- Publishing a Task links, by itself, every unlinked episode of the workspace
  whose task label is exactly the Task's name, in other datasets too.
  `qualia task link-episodes "$TASK" --dry-run` lists what that would be.
- `qualia dataset link-task "$DATASET" "$TASK"` links the dataset instead. Only
  an episode with no link of its own reads that link, so it leaves the episodes
  preparation linked where they are. `assign-task` moves them.

Preparation leaves an episode you linked alone, so link before it ends and the
check that starts by itself judges your Tasks. Link later and the links move
all the same, but what that check read against the other Tasks no longer
stands: run it again in step 5.

You can keep the Tasks preparation gave instead.
`qualia dataset preparation "$DATASET"` lists each task it found with the id of
its Task, and `qualia task get ID` shows what it says. Against a draft, a check
raises notes only: publish it (`qualia task publish ID`), then run the check
again in step 5. Two things to know. Preparation's link also sets the task
label of the episodes it links to the Task's name: `--facet task` shows it, and
`assign-task --label` puts a string back. And saving an edit to a Task that was
never published starts the free and the paid check on its dataset again, by
themselves, about two minutes after the last edit.

A link keeps the revision the Task had when it was made, and checks judge that
revision: publishing a newer one starts no check. Each run names what it
judged: `qualia check status <run>` prints `Judged:    Restock shelf v3`. When a
newer revision is out, `qualia dataset get` says so
(`v4 is published: qualia dataset relink-task`), and the dataset page's Task
panel reads "Restock shelf v4 is out; checks ran on v3." One call moves the
dataset's own link, and the episodes linked to that same Task, to the newest
revision and reruns the checks preparation starts, the VLM check among them:
`qualia dataset relink-task "$DATASET"` (`client.datasets.relink_task`), or the
panel's "Relink to v4 and rerun checks". Both show the credit bound first:
`--validate-only` (`validate_only=True`) answers it and changes nothing, and the
dashboard asks before it starts. It needs the dataset itself to link the Task
(`link-task`): a dataset linked only through its episodes answers
`links no Task; link one first`. It never moves episodes linked to another Task
than the dataset's own.

```python
task = client.tasks.create(
    "Pour coffee",
    draft={
        "description": "Pour coffee from the pot into the cup",
        "success": "The cup is full and no coffee is on the table",
        "failure": "Coffee is spilled, or the cup tips over",
    },
)
client.tasks.publish(task.name, note="First revision")
client.episodes.assign_task(
    dataset,
    "pour coffee into the cup",
    filter='task = "pour coffee into the cup"',
    linked_task=task.name,
)
```

See [`qualia task`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-task), [tasks](https://junction-docs.qualiastudios.dev/sdk/resources/tasks.md)
and [Robots and tasks](https://junction-docs.qualiastudios.dev/global/guides/robots-and-tasks.md).

### 5. Wait, then read the results

A check never keeps, rejects or deletes an episode. It raises issues and notes.
The runs that start by themselves come when preparation ends. Wait for it, then
for them, before you read a result, and give every wait a limit:

```bash
wait_until() {   # wait_until SECONDS COMMAND...: ask every 20 s, give up after SECONDS
  local deadline=$((SECONDS + $1)); shift
  until "$@"; do
    [ "$SECONDS" -lt "$deadline" ] || { echo "gave up waiting for: $*" >&2; return 1; }
    sleep 20
  done
}
prepared() {   # prepared UPLOAD: its preparation is neither queued nor running
  qualia dataset preparation "$1" --json | jq -e '.state != "QUEUED" and .state != "RUNNING"' > /dev/null
}
followed() {   # followed DATASET SINCE KIND...: a run of each KIND started by itself after SINCE, all ended
  qualia check runs --dataset "$1" --json | jq -e --arg since "$2" '
    [.checkRuns[]? | select(.creator.agent == "qds-scheduler" and .createTime > $since)
      | .name | split("/")[3]] as $came    # accounts/A/checks/KIND/runs/R
    | all($ARGS.positional[]; IN($came[]))
      and all(.checkRuns[]?; .state == "SUCCEEDED" or .state == "FAILED")' --args "${@:3}" > /dev/null
}
wait_until 1800 prepared "$DATASET" &&
  PREP=$(qualia dataset preparation "$DATASET" --json) &&
  [ "$(jq -r .state <<< "$PREP")" = SUCCEEDED ] &&
  wait_until 1800 followed "$DATASET" "$(jq -r .createTime <<< "$PREP")" heuristic vlm
```

```python
import time


def wait_until(done, seconds=1800, every=20):
    deadline = time.monotonic() + seconds
    while not done():
        if time.monotonic() > deadline:
            raise TimeoutError("gave up waiting")
        time.sleep(every)


def followed(dataset, since, kinds=("heuristic", "vlm")):
    """A run of each kind started by itself after `since`, and every run has ended."""
    runs = client.checks.list_runs(dataset=dataset).items
    came = {
        r.name.split("/")[3]  # accounts/A/checks/KIND/runs/R
        for r in runs
        if r.creator and r.creator.agent == "qds-scheduler" and r.create_time > since
    }
    return came >= set(kinds) and all(r.state in ("SUCCEEDED", "FAILED") for r in runs)


wait_until(lambda: client.preparation.get(dataset).state not in ("QUEUED", "RUNNING"))
prep = client.preparation.get(dataset)
if prep.state == "SUCCEEDED":
    wait_until(lambda: followed(dataset, prep.create_time))
```

An empty list of runs does not mean they ended. Right after preparation the
runs may not have started yet, and the list may hold only a run you started or
older ones. One finished run does not mean it either: the free run starts
first, and the paid one can come minutes after it ended, since it waits for any
VLM run already on the dataset, yours too. So the wait names both kinds,
`heuristic` and `vlm`, the fourth part of a run's name. It looks for a run of
each that started by itself (`creator.agent` is `qds-scheduler`) after
preparation was queued: none starts while preparation is queued or running.
Then it waits for every run on the dataset to end. Name only `heuristic` where
no episode links a Task, published or draft: no paid run comes then. Only a
preparation that `SUCCEEDED` is sure to be followed. After `FAILED` (its
`error` says why) or `NONE`, the wait stops there: start the checks yourself,
below. At the limit, `qualia check runs` shows which kind never came.

You need not wait for preparation to start a run of your own. Once your
episodes link your published Tasks (step 4), a run you start begins at once,
whatever preparation is doing. Know what follows:

- When preparation ends, the automatic runs start all the same. The paid one
  takes every linked episode again and reserves its whole estimate again. A
  reading it can reuse is not charged twice ([What it costs](#what-it-costs)),
  but nothing says beforehand how many it will reuse. Cancel it (step 3) if the
  account cannot spare the reservation, or if you have the verdicts you need.
  Once cancelled it stays away until the dataset is touched again: by a new
  upload, or by an edit to a Task that was never published.
- Two VLM runs never read one dataset at once. While yours runs, the automatic
  one waits for it to end. A run you ask for while another is on the dataset is
  queued behind it, and reserves credits only when it starts.
- A take that preparation splits after your run is replaced by its pieces,
  which your run did not read.

Two refusals only mean "wait". Until preparation has linked the episodes,
`qualia check estimate` answers
`no selected episode is linked to a published task or has a draft task`. While
the free checks run, starting them again answers
`… already running on this dataset in …; wait for it to end`.

Start the checks yourself where none starts by itself, when you will not wait
for preparation, or after you linked Tasks late:

```bash
qualia dataset checks list "$DATASET"    # every check: whether it can run here, its last result
qualia dataset checks run "$DATASET" --kind heuristic --all --wait
qualia check estimate "$DATASET"
qualia check run "$DATASET" --max-cost 200 --wait
```

The heuristic checks read each episode's data with no model and cost nothing.
The VLM check reads each episode against its linked Task with a model and costs
credits: `qualia check estimate` starts and charges nothing, and `--max-cost`
caps what the run may reserve and spend ([What it costs](#what-it-costs)). VLM
catalog checks cannot run on this deployment.

```python
heuristic = client.checks.run(dataset, kind="heuristic", all=True)
client.checks.wait(heuristic.check_run)
estimate = client.checks.estimate(dataset)
started = client.checks.run(dataset, max_cost=200)
vlm_run = client.checks.wait(started.check_runs[0])  # a VLM check starts one run
```

Read the results as one row per episode, from three lists:

```bash
qualia dataset episodes "$DATASET" --filter 'child_count = 0' -n 1000 --json > episodes.json
qualia dataset checks findings "$DATASET" --check graph.evaluate -n 1000 --json > verdicts.json
qualia check findings "$DATASET" -n 1000 --json > fails.json
```

- `episodes.json`, under `episodes`: every episode, with its `name`, `index`,
  `task`, `state` and `openIssueCodes` (what blocks it, from any check), and a
  `decision` once someone has ruled on it. The filter leaves out a take that
  was split and keeps its pieces. Without it the list holds both, the take
  with `childCount` above 0 and each piece naming it in `parentEpisode`, while
  the dataset's own counts and a version hold the pieces only.
- `verdicts.json`, under `checkFindings`: what the VLM check concluded on each
  episode it read, passes too: `episode`, `verdict` (`PASS`, `FAIL` or
  `UNKNOWN`), `confidence`, `explanation`, `taskRevision` (the Task it was
  judged against) and `checkRun`. An episode with no finding here was **not
  checked**, whatever its `state` says.
- `fails.json`, under `issues`: only the failures and doubts, each with
  `blocks`, its `state` and, where the model pointed at any, moments in
  `finding.moments[].timeSeconds`: seconds into the recording. A pass has none,
  and a failure can have none either. A finding names its issue in `issue`,
  which is the issue's `name`. Where a finding says `UNKNOWN`, its issue says
  `UNCERTAIN` and the run counts it in `uncertainCount`: all three mean the
  model could not tell.

One check goes by three names. You run it as `vlm`. Its findings are filed
under `graph.evaluate`, marked `legacy: true` and `checkVersion: 0` even when
new. Its issues carry the code `graph_outcome`, the verdict on the task as a
whole. A Task with a step graph adds findings about its steps (`graph_walk`,
`graph_count_…`), so an episode can have several, and one `FAIL` fails it.

What a failure does depends on how sure the model was. At 80% confidence or
more, against a published Task, it raises an issue that blocks
(`blocks: true`): the episode's `state` is `UNREVIEWED` until someone rules on
it. A failure below 80%, a doubt, anything read against a draft Task, and a
failure on an episode you had already kept raise a note (`blocks: false`),
which holds nothing. So `state = KEPT` is not a verdict. An episode nobody
read, one read against a draft, and one that failed at 60% all stand kept.

```python
episodes = client.episodes.list(dataset, filter="child_count = 0", limit=1000).items
verdicts = client.checks.list_findings(dataset, "graph.evaluate", limit=1000).items
fails = {i.name: i for i in client.checks.findings(dataset, limit=1000).items}

rows = {
    e.name: {"index": e.index, "task": e.task, "state": e.state, "checked": False}
    for e in episodes
}
for f in verdicts:
    row = rows.get(f.episode)
    if row is None:  # a finding on a take that was split since
        continue
    issue = fails.get(f.issue)
    row["checked"] = True
    row["blocks"] = row.get("blocks", False) or bool(
        issue and issue.blocks and issue.state == "OPEN"
    )
    if f.verdict == "FAIL" or (
        f.finding_key == "graph_outcome" and row.get("verdict") != "FAIL"
    ):
        row.update(
            verdict=f.verdict,
            confidence=f.confidence,
            reason=f.explanation,
            judged=f.task_revision,
            moments=[m.time_seconds for m in issue.finding.moments]
            if issue and issue.finding
            else [],
        )
```

`qualia dataset checks findings` lists the findings that stand: per check and
episode, the newest run's at the Task revision the episode links.
`--all-revisions` adds the ones later runs superseded. Each call returns one
page of at most 1000: past that, pass `--cursor`
(`cursor=page.page_info.end_cursor`) for the next. To look at a moment,
`qualia dataset videos <EPISODE> -d "$DATASET" --download ./evidence/ep-7`
saves `<camera>.mp4` for each camera. Give each episode its own folder: a file
of the same name is replaced.

`qualia check status <run>` prints where a run stands once and returns. Run it
again until the run ends, or wait with `--wait` on `qualia check run` or with
`client.checks.wait(run)`. A VLM run that could not read some episodes still
ends `SUCCEEDED`, and its status says so:
`Result: completed, 4 of 30 episodes not checked`, then each of those episodes
with why. Each also gets an open `not_checked` issue. Two kinds of run show the
count alone, with no episode named and no `not_checked` issue: a run made
before runs named what they could not read, and a run that was cancelled.
There the rule above is the one to use: no finding, not checked.

When a VLM run has questions about the episodes it failed, `--wait` and
`qualia check status` print them last, under `Review health:`, each with example
episodes. Look at those episodes before you reject on the run's findings.

See [`qualia dataset checks`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-checks) and
[`qualia check`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-check), which also reads check runs,
reference ranges and the reach check, and [checks](https://junction-docs.qualiastudios.dev/sdk/resources/checks.md).

### 6. Rule

Read what is there, then rule. Rule after the checks: a failure a later run
finds on an episode you kept is only a note. Pick episodes by name, or with the
filter the dataset page's filter bar writes.

```bash
qualia dataset summary "$DATASET" --facet state --facet open_issue_codes
xargs -r qualia dataset keep "$DATASET" --reason "passed the VLM check, nothing open" < keep.txt
qualia dataset reject "$DATASET" --filter 'open_issue_codes:"graph_outcome"' --reason "failed the VLM check"
qualia dataset reject "$DATASET" --filter 'metrics.duration_seconds < 2' --reason "too short"
EPISODE=$(jq -r '.episodes[0].name' episodes.json)
qualia dataset reset "$DATASET" "$EPISODE"    # withdraws a ruling
```

`keep.txt` holds one episode name per line: the rows of your table that passed
and have nothing else open, so their `state` is `KEPT`. A keep or reject that
names no episode is refused: the CLI exits 2, and the SDK raises
`selection.names is empty`. So `xargs -r` runs nothing when the file is empty,
and the Python below skips an empty list. A filter that selects nothing
changes nothing and succeeds.

A keep is yours, and it is what a version takes by default (step 7). It also
closes the blocking issues open on the episode, so never keep by a filter you
have not read. A reject leaves its issues open, as the record of why. The
first filter above rejects what the verdict on the whole task blocks; pass
names, as for `keep`, to reject by your table.

[`keep`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-keep) and
[`reject`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-reject) never delete anything, and
[`reset`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-reset) withdraws them. A take a person
has open in the studio is left as it was: the command lists it and still exits 0. Run it again once they are done.

Issues are settled apart from rulings:

```bash
qualia dataset issues summary "$DATASET"    # counts per code and state
qualia dataset issues list "$DATASET" --filter 'state = OPEN AND blocks = true'
qualia dataset issues dismiss "$DATASET" --filter 'code = "idle_tail"' --reason "fine for this task"
qualia dataset issues add "$DATASET" "Gripper slips at the second grasp" -e "$EPISODE" --at 12.5
```

See [`qualia dataset issues`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-issues-list).

A trim or a split is drawn on the take's draft. The dataset does not change:
only a version published from a preview holds the drawing (step 7).
`qualia dataset trim --clear` draws the full extent again, and
`qualia dataset unsplit` discards a drawn split. `--dry-run` plans a split and
writes nothing. `qualia dataset edit` changes labels, on the dataset itself.

```bash
qualia dataset edit "$EPISODE" -d "$DATASET" --outcome SUCCESS
qualia dataset trim "$EPISODE" -d "$DATASET" --start 1.5 --end 12
qualia dataset split "$EPISODE" -d "$DATASET" --keep 0.5-6 --keep 8-14 --dry-run
```

See [`qualia dataset edit`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-edit),
[`trim`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-trim) and
[`split`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-split).

```python
passed = [
    n for n, r in rows.items() if r.get("verdict") == "PASS" and r["state"] == "KEPT"
]
failed = [n for n, r in rows.items() if r.get("verdict") == "FAIL" and r["blocks"]]
if passed:  # an empty list of names is refused
    client.episodes.keep(dataset, passed, reason="passed the VLM check, nothing open")
if failed:
    client.episodes.reject(dataset, failed, reason="failed the VLM check")

episode = episodes[0].name
client.episodes.reset(dataset, [episode])  # withdraws its ruling
client.issues.dismiss(dataset, filter='code = "idle_tail"', reason="fine for this task")
client.episodes.update(episode, dataset=dataset, outcome="SUCCESS")
```

See [episodes](https://junction-docs.qualiastudios.dev/sdk/resources/episodes.md) and [issues](https://junction-docs.qualiastudios.dev/sdk/resources/issues.md).

### 7. Pin a version

A version freezes a selection. Training and downloads read versions, never the
live dataset. By default a version takes the episodes kept by a person, a key's
keeps included, with the trims and splits saved on them. An episode that is
only kept automatically is left out: keep it yourself, or pass
`--include kept`.

```bash
qualia dataset version-preview "$DATASET" --json > preview.json
jq '.preview | {episodeCount, unprocessable}' preview.json
qualia dataset pin "$DATASET" 1.0 --preview "$(jq -r .preview.name preview.json)" --note "first cut"
qualia dataset version list "$DATASET"
```

The preview freezes and measures the selection and changes nothing. Read how
many episodes it holds and the takes it leaves out, each with why, then pin
exactly that preview. While it leaves a take out, the pin is refused:
`--allow-left-out` (`allow_left_out=True`) accepts that. Each item names its
take in `take`. Its `label` counts from 1, so `episode 8` is the take whose
`index` is 7.

`--as-recorded` and `--digest` tag head instead: every kept episode as
recorded, the automatic keeps too, with saved trims and splits left out.
`--digest` takes the `targetDigest` that `qualia dataset version diff --json`
prints, and is refused if head moved since. A mixture always tags head. Tags
never move. [`qualia dataset version create`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-version-create)
takes the same flags as [`qualia dataset pin`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-pin).

```python
preview = client.versions.preview(dataset)
print(preview.episode_count, preview.unprocessable)
tag = client.versions.create(dataset, "1.0", preview=preview, note="first cut")
```

See [versions](https://junction-docs.qualiastudios.dev/sdk/resources/versions.md) and
[Versions](https://junction-docs.qualiastudios.dev/global/guides/curation.md#versions).

### 8. Download

```bash
qualia dataset download "$DATASET:1.0" ./session-0412-1.0
```

The CLI checks every file against its size and sha256, shows progress on
stderr, and creates the folder only once the whole tree is there. A run that
stops picks up where it left off.

The first download of a version in a format waits while the server renders it.
That takes minutes, one to three for 10 to 24 episodes when measured, behind
one spinner line, or behind nothing with `--json`. Starting a rendering needs a
key that can write to the dataset (`data:write`); a rendered version downloads
with read access (`data:read`). A rendering is not kept for good: a new
renderer starts over, so a version rendered last week can render again today.
`client.versions.get_materialization("<dataset>:1.0", layout="lerobot_v3")`
reads its state (`NOT_STARTED`, `RUNNING`, `SUCCEEDED` or `INTERRUPTED`) and
starts nothing.

`--format` takes `lerobot_v3` (the default), `lerobot_v2_1`, `mcap_ros2`,
`pi_mcap_v1` and `dreamlabs_v2_1` (files to send to Physical Intelligence or
Dream Labs), or `original` (every file uploaded to the dataset, byte for byte;
no version, and an organization key for now). See
[`qualia dataset download`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-download) and
[Download a version](https://junction-docs.qualiastudios.dev/global/guides/download.md).

A LeRobot download holds the version's episodes renumbered from 0, their task
labels, and state and action values equal to the upload's. A handheld (UMI)
recording whose poses were resolved at the TCP is the exception: its
`observation.state` holds the calibrated TCP poses instead, and it has no
action columns
([What a LeRobot download holds](https://junction-docs.qualiastudios.dev/global/guides/download.md#what-a-lerobot-download-holds)).
The download says where each episode came from: `robot_type` in
`meta/info.json` (the upload's own, else `unknown`), and per episode
`qds_episode_id`, `qds_source_dataset_id` (the upload) and
`qds_take_episode_id` (the take a piece was cut from), in
`meta/episodes.jsonl` (v2.1) or `meta/episodes/chunk-000/file-000.parquet`
(v3). It holds no verdict, reason, confidence, moment, decision or subtask:
keep your table beside it. `qds_episode_id` is the last part of an episode's
`name`, and that episode's `index` is its `episode_index` in the folder you
uploaded; a piece of a split take carries its take's as `parentIndex`.

A LeRobot download's video is AV1. A camera recorded in another codec is
encoded again, at the same size and frame rate, so its pixels are close to the
upload's but not equal. For the robot's own pixels, download `original`.

```python
result = client.data.download(f"{dataset}:1.0", "./session-0412-1.0")
```

## What it costs

The VLM check spends credits. The heuristic checks spend none.

- **The estimate is a ceiling, not the price.** A run reserves its estimate
  when it starts and is charged for the calls it made when it ends, never more.
  The rest comes back. A run that fails on the platform's side is refunded in
  full; one you cancel pays for the calls it had made. A run usually settles
  well below its estimate.
- **The balance must cover the ceiling.** A run is refused when the account
  holds fewer credits than it would reserve, even if it would have cost less.
- **A reading is reused when nothing it read has changed**: the same episode,
  not edited since, the same Task revision and the same model, in the same
  account, and an earlier reading that ended well (one that failed or was cut
  off by its cap is read again). A reused reading is not charged again. The
  run still reserves its whole estimate first, and neither the estimate nor
  the run says how many readings were reused.
- **Preparation is not charged.** It reserves and spends none of the
  account's credits. Nothing in the API says so: the balance just does not
  move while it runs.
- **`--max-cost` caps a run you start**: what it may reserve and spend. The
  run that starts by itself sets its own cap, at its estimate.
- **No command shows what one run settled at.** A check run carries no credits.
  `qualia credit --json` (`client.credits.get()`) gives the account's balance:
  read it before the run and after it ends. Other work in the account moves the
  same balance, so the difference is this run's only when nothing else ran.

## Over plain HTTP

Every step but the upload is one or two calls, from any language. Send the key
in the `X-API-Key` header. Everything on the path lives under
`/v1/accounts/{account}`, where `{account}` is the `account_id` that
`GET /v1/whoami` answers; `-` in that place of the path also means the key's
own account. The [qualia.v1 reference](https://junction-docs.qualiastudios.dev/api/qualia-v1.md) has every call and
body.

| Step                         | Call under `/v1/accounts/{account}`                                                                 |
| ---------------------------- | --------------------------------------------------------------------------------------------------- |
| Find a dataset by its slug   | `GET /datasets?filter=slug = "session-0412"`                                                        |
| Preparation                  | `GET /datasets/{dataset}/preparation`                                                               |
| Runs on a dataset            | `GET /checks/-/runs?filter=dataset = "accounts/{account}/datasets/{dataset}"`                       |
| One run; stop it             | `GET /checks/vlm/runs/{run}`; `POST /checks/vlm/runs/{run}:cancel`                                  |
| Create, edit, publish a Task | `POST /tasks`; `PATCH /tasks/{task}`; `POST /tasks/{task}:publish`                                  |
| Link episodes to a Task      | `POST /datasets/{dataset}/episodes:assignTask`                                                      |
| Estimate the VLM check       | `POST /checks/vlm:run` with `{"dataset": "accounts/…/datasets/…", "validateOnly": true}`            |
| Run it                       | the same call with `"maxCost"` and no `validateOnly`                                                |
| Episodes                     | `GET /datasets/{dataset}/episodes?pageSize=1000&filter=child_count = 0`                             |
| Verdicts, passes too         | `GET /datasets/{dataset}/checks/graph.evaluate/findings?pageSize=1000`                              |
| Failures, with moments       | `GET /datasets/{dataset}/issues?filter=check = "accounts/{account}/checks/vlm"`                     |
| Keep, reject                 | `POST /datasets/{dataset}/episodes:keep`; `POST /datasets/{dataset}/episodes:reject`                |
| Preview, then pin            | `POST /datasets/{dataset}/versionPreviews`; `POST /datasets/{dataset}/versionPreviews/{id}:publish` |
| Versions; what one holds     | `GET /datasets/{dataset}/versions`; `GET /datasets/{dataset}/episodes?version={tag}`                |
| Render a version             | `POST /datasets/{dataset}/versions/{tag}/materializations/lerobot-v3:start`                         |
| Its state; its files         | `GET` the same without `:start`; `GET` the same with `/artifacts` in its place                      |
| Balance                      | `GET /v1/credits`, outside the account                                                              |

```bash
ACCOUNT=$(curl -s -H "X-API-Key: $QUALIA_API_KEY" "$QUALIA_BASE_URL/v1/whoami" | jq -r .account_id)
BASE="$QUALIA_BASE_URL/v1/accounts/$ACCOUNT"
curl -sG -H "X-API-Key: $QUALIA_API_KEY" "$BASE/datasets" --data-urlencode 'filter=slug = "session-0412"'
curl -sG -H "X-API-Key: $QUALIA_API_KEY" "$BASE/datasets/$DATASET/episodes" \
  --data-urlencode pageSize=1000 --data-urlencode 'filter=child_count = 0'
curl -s -H "X-API-Key: $QUALIA_API_KEY" -H "Content-Type: application/json" \
  -d "{\"dataset\": \"accounts/$ACCOUNT/datasets/$DATASET\", \"validateOnly\": true}" "$BASE/checks/vlm:run"
```

- A filter is one query parameter, URL-encoded: `slug = "session-0412"`,
  `all_revisions = true` (on findings, for the superseded ones too).
- A list answers one page: ask for `pageSize` and send `nextPageToken` back as
  `pageToken`. The most a page holds differs by list: 1000 for episodes,
  findings, issues and runs, 500 for datasets and versions, 512 for a
  rendering's files. A larger `pageSize` is refused (400), not cut down.
- The episode list holds a take that was split beside its pieces: the take
  has `childCount` above 0, and each piece names it in `parentEpisode`. The
  dataset's `counts` count the rows with `childCount` 0, and
  `filter=child_count = 0` asks for those.
- A count is a JSON string in some fields and a number in others: a dataset's
  `counts.episodes` is `"24"`, a version's `episodeCount` is `"24"` and an
  estimate's `credits` is `"70"`, while a run's `vlm.episodeCount` is `24`.
  Convert every count before you compute with it.
- A body with a field the call does not know is refused (400), so a misspelt
  `validateOnly` starts nothing.
- The preview and the rendering answer at once and work in the background:
  read them again until `READY` (then `PUBLISHED` after the publish), or
  `SUCCEEDED`. Each file in the artifact list has a signed `uri`, its `path`
  and its `sha256`.
- Errors come in two shapes: the API's
  `{"code": "NOT_FOUND", "message": "…", "retryable": false, "details": []}`
  and the gateway's `{"detail": "…"}`, for a missing key or an unknown route.
  Read `code` if the body has one, else `detail`.

An upload is many calls that hash, split and resume files. Use the CLI or the
SDK for that step.

## A new batch every day

Several uploads become one dataset by a merge. A merge takes at least two
uploads, so the first batch is an upload on its own, the second one makes the
merged dataset, and each later one is appended to it:

```bash
qualia dataset upload ./batch-1 batch-1 --standalone --json > day1.json
qualia dataset upload ./batch-2 batch-2 --standalone --json > day2.json
ONE=$(jq -r .dataset_id day1.json); TWO=$(jq -r .dataset_id day2.json)
qualia dataset merge "$ONE" "$TWO" --name shelf-restock --dry-run
MERGED=$(qualia dataset merge "$ONE" "$TWO" --name shelf-restock --yes --json | jq -r .name)
qualia dataset upload ./batch-3 batch-3 --merge-target "$MERGED" --json > day3.json
jq .merge_append day3.json
```

- **A merge is final.** It cannot be undone as a whole, and the uploads become
  the merged dataset's storage. `--dry-run` lists what differs between them
  first. Uploads merge only when they record the same cameras, features,
  channel names and frame rate, with poses in the same place (root or TCP).
- **After the merge, work on the merged dataset.** Rulings, issues, edits and
  task links go through it and land in the upload that holds the episode.
  Checks, versions and downloads are the merged dataset's own. A write that
  names an upload a merge took is refused. What you ruled before the merge
  reads the same after it, and a version tagged on an upload keeps resolving
  under the upload's own name.
- **`merge_append` in the receipt says whether the append happened**:
  `appended`, or `action_needed` with why. Then append it yourself:
  `qualia dataset merge <UPLOAD> --into "$MERGED" --yes`.
- **The merged dataset is a new dataset.** It has its own id and slug, and
  its own tags: a version pinned on the first upload stays under that upload's
  name (`batch-1:1.0`), and `version list` on the merged dataset does not show
  it.
- **Each appended upload is prepared on its own**, where uploads are prepared:
  `qualia dataset preparation <UPLOAD>` reads it, with the `dataset_id` of its
  receipt. The automatic checks that follow run on the merged dataset as a
  whole. The paid one takes every linked episode in it, the old batches too,
  and reserves for all of them: 111 credits for 30 episodes of which 15 were
  new, in one case. What it can reuse is not charged again, but its estimate
  does not show that. To pay for the new batch only, cancel that run and start
  your own on the new upload's episodes, as below.
- **Verdicts made before the merge are listed under the upload.**
  `dataset checks findings` through the merged dataset lists only what runs
  on the merged dataset found, so a batch checked while it was still an upload
  looks unchecked there. Read the findings of each upload the merged dataset
  holds (its `sourceDatasets`) instead: that lists every verdict on the
  upload's episodes, whichever run made it. An episode keeps the last part of
  its name through the merge, so join on that. Issues, with their moments, do
  read through the merged dataset.
- **Tell the batches apart by `upload`.** Every episode names the upload it
  came in with, and a filter takes it:
  `--filter 'upload = "accounts/<ACCOUNT>/datasets/<UPLOAD>"'`. Its `index`
  counts inside its upload, as the folder did; `position` counts across the
  merged dataset. A download names it as `qds_source_dataset_id`.
- **A new version takes what is kept now**, the new batch's kept episodes with
  the old. `qualia dataset version diff "$MERGED"` shows what came in since the
  newest tag. Earlier tags do not change.

```bash
NEW="${MERGED%/*}/$(jq -r .dataset_id day3.json)"   # the new upload's whole name
qualia dataset assign-task -d "$MERGED" --task "$TASK" --label "pour coffee into the cup" \
  --filter "upload = \"$NEW\" AND task = \"pour coffee into the cup\""   # day one's Task
qualia check estimate "$MERGED" --filter "upload = \"$NEW\""
qualia check run "$MERGED" --filter "upload = \"$NEW\"" --max-cost 60 --wait
for UPLOAD in $(qualia dataset get "$MERGED" --json | jq -r '.sourceDatasets[]'); do
  qualia dataset checks findings "$UPLOAD" --check graph.evaluate -n 1000 --json
done | jq -s '[.[].checkFindings[]]' > verdicts.json   # every verdict of every batch
```

So each day is the same path with `$MERGED` as the dataset, with four
differences:

- Link only the new episodes, and to the Tasks of the first day. Keep each
  Task's name (`$TASK`; `qualia task list` lists them) and put the new upload
  in the filter, as above. `assign-task` moves every episode it selects that
  links another Task. So a new Task, or a filter without `upload`, also moves
  the old batches' episodes, and the verdicts they had against the old Task
  no longer stand.
- Wait for the preparation of the new upload, then for the runs on `$MERGED`:
  `prepared "$NEW"`, then `followed "$MERGED"` with that preparation's
  `createTime` and `heuristic vlm` (step 5).
- Check the new batch with a filter if you will not pay to reserve for the
  old ones.
- Read the verdicts upload by upload.

Then rule, pin the next tag and download.

`qualia dataset destination set "$MERGED"` saves the merged dataset for later
uploads without `--merge-target`. The saved destination applies only when the
CLI runs against the same base URL and workspace and is given the account
(`QUALIA_ACCOUNT`). With only a key and a base URL in the environment it is
not used, so a script is surer with the flag.

```python
day1 = client.data.upload("./batch-1", name="batch-1", standalone=True)
day2 = client.data.upload("./batch-2", name="batch-2", standalone=True)
merged = client.datasets.merge(
    [day1["dataset_id"], day2["dataset_id"]], display_name="shelf-restock"
)
day3 = client.data.upload("./batch-3", name="batch-3", merge_target=merged.name)
print(day3["merge_append"])

new = f"{merged.name.rsplit('/', 1)[0]}/{day3['dataset_id']}"  # the new upload's whole name
client.episodes.assign_task(
    merged.name,
    "pour coffee into the cup",
    filter=f'upload = "{new}" AND task = "pour coffee into the cup"',
    linked_task=task.name,  # day one's Task
)
started = client.checks.run(merged.name, filter=f'upload = "{new}"', max_cost=60)
for run in started.check_runs:  # run() returns at once: wait before reading
    if client.checks.wait(run, timeout=3600).state != "SUCCEEDED":
        raise RuntimeError(f"{run} did not succeed")  # its findings were not written
verdicts = [
    f
    for upload in client.datasets.get(merged.name).source_datasets
    for f in client.checks.list_findings(upload, "graph.evaluate", limit=1000).items
]
```

See [`qualia dataset merge`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-merge) and
[datasets](https://junction-docs.qualiastudios.dev/sdk/resources/datasets.md).

## Robots and grippers

Curation needs no robot. Set one up when the dataset is played on it:

```bash
ROBOT=$(qualia robot create lab-arm --urdf lab_arm.zip --json | jq -r .name)
qualia robot model "$ROBOT"
qualia robot configure "$ROBOT" --solver dls
GRIPPER=$(qualia gripper create parallel_gripper.urdf --json | jq -r .name)
qualia gripper tcp "$GRIPPER" tcp.json
qualia robot carry "$ROBOT" "$GRIPPER"
qualia dataset update "$DATASET" --target-robot "$ROBOT"
```

[`qualia robot create`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-robot-create) with `--urdf`
imports the file and waits for it.
[`qualia robot model`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-robot-model) shows the arms it
guessed, and [`qualia robot configure`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-robot-configure)
corrects them. [`qualia gripper tcp`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-gripper-tcp) says
where the tool centre point is.
[`qualia robot carry`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-robot-carry) gives the robot its own
copy of the gripper, so change the copy from then on.
[`qualia dataset update`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-dataset-update) picks the robot
the dataset plays on.

```python
robot = client.robots.create_from_urdf("lab_arm.zip")
model = client.robots.get_model(robot.name)
client.robots.update_model(robot.name, solver="dls", etag=model.etag)

gripper = client.robots.create_from_urdf("parallel_gripper.urdf", kind="gripper")
client.robots.set_gripper_tcp(gripper.name, file="tcp.json")
client.robots.set_gripper(robot.name, gripper.name)
client.datasets.update(dataset, target_robot=robot.name)
```

See [robots](https://junction-docs.qualiastudios.dev/sdk/resources/robots.md).

## Find your way around

| To see                   | CLI                                                           | SDK                               |
| ------------------------ | ------------------------------------------------------------- | --------------------------------- |
| Workspaces               | `qualia project list`                                         | `client.projects.list()`          |
| One workspace            | `qualia project show`                                         | `client.projects.get(project_id)` |
| What people and keys did | [`qualia activity list`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-activity-list) | `client.activity.list()`          |
| Running work             | [`qualia status list`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-status-list)     | `client.statuses.list()`          |
| Stop running work        | [`qualia status cancel`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-status-cancel) | `client.statuses.cancel(name)`    |

```bash
qualia activity list --org --filter 'verb = KEEP AND actor = API_KEY' --json
qualia status list --running --json
```

## Admin

Admin work needs an opt-in scope ticked when the key was made. The scope opens
the door. The role of the key's creator still decides: a Contributor's key with
`members:write` cannot change a role, and no key grants a role above its
creator's own. A guest's key can do none of this. Activity records each write
with the key's name. `credits:write` is an ordinary scope that every new key
holds.

- **People and roles** (`members:read`, `members:write`):
  [`qualia member`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-member) `list`, `show`, `role`,
  `remove`; [`client.members`](https://junction-docs.qualiastudios.dev/sdk/resources/members.md) `list`, `get`,
  `set_role`, `remove`.
- **Invitations** (`members:read`, `members:write`):
  [`qualia invitation`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-invitation) `list`, `send`,
  `withdraw`; [`client.invitations`](https://junction-docs.qualiastudios.dev/sdk/resources/members.md) `list`, `send`,
  `withdraw`.
- **Integration keys** (`integrations:write`):
  [`qualia integration`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-integration) `list`, `add`,
  `remove`, `default`; [`client.integrations`](https://junction-docs.qualiastudios.dev/sdk/resources/integrations.md)
  `list`, `add`, `remove`, `set_default`.

An integration key's value goes in once and is never shown again, so pipe it in
and keep it out of your shell history:

```bash
printf %s "$HF_TOKEN" | qualia integration add huggingface --name ci --secret-stdin
```

## What a key cannot do

These stay in the dashboard:

- **Log in or sign up.** A person makes the key.
- **Reset a password.** That is the person's.
- **Change appearance.** Theme is a browser setting.
- **See presence.** Who is on a page is shown to signed-in people.
- **Hold an edit lease.** A person editing a take blocks a key's write to it.
- **Create or delete an organization.** A person does that.

## Try the whole path

`sdk/cli/examples/agent_path.sh`, in the CLI's source, runs most of these
commands with `--json` against the stack named by `QUALIA_BASE_URL` and
`QUALIA_API_KEY`: an upload, a robot and gripper, a task, checks, rulings, a
version and an export. It makes its own data, prints a line per step, and exits
non-zero if a step fails. Where a stack cannot finish a step, it says so.

On exit it deletes what it made and cancels an upload it did not finish. An
upload's repository and stored bytes stay, because no command deletes them. The
script prints a line for each one.