Checks

client.checks runs the checks the dataset page runs and reads what they found. A check never keeps, rejects or deletes an episode: its findings are issues (confident failures) and notes (uncertain readings).

The default check is the VLM check (vlm): a model reads each episode against the Task it is linked to. It costs credits; estimate shows the price and starts nothing. The catalog’s checks (heuristic.jerk, task.graph, …) run as one selection of one kind with check_ids, categories or all.

Checks speak qualia.v1, in the API key’s own account. Run and findings methods take a dataset id or its full name (accounts/{account}/datasets/{dataset}). Automatic policy methods also accept an organization or workspace id/name.

Automatic checks on new uploads

Free quality (heuristic) checks are on by default; video/graph checks use credits and are off by default. At each level, a half can inherit or be explicitly enabled or disabled. Each half resolves separately: dataset → workspace → organization → platform. The returned heuristic_source and graph_source name the level supplying its effective value; heuristic and graph are the override at the requested level (SETTING_UNSPECIFIED means inherit). No policy rows are created for existing organizations, workspaces or datasets at deployment.

# Inspect the effective values and where they came from.
org = client.checks.automatic_policy("<ORG_UUID>", org=True)
workspace = client.checks.automatic_policy("<WORKSPACE_UUID>", workspace=True)
dataset = client.checks.automatic_policy("<DATASET_UUID>")
print(dataset.heuristic_enabled, dataset.heuristic_source)
print(dataset.graph_enabled, dataset.graph_source)

# An organization's Admin or Owner may opt in to paid video checks.
client.checks.set_automatic_policy("<ORG_UUID>", org=True, graph=True)
# A workspace Admin or Owner may override only video, leaving quality alone.
client.checks.set_automatic_policy("<WORKSPACE_UUID>", workspace=True, graph=False)
# A dataset writer may inherit that workspace value again.
client.checks.set_automatic_policy("<DATASET_UUID>", reset=("graph",))

set_automatic_policy(parent, *, org=False, workspace=False, heuristic=None, graph=None, reset=()) changes only named halves: True is on, False off; reset=("heuristic", "graph") removes both explicit overrides. Never pass org=True and workspace=True together. A switch starts no checks and never backfills old episodes: Applies to new uploads. Run checks to check existing episodes. A person’s run remains available when automatic checks are off. Use estimate before a manual paid run to see its credit bound.

The public Checks API provides GET and masked PATCH on the singleton checkPolicies/automatic under each parent:

Parent	GET/PATCH resource name
Organization	/v1/accounts/{account}/checkPolicies/automatic
Workspace	/v1/accounts/{account}/workspaces/{workspace}/checkPolicies/automatic
Dataset	/v1/accounts/{account}/datasets/{dataset}/checkPolicies/automatic

GetAutomaticCheckPolicy returns effective heuristicEnabled, graphEnabled, heuristicSource, graphSource and the requested level’s heuristic / graph overrides. UpdateAutomaticCheckPolicy PATCH takes the policy body and updateMask=heuristic, graph, or both (comma-separated). Select a half in the mask and set it to ENABLED, DISABLED or SETTING_UNSPECIFIED (inherit); leaving a half out of the mask does not change it. A dataset read/write takes data:read / data:write; changing a workspace or organization default takes an Admin or Owner at that level and a key with opt-in members:write scope. A key bound to a workspace cannot change the organization’s default.

The VLM check

from qualia import Qualia

client = Qualia()

estimate = client.checks.estimate(dataset_id, max_episodes=50)
print(estimate.credits, estimate.episode_count, estimate.models)

started = client.checks.run(dataset_id, max_episodes=50, max_cost=estimate.credits)
run = client.checks.wait(started.check_runs[-1])   # SUCCEEDED or FAILED
for issue in client.checks.findings(dataset_id).items:
    # An episode the run could not read has no reading: code "not_checked".
    verdict = issue.finding.verdict if issue.finding else issue.code
    print(issue.episode, verdict, issue.text)

While another VLM run is on the dataset (the automatic one after preparation, or anyone’s), run queues the new run behind it instead of starting it: started.queued_behind names the run ahead and started.queued_reason says so in a sentence. The queued run starts when that one ends and reserves credits only then; wait covers the wait. estimate answers either way.

A run that could not read some episodes (no published video, a refused call, an invalid answer) still ends SUCCEEDED. It says so, and names each episode with why:

if run.vlm.not_checked_summary:          # "completed, 4 of 30 episodes not checked"
    print(run.vlm.not_checked_summary)
    for take in run.vlm.unread_episodes:  # the first 100
        print(take.episode, take.reason)

Each of those episodes also has an open issue, code not_checked, so it is no longer kept. The next run that reads the episode closes it.

estimate.models lists the Qualia model tiers the check offers, the default first. The VLM check reads each episode’s task graph on qualia-vlm (the strongest) only, so it lists that one. Another tier passed as model= is refused (400) before anything starts.

check="reach" runs the reach solver instead; check="-" runs the dataset’s default set (episode-rules on an upload, mixture on a mixture).

Is the VLM check set up on this deployment?

client.checks.get("vlm").runnable

Checks from the catalog

for check in client.checks.list(dataset_id).items:
    print(check.check_id, check.kind, check.runnable, check.unavailable_reason)

bound = client.checks.estimate(dataset_id, kind="heuristic", categories=["heuristic"])
started = client.checks.run(dataset_id, kind="heuristic", all=True)
run = client.checks.wait(started.check_run)

page = client.checks.list_findings(dataset_id, verdict="FAIL")
for finding in page.items:
    print(finding.episode, finding.title, finding.severity)

Each catalog check has a kind. Heuristic checks read the episode’s data with no model and cost nothing. VLM checks have a model read it, one call per episode for every selected VLM check, and cost credits. One run holds one kind: kind ("heuristic" or "vlm") pins the kind categories and all take. Without it this deployment selects runnable heuristics; VLM catalog checks are unavailable. It lists skipped checks and why. If an explicitly pinned kind takes nothing, the ValidationError suggests the other runnable kind, if any.

A VLM run needs max_cost covering the bound estimate gives for the same selection; a heuristic run’s bound is 0, so it needs none. run sends a request_id, so a retry with the same id returns the first run.

list_findings(dataset, check="-", *, check_run=, episode=, verdict=, all_revisions=) reads one check’s findings ("task.graph") or every check’s: "-" lists VLM and recording (heuristic) findings together, newest first. It returns the findings that stand: per check and episode, the newest run’s at the episode’s linked Task revision (its own link, else its dataset’s), each with the task_revision it read. Publishing a newer revision leaves them standing; after datasets.relink_task(), an episode has none until a run reads the new revision. all_revisions=True returns every run’s findings, superseded ones included.

Recording (heuristic) and graph check findings, and VLM verdicts below their confidence bar, open notes on their episodes, not blocking issues. Filter episodes by how many they carry with note_count, as open_issue_count counts issues:

client.episodes.list(dataset_id, filter="note_count > 0")

Reference ranges

The episode rules judge each episode against its robot’s reference ranges: one per robot interface (or robot) and metric, with the fences an episode is judged against.

page = client.checks.reference_ranges(interface_hash="b3:…", metric="duration_seconds")
r = page.items[0]
print(r.metric, r.fence_low, r.fence_high, r.sample_count)

The heuristic check judges a merged dataset or a mixture against the ranges of the interface its uploads share, and of their robot when they share one. While its uploads do not share one interface (a merged dataset’s are still settling to one) it has no ranges, and the check reports jerk, noise and velocity as unassessed.

reference_range(name) reads one.

Reach

status = client.checks.reach_status(dataset_id)
print(status.target_robot, status.current_key, status.stale)
if status.last_run:
    print(status.last_run.state, status.last_run.done, status.last_run.total)

client.checks.run(dataset_id, "reach")        # the solver; queued, not waited for
client.checks.cancel_reach(dataset_id)        # stop it; solved takes keep results

for result in client.checks.results(dataset_id, state="SUCCEEDED").items:
    print(result.episode, result.reach.longest_seconds, len(result.reach.stretches))

results(dataset, check="reach", *, episode=, state=, robot=, filter=) lists each take on the dataset’s target robot at its current reach key, NOT_RUN where nothing is solved under it yet. Pass None for every stored result in your workspace.

Methods

Method	Call	Returns
automatic_policy(parent, *, org=False, workspace=False)	GetAutomaticCheckPolicy	AutomaticCheckPolicy
set_automatic_policy(parent, *, org=False, workspace=False, heuristic=None, graph=None, reset=())	UpdateAutomaticCheckPolicy	AutomaticCheckPolicy
get(check="vlm")	GetCheck	Check
estimate(dataset, *, check_ids=, categories=, all=, kind=, episodes=, filter=, model=, max_episodes=, max_cost=)	RunCheck or RunDatasetChecks, validate_only	CheckEstimate, or RunDatasetChecksResponse for a catalog selection
run(dataset, check="vlm", *, …same…, request_id=)	RunCheck or RunDatasetChecks	RunCheckResponse or RunDatasetChecksResponse
get_run(run), wait(run, *, timeout=, interval=)	GetCheckRun	CheckRun
list_runs(check="-", *, dataset=, state=, filter=)	ListCheckRuns	page of CheckRun
list(dataset)	ListDatasetChecks	page of DatasetCheck
list_findings(dataset, check="-", *, check_run=, episode=, verdict=, all_revisions=)	ListCheckFindings	page of CheckFinding
findings(dataset, check="vlm", *, filter=)	Issues.ListIssues	page of Issue
reference_ranges(check="episode-rules", *, interface_hash=, robot=, metric=, filter=)	ListReferenceRanges	page of ReferenceRange
reference_range(name)	GetReferenceRange	ReferenceRange
reach_status(dataset)	GetReachStatus	ReachStatus
cancel_reach(dataset)	CancelReachStatus	ReachStatus
results(dataset, check="reach", *, episode=, state=, robot=, filter=)	ListCheckResults	page of CheckResult

Pages take limit= and cursor= and return items with page_info.end_cursor. A key restricted to workspaces reaches none of these; the CLI is qualia check.