# Checks > Run checks, read their findings, thresholds and reach `client.checks` runs the checks the dataset page runs and reads what they found. A check never keeps, rejects or deletes an episode: its findings are issues (confident failures) and notes (uncertain readings). The default check is the VLM check (`vlm`): a model reads each episode against the Task it is linked to. It costs credits; `estimate` shows the price and starts nothing. The catalog's checks (`heuristic.jerk`, `task.graph`, …) run as one selection of one `kind` with `check_ids`, `categories` or `all`. Checks speak qualia.v1, in the API key's own account. Run and findings methods take a dataset id or its full name (`accounts/{account}/datasets/{dataset}`). Automatic policy methods also accept an organization or workspace id/name. ## Automatic checks on new uploads Free quality (heuristic) checks are **on** by default; video/graph checks use credits and are **off** by default. At each level, a half can inherit or be explicitly enabled or disabled. Each half resolves separately: **dataset → workspace → organization → platform**. The returned `heuristic_source` and `graph_source` name the level supplying its effective value; `heuristic` and `graph` are the override at the requested level (`SETTING_UNSPECIFIED` means inherit). No policy rows are created for existing organizations, workspaces or datasets at deployment. ```python # Inspect the effective values and where they came from. org = client.checks.automatic_policy("", org=True) workspace = client.checks.automatic_policy("", workspace=True) dataset = client.checks.automatic_policy("") print(dataset.heuristic_enabled, dataset.heuristic_source) print(dataset.graph_enabled, dataset.graph_source) # An organization's Admin or Owner may opt in to paid video checks. client.checks.set_automatic_policy("", org=True, graph=True) # A workspace Admin or Owner may override only video, leaving quality alone. client.checks.set_automatic_policy("", workspace=True, graph=False) # A dataset writer may inherit that workspace value again. client.checks.set_automatic_policy("", reset=("graph",)) ``` `set_automatic_policy(parent, *, org=False, workspace=False, heuristic=None, graph=None, reset=())` changes only named halves: `True` is on, `False` off; `reset=("heuristic", "graph")` removes both explicit overrides. Never pass `org=True` and `workspace=True` together. A switch starts **no** checks and never backfills old episodes: **Applies to new uploads. Run checks to check existing episodes.** A person's `run` remains available when automatic checks are off. Use `estimate` before a manual paid run to see its credit bound. The public `Checks` API provides GET and masked PATCH on the singleton `checkPolicies/automatic` under each parent: | Parent | GET/PATCH resource name | | ------------ | ----------------------------------------------------------------------- | | Organization | `/v1/accounts/{account}/checkPolicies/automatic` | | Workspace | `/v1/accounts/{account}/workspaces/{workspace}/checkPolicies/automatic` | | Dataset | `/v1/accounts/{account}/datasets/{dataset}/checkPolicies/automatic` | `GetAutomaticCheckPolicy` returns effective `heuristicEnabled`, `graphEnabled`, `heuristicSource`, `graphSource` and the requested level's `heuristic` / `graph` overrides. `UpdateAutomaticCheckPolicy` PATCH takes the policy body and `updateMask=heuristic`, `graph`, or both (comma-separated). Select a half in the mask and set it to `ENABLED`, `DISABLED` or `SETTING_UNSPECIFIED` (inherit); leaving a half out of the mask does not change it. A dataset read/write takes `data:read` / `data:write`; changing a workspace or organization default takes an Admin or Owner at that level and a key with opt-in `members:write` scope. A key bound to a workspace cannot change the organization's default. ## The VLM check ```python from qualia import Qualia client = Qualia() estimate = client.checks.estimate(dataset_id, max_episodes=50) print(estimate.credits, estimate.episode_count, estimate.models) started = client.checks.run(dataset_id, max_episodes=50, max_cost=estimate.credits) run = client.checks.wait(started.check_runs[-1]) # SUCCEEDED or FAILED for issue in client.checks.findings(dataset_id).items: # An episode the run could not read has no reading: code "not_checked". verdict = issue.finding.verdict if issue.finding else issue.code print(issue.episode, verdict, issue.text) ``` While another VLM run is on the dataset (the automatic one after preparation, or anyone's), `run` queues the new run behind it instead of starting it: `started.queued_behind` names the run ahead and `started.queued_reason` says so in a sentence. The queued run starts when that one ends and reserves credits only then; `wait` covers the wait. `estimate` answers either way. A run that could not read some episodes (no published video, a refused call, an invalid answer) still ends `SUCCEEDED`. It says so, and names each episode with why: ```python if run.vlm.not_checked_summary: # "completed, 4 of 30 episodes not checked" print(run.vlm.not_checked_summary) for take in run.vlm.unread_episodes: # the first 100 print(take.episode, take.reason) ``` Each of those episodes also has an open issue, code `not_checked`, so it is no longer kept. The next run that reads the episode closes it. `estimate.models` lists the Qualia model tiers the check offers, the default first. The VLM check reads each episode's task graph on `qualia-vlm` (the strongest) only, so it lists that one. Another tier passed as `model=` is refused (400) before anything starts. `check="reach"` runs the reach solver instead; `check="-"` runs the dataset's default set (`episode-rules` on an upload, `mixture` on a mixture). Is the VLM check set up on this deployment? ```python client.checks.get("vlm").runnable ``` ## Checks from the catalog ```python for check in client.checks.list(dataset_id).items: print(check.check_id, check.kind, check.runnable, check.unavailable_reason) bound = client.checks.estimate(dataset_id, kind="heuristic", categories=["heuristic"]) started = client.checks.run(dataset_id, kind="heuristic", all=True) run = client.checks.wait(started.check_run) page = client.checks.list_findings(dataset_id, verdict="FAIL") for finding in page.items: print(finding.episode, finding.title, finding.severity) ``` Each catalog check has a kind. Heuristic checks read the episode's data with no model and cost nothing. VLM checks have a model read it, one call per episode for every selected VLM check, and cost credits. One run holds one kind: `kind` (`"heuristic"` or `"vlm"`) pins the kind `categories` and `all` take. Without it this deployment selects runnable heuristics; VLM catalog checks are unavailable. It lists skipped checks and why. If an explicitly pinned kind takes nothing, the `ValidationError` suggests the other runnable kind, if any. A VLM run needs `max_cost` covering the bound `estimate` gives for the same selection; a heuristic run's bound is 0, so it needs none. `run` sends a `request_id`, so a retry with the same id returns the first run. `list_findings(dataset, check="-", *, check_run=, episode=, verdict=, all_revisions=)` reads one check's findings (`"task.graph"`) or every check's: `"-"` lists VLM and recording (heuristic) findings together, newest first. It returns the findings that stand: per check and episode, the newest run's at the episode's linked Task revision (its own link, else its dataset's), each with the `task_revision` it read. Publishing a newer revision leaves them standing; after `datasets.relink_task()`, an episode has none until a run reads the new revision. `all_revisions=True` returns every run's findings, superseded ones included. Recording (heuristic) and graph check findings, and VLM verdicts below their confidence bar, open notes on their episodes, not blocking issues. Filter episodes by how many they carry with `note_count`, as `open_issue_count` counts issues: ```python client.episodes.list(dataset_id, filter="note_count > 0") ``` ## Reference ranges The episode rules judge each episode against its robot's reference ranges: one per robot interface (or robot) and metric, with the fences an episode is judged against. ```python page = client.checks.reference_ranges(interface_hash="b3:…", metric="duration_seconds") r = page.items[0] print(r.metric, r.fence_low, r.fence_high, r.sample_count) ``` The heuristic check judges a merged dataset or a mixture against the ranges of the interface its uploads share, and of their robot when they share one. While its uploads do not share one interface (a merged dataset's are still settling to one) it has no ranges, and the check reports jerk, noise and velocity as unassessed. `reference_range(name)` reads one. ## Reach ```python status = client.checks.reach_status(dataset_id) print(status.target_robot, status.current_key, status.stale) if status.last_run: print(status.last_run.state, status.last_run.done, status.last_run.total) client.checks.run(dataset_id, "reach") # the solver; queued, not waited for client.checks.cancel_reach(dataset_id) # stop it; solved takes keep results for result in client.checks.results(dataset_id, state="SUCCEEDED").items: print(result.episode, result.reach.longest_seconds, len(result.reach.stretches)) ``` `results(dataset, check="reach", *, episode=, state=, robot=, filter=)` lists each take on the dataset's target robot at its current reach key, `NOT_RUN` where nothing is solved under it yet. Pass `None` for every stored result in your workspace. ## Methods | Method | Call | Returns | | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------- | ---------------------------------------------------------------------- | | `automatic_policy(parent, *, org=False, workspace=False)` | `GetAutomaticCheckPolicy` | `AutomaticCheckPolicy` | | `set_automatic_policy(parent, *, org=False, workspace=False, heuristic=None, graph=None, reset=())` | `UpdateAutomaticCheckPolicy` | `AutomaticCheckPolicy` | | `get(check="vlm")` | `GetCheck` | `Check` | | `estimate(dataset, *, check_ids=, categories=, all=, kind=, episodes=, filter=, model=, max_episodes=, max_cost=)` | `RunCheck` or `RunDatasetChecks`, `validate_only` | `CheckEstimate`, or `RunDatasetChecksResponse` for a catalog selection | | `run(dataset, check="vlm", *, …same…, request_id=)` | `RunCheck` or `RunDatasetChecks` | `RunCheckResponse` or `RunDatasetChecksResponse` | | `get_run(run)`, `wait(run, *, timeout=, interval=)` | `GetCheckRun` | `CheckRun` | | `list_runs(check="-", *, dataset=, state=, filter=)` | `ListCheckRuns` | page of `CheckRun` | | `list(dataset)` | `ListDatasetChecks` | page of `DatasetCheck` | | `list_findings(dataset, check="-", *, check_run=, episode=, verdict=, all_revisions=)` | `ListCheckFindings` | page of `CheckFinding` | | `findings(dataset, check="vlm", *, filter=)` | `Issues.ListIssues` | page of `Issue` | | `reference_ranges(check="episode-rules", *, interface_hash=, robot=, metric=, filter=)` | `ListReferenceRanges` | page of `ReferenceRange` | | `reference_range(name)` | `GetReferenceRange` | `ReferenceRange` | | `reach_status(dataset)` | `GetReachStatus` | `ReachStatus` | | `cancel_reach(dataset)` | `CancelReachStatus` | `ReachStatus` | | `results(dataset, check="reach", *, episode=, state=, robot=, filter=)` | `ListCheckResults` | page of `CheckResult` | Pages take `limit=` and `cursor=` and return `items` with `page_info.end_cursor`. A key restricted to workspaces reaches none of these; the CLI is [`qualia check`](https://junction-docs.qualiastudios.dev/cli/commands.md#qualia-check).