Manage Reviewer Feedback and Task Checklists

This guide is the operating loop for the two optional quality tabs on a live job. It assumes a claimed employer token with canonical access to the job and the dedicated Reviewer Feedback and Task Checklist scopes.

1. Discover the live capabilities

opentrain capabilities
opentrain feedback settings --job-id <job-id>
opentrain checklists capabilities --job-id <job-id>

Do not confuse the account rollout with the job setting. The account can report an operation family as available while the feature remains off for this job.

2. Prepare checklists while the tab is off

Keep Task Checklists disabled while building. Create separate definitions for distinct work—for example, Initial Annotation and Quality Control—rather than putting every role into one large conditional tree.

For each definition:

1. Scaffold or check out the local bundle.
2. Edit the typed tree with stable node and option IDs.
3. Validate the complete bundle on the server.
4. Resolve representative answers for every branch and repeat count.
5. Push the draft with its version guard.
6. Open the exact-revision browser preview at desktop and narrow widths.
7. Publish only the revision you reviewed.

opentrain checklists validate --dir ./initial-annotation
opentrain checklists push --dir ./initial-annotation --job-id <job-id>
opentrain checklists preview --dir ./initial-annotation --source draft --open
opentrain checklists publish --dir ./initial-annotation --confirm-live

Repeat the workflow for the QC definition, then list and order both. Publish impact is operational evidence: if a new version marks open-run answers stale, plan for the affected AI trainers to reconfirm those fields.

3. Enable the checklist tab

After all worker-visible definitions are published and ordered:

opentrain checklists enable \
  --job-id <job-id> \
  --key checklist-enable-<job-id>-v1

AI trainers and reviewers can now choose among the published definitions in the job’s Checklist tab. A required definition requires an external task ID and a complete valid answer set before submission.

4. Enable Reviewer Feedback and assign reviewers

Reviewer Feedback starts with no implied reviewer audience. Enable the job feature, read the active-contract candidate list, and grant only the people who should file reports:

opentrain feedback enable \
  --job-id <job-id> \
  --confirm \
  --key feedback-enable-<job-id>-v1

opentrain feedback access list --job-id <job-id>

opentrain feedback access bulk \
  --job-id <job-id> \
  --user-ids <qc-user-1>,<qc-user-2> \
  --grant \
  --key feedback-grant-qc-v1

The access list is intentionally bounded to people with active contracts on this job and includes their current grant state. A bulk grant is all-or-nothing; resolve an ineligible target instead of accepting a partial audience.

5. Review reports as an employer

Use summary for the management view and list for the evidence rows:

opentrain feedback summary --job-id <job-id>
opentrain feedback list --job-id <job-id> --status OPEN --page 1

Investigate one report by reading its body, canonical subject, external IDs, severity, suggested action, reporter, and append-only history. A suggested action is a recommendation, not an automatic personnel decision. Verify the canonical work evidence and applicable job policy before warning, retraining, or offboarding anyone.

Move the report to IN_REVIEW while investigating. Finish with ACTIONED when a concrete response has been taken, or DISMISSED when the report should not drive action. Record the reason in the required note.

opentrain feedback get --report-id <report-id>

opentrain feedback triage \
  --report-id <report-id> \
  --to-status ACTIONED \
  --expected-updated-at <latest-updatedAtIso> \
  --note 'Assigned focused retraining and paused production work pending completion' \
  --key feedback-actioned-<report-id>-v1

6. Review checklist submissions

Use the submission list to find all results, one checklist, one AI trainer, one external task ID, or one published version:

opentrain checklists submissions \
  --job-id <job-id> \
  --definition-id <definition-id> \
  --page 1

Fetch the full record before drawing a conclusion:

opentrain checklists submissions --submission-id <submission-id>

Interpret the result against its pinned version, not the current draft. Conditional and repeated paths that were inactive at submission are represented by audit metadata without exposing discarded raw values.

7. Evolve a live checklist safely

Check out the latest draft before editing:

opentrain checklists checkout \
  --definition-id <definition-id> \
  --dir ./initial-annotation \
  --force

After editing, run diff, validate, push, and preview again. Reordering a node without changing its meaning preserves active answers. Rewording an answerable node or changing its type, options, constraint, required mode, repeat rule, or visibility logic requires reconfirmation in open runs.

Never treat a local file as the current server revision after a conflict. Check out again, reconcile deliberately, and repeat visual review.

8. Disable without destroying history

Both features can be hidden from the job without deleting their records:

opentrain checklists disable \
  --job-id <job-id> --confirm --key checklist-disable-<job-id>-v1

opentrain feedback disable \
  --job-id <job-id> --confirm --key feedback-disable-<job-id>-v1

Checklist definitions, versions, and submissions remain stored. Feedback reports, event history, subject links, and reviewer grants also remain stored for a later re-enable.

Related

CLI reference

Every Reviewer Feedback and Task Checklist command and guard.

HTTP reference

Endpoint map, request examples, schemas, and error recovery.

MCP reference

The matching local and hosted MCP tools.

Concepts

Privacy, adaptive trees, versioning, and agent-surface safety.