Reviewer Feedback and Task Checklists

OpenTrain provides two optional, job-scoped quality workflows:

• Reviewer Feedback records private reports about an AI trainer’s work and understanding. Employers can review, triage, and act on those reports.
• Task Checklists give AI trainers and reviewers repeatable, adaptive steps to complete for each external task. Employers author multiple checklists, publish versioned changes, and review the resulting submissions.

Both features are off for each job until an employer enables them. Disabling a feature hides its job tab but preserves the existing definitions, reports, history, grants, and submissions.

​

Reviewer Feedback Is Employer-Private

Reviewer Feedback is a report ledger, not a message to the reported AI trainer and not an annotation on the external task. A report can include:

• The canonical AI trainer and contract, when known
• An external masked AI trainer ID and task ID
• Severity: MINOR, MODERATE, MAJOR, or CRITICAL
• A suggested action: WARNING, RETRAINING, OFFBOARDING, or no suggestion
• One report body explaining the observed mistake or knowledge gap

Only the employer side can list reports, read report history, search, aggregate, link legacy reports, or change status. OpenTrain does not expose the report, author, severity, recommendation, notes, history, or aggregate counts to the subject AI trainer through the API, CLI, MCP, or worker interface. Reports move through OPEN, IN_REVIEW, ACTIONED, and DISMISSED. Terminal transitions and reopening require a reason. Every transition is append-only in the report’s event history, and writes use concurrency and idempotency guards.

​

Reviewer Access

Employers explicitly choose who may submit Reviewer Feedback. An access grant is available only for a person with an active contract on that job. Ending a contract makes that person ineligible for a new grant; revocation removes the authoring permission without deleting prior reports. The employer API exposes the eligible active-contract roster, current grants, single-person grant or revoke, and transactional bulk grant or revoke. Employer-authored reports do not require a reviewer grant.

​

Adaptive Task Checklists

A job can have multiple published checklists—for example, one for initial annotation and another for quality control. AI trainers and reviewers can choose the checklist that fits their role from the job’s Checklist tab. Checklist definitions are ordered trees with up to three authored levels. Their node types are: Visibility rules can depend only on earlier answers. For example, a domain choice can reveal a domain-specific section, or a numeric car count can create one repeated inspection block per car. The server validates references, ordering, depth, limits, options, numeric constraints, and worst-case expansion before publication.

​

Draft, Preview, Publish, Enable

Checklist authoring separates private drafts from the version shown to workers:

1. Build a complete local bundle with a title, required mode, and typed tree.
2. Validate it without writing anything.
3. Create a new definition or update an existing draft using the exact draft version last read.
4. Resolve sample answers and visually review the exact draft revision in a browser.
5. Publish with both the expected draft revision and expected live version.
6. Review the returned change impact and any answers that open runs must reconfirm.
7. Enable Task Checklists for the job only after the published definitions are ready.

Publishing creates an immutable version. Rewording or reconfiguring an answerable node makes affected open-run answers stale; moving a node without changing its meaning preserves the answer. Submitted results remain pinned to their original version, title, tree, and active answers.

​

Agent-Surface Safety

API tokens require the dedicated least-privilege scopes: Write scopes imply the matching read scope. Every request also requires canonical employer access to the exact job. A denied job remains opaque, and a global rollout gate can keep either operation family unavailable even when the token has the scope. Call the capabilities endpoint before using either family. Per-job settings are separate: an operation family can be available to the account while the corresponding job tab remains disabled.

​

Related