# Screen applicants with a quiz Add an application screening quiz to a job with the CLI or MCP: create a course with a quiz, link it with stage SCREENING before publishing, review and grade each applicant, rank by screening score, and backfill earlier proposals. Source: https://www.opentrain.ai/docs/developers/guides/screen-applicants-with-a-quiz/ An application screening quiz is an LMS course with a graded quiz that every applicant to a job completes as part of their application. You link a published course to the job with the `SCREENING` stage. From then on, every new proposal on the job is enrolled automatically: the AI trainer submits the proposal, then completes the course and quiz. You review each enrollment by proposal, rank proposals by screening score, and still make the hiring decision yourself. This is different from quizzes assigned as [Project To-dos](https://www.opentrain.ai/docs/developers/cli/project-todos/), which go to AI trainers you have already hired. For the employer view of the same feature, see [Application screening quiz](https://www.opentrain.ai/docs/employers/application-screening-quiz/). > Warning > > **Link the screening course before you publish the job.** OpenTrain enrolls > only proposals submitted after the link exists. Proposals submitted earlier > are not enrolled retroactively; each one needs a separate backfill call > ([Step 8](#step-8-backfill-proposals-submitted-before-the-link)). When you > draft a job, ask the human whether to screen applicants with a quiz, and link > the course before you call publish. ## Before you start - **Versions:** `@opentrain-ai/cli` or `@opentrain-ai/mcp` **0.27.0 or later**. Per-question applicant results (Step 7) need **0.30.0 or later**. The hosted MCP endpoint at `https://app.opentrain.ai/mcp` exposes the same tools. - **Permissions:** `lms:read` and `lms:write` for courses, the screening link, status, and review; `proposals:read` to list and rank proposals; `jobs:write` to publish the job. See [Scopes and Capabilities](https://www.opentrain.ai/docs/developers/concepts/scopes-and-capabilities/). - **Discovery:** call `opentrain lms capabilities --json` (MCP: `opentrain_lms_capabilities`) for the current question types, settings enums, and limits instead of relying on remembered values. - **A job draft:** create one with the [Post a job](https://www.opentrain.ai/docs/developers/guides/post-a-job/) flow, and stop before Step 3 (Publish) of that guide. The full sequence: ```text 1. Create and publish the quiz lms assessments create / publish 2. Put it in a course and publish it lms courses create → checkout → push → publish 3. Link the course with stage SCREENING lms training link --stage SCREENING --max-attempts 4. Publish the job jobs publish 5. Watch enrollment lms screening status 6. Review, grade, release, decide lms review / grade / release-result / decide --proposal 7. Rank applicants proposals list --sort screening-score 8. Backfill proposals that predate the link lms screening assign ``` Placeholders such as ``, ``, ``, and `` stand for IDs returned by earlier calls. Idempotency keys in the examples are illustrative; use your own unique key per operation. ## Step 1: Create and publish the quiz A quiz is a Native Forms assessment. Save a definition like this as `screening-quiz.json`: ```json { "title": "Applicant screening quiz", "definition": { "schemaVersion": 2, "questions": [ { "id": "q1", "type": "MULTIPLE_CHOICE", "prompt": "Which sentiment label fits a sentence that states a fact with no opinion?", "required": true, "options": [ { "id": "a", "label": "Positive" }, { "id": "b", "label": "Neutral" }, { "id": "c", "label": "Negative" } ], "grading": { "points": 1, "answerKey": { "optionId": "b" } } }, { "id": "q2", "type": "PARAGRAPH", "prompt": "Explain how you would label a sarcastic product review.", "required": true, "manualGrading": { "points": 4 } } ] }, "settings": { "attemptPolicy": { "mode": "LIMITED", "maxAttempts": 2 }, "passRule": { "mode": "PERCENTAGE", "thresholdPercent": 70 }, "resultRelease": "IMMEDIATE" } } ``` ### CLI ```bash opentrain lms assessments create \ --input screening-quiz.json \ --key screening-quiz-create-01 opentrain lms assessments publish \ --assessment \ --expected-revision \ --confirm-publish \ --key screening-quiz-publish-01 ``` Run `opentrain lms assessments create --help` for every question type and grading option. ### MCP Call `opentrain_lms_manage_assessment`: ```json { "operation": { "action": "create", "title": "Applicant screening quiz", "definition": { "schemaVersion": 2, "questions": ["...as above..."] }, "settings": { "attemptPolicy": { "mode": "LIMITED", "maxAttempts": 2 }, "passRule": { "mode": "PERCENTAGE", "thresholdPercent": 70 }, "resultRelease": "IMMEDIATE" }, "idempotencyKey": "screening-quiz-create-01" } } ``` Then call `opentrain_lms_publish_assessment`: ```json { "formId": "", "expectedDraftRevision": , "confirmPublish": true, "idempotencyKey": "screening-quiz-publish-01" } ``` Use the integer draft revision returned by the create call or by `opentrain_lms_get_assessment`. How the quiz settings behave for applicants: - **`attemptPolicy`**: the link's attempt ceiling (Step 3) also applies. Applicants get the lower of the two; an `UNLIMITED` quiz gets the ceiling. - **`passRule`**: required for a pass or fail result (`passed`). Without it, a fully graded attempt has a percentage but no pass or fail. - **`resultRelease`**: `IMMEDIATE` shows the applicant their score once the attempt is graded. `MANUAL` holds the score until you release it in Step 6. - **`manualGrading` questions**: the score stays pending until you grade them in Step 6. Proposals awaiting grades rank below fully graded ones. You can also build and publish the quiz in **Forms** in the OpenTrain app; the published form works the same way. ## Step 2: Put the quiz in a course and publish it The course needs an `ASSESSMENT` lesson that references the published quiz. Lessons before the quiz, such as a project overview, are optional. A module with only the quiz looks like this: ```json { "key": "screening", "title": "Screening", "lessons": [ { "key": "screening-quiz", "title": "Screening quiz", "kind": "ASSESSMENT", "required": true, "assessment": { "formId": "", "passRequired": true, "manualReview": false } } ] } ``` With `passRequired: true`, an applicant who fails can retake the quiz while attempts remain. Set `manualReview: true` only if you want to record the final pass or fail yourself with `decide` (Step 6). ### CLI ```bash opentrain lms courses create --title "Applicant screening" --key screening-course-create-01 opentrain lms checkout --course --dir ./screening-course ``` Save the module above as `./screening-course/modules/01-screening.json` and add that path to `moduleFiles` in `./screening-course/course.yaml`. Then validate, push, and publish: ```bash opentrain lms validate --dir ./screening-course --remote --key screening-course-validate-01 opentrain lms push --dir ./screening-course opentrain lms publish --dir ./screening-course --confirm-publish --key screening-course-publish-01 ``` ### MCP Call `opentrain_lms_manage_course` with the module in `draftTree`: ```json { "operation": { "action": "create", "title": "Applicant screening", "draftTree": { "schemaVersion": 1, "modules": ["...the module above..."] }, "idempotencyKey": "screening-course-create-01" } } ``` Then call `opentrain_lms_validate_course` with `{ "courseId": "", "idempotencyKey": "screening-course-validate-01" }` and publish with `opentrain_lms_publish_course`: ```json { "courseId": "", "expectedDraftRevision": , "confirmPublish": true, "idempotencyKey": "screening-course-publish-01" } ``` Use the integer draft revision returned by the create call or by `opentrain_lms_get_course`. If validation returns warnings, publishing requires acknowledging them. See [Validate LMS authoring before publishing](https://www.opentrain.ai/docs/developers/cli/commands/#validate-lms-authoring-before-publishing). An employer can also build the course under **Training** in the OpenTrain app; link it in Step 3 the same way. ## Step 3: Link the course to the job as its screening course ### CLI ```bash opentrain lms training link \ --job \ --course \ --stage SCREENING \ --max-attempts 2 \ --confirm-link \ --key screening-link-01 ``` ### MCP Call `opentrain_lms_link_job_course`: ```json { "jobId": "", "courseId": "", "stage": "SCREENING", "screeningMaxAttempts": 2, "confirmLink": true, "idempotencyKey": "screening-link-01" } ``` Rules for a screening link: - The course must have a **published** version when you link it. - A job has at most **one** active screening link. - `--max-attempts` (MCP: `screeningMaxAttempts`) is an integer from 1 to 100 and defaults to 1. It applies only with the `SCREENING` stage. - Linking enrolls nobody who has already applied. It does not need the job's Training setting and turns no job settings on. - Without `--stage`, a link that already exists keeps its stage and ceiling. The stage and ceiling are part of the idempotency receipt, so retrying the same key with different values returns `409`. - Linking from a job's **Training** tab in the OpenTrain app creates a post-hire training link, not a screening link. Confirm the link before you publish: ```bash opentrain lms screening status --job --pretty ``` ### Make the quiz the only screening step A screening quiz does not replace the AI interview. By default the AI interview is required, so applicants take both. To screen with the quiz alone, turn the interview requirement off: ### CLI ```bash opentrain jobs interview set --job-id --not-required ``` ### MCP On a draft, call `opentrain_update_job_draft_fields` with `{ "jobId": "", "patch": { "aiInterviewRequired": false } }`. On a live job, use `opentrain_update_published_job` with the same patch. ## Step 4: Publish the job Publish as usual with `opentrain jobs publish --job-id ` (MCP: `opentrain_publish_job`). See [Post a job](https://www.opentrain.ai/docs/developers/guides/post-a-job/#step-3-publish). Every proposal submitted from now on is enrolled in the screening course. ## Step 5: Watch enrollment ### CLI ```bash opentrain lms screening status --job --limit 500 ``` ### MCP Call `opentrain_lms_screening_status` with `{ "jobId": "", "limit": 500 }`. The response describes the link and every proposal on the job, newest first: | Field | Meaning | | --- | --- | | `screening.link` | The active screening course, its attempt ceiling (`maxAttempts`), and its current published version. `null` when the job has no screening link. | | `screening.proposals[].enrollmentKey` | `proposal:`. Pass it as the assignment ID to the review tools in Step 6. | | `screening.proposals[].binding.status` | `OPEN` (not started), `IN_PROGRESS`, or `SUBMITTED`. | | `screening.proposals[].binding.waitingOn` | `WORKER` (the applicant's move), `EMPLOYER` (a manual grade or decision is pending), or `NONE`. | | `screening.proposals[].binding.courseVersionCurrent` | `false` when the applicant is on an older course version. | | `screening.proposals[].binding` = `null` | A binding gap: this proposal is not enrolled. | | `screening.bindingGaps` | How many proposals are not enrolled. Backfill them in Step 8. | | `screening.staleVersionCount` | How many applicants are on an older course version, so their scores may not compare directly. | | `screening.hasMore` | `true` when the job has more proposals than `limit` (default and maximum 500). | ## Step 6: Review, grade, release, and decide each applicant The review commands are the same ones used for post-hire training. Address a screening enrollment with `--proposal ` instead of `--assignment `. In MCP, pass the enrollment key `proposal:` as `assignmentId`. ### CLI ```bash # Read submitted answers, grades, and each lesson's review target opentrain lms review --proposal # Grade manually graded questions opentrain lms grade --proposal \ --lesson screening-quiz --attempt \ --grades-json '[{"questionId":"q2","awardedPoints":3,"reason":"Covers tone, misses context."}]' \ --expected-target \ --key screening-grade-01 # Make a held (MANUAL release) result visible to the applicant opentrain lms release-result --proposal \ --lesson screening-quiz --attempt \ --confirm-release --expected-target \ --key screening-release-01 # Record a final verdict on a manual-review lesson opentrain lms decide --proposal \ --lesson screening-quiz --decision PASS \ --confirm-decision --expected-target \ --key screening-decide-01 ``` `opentrain lms attempts list --proposal ` lists every attempt; `opentrain lms attempts get --proposal --attempt ` shows per-question evidence for one attempt. ### MCP 1. `opentrain_lms_review_assignment` with `{ "assignmentId": "proposal:" }`. 2. `opentrain_lms_grade_assessment` with `assignmentId`, `lessonKey`, `attemptId`, `grades`, `expectedTargetHash`, and `idempotencyKey`. 3. `opentrain_lms_release_result` with `assignmentId`, `lessonKey`, `attemptId`, `confirmRelease: true`, `expectedTargetHash`, and `idempotencyKey`. 4. `opentrain_lms_decide_assessment` with `assignmentId`, `lessonKey`, `decision` (`PASS` or `FAIL`), `confirmDecision: true`, `expectedTargetHash`, and `idempotencyKey`. Order and safety: - Grade first, then release, then decide. Release is blocked while manual grades are outstanding. - `review` returns a `reviewTarget` with a `targetHash` for each lesson. Pass it as `--expected-target` (MCP: `expectedTargetHash`). If the applicant resubmitted after you reviewed, the call fails with `409` and `REVIEW_TARGET_STALE` instead of acting on a different attempt. - Releasing reveals the score, per-question detail, and grading notes to the applicant. - Decisions are final. `FAIL` requires `--reason` (MCP: `reason`), which the applicant sees as feedback. - None of these calls hires or declines anyone. The hiring decision stays on the proposal. ## Step 7: Rank applicants by screening score ### CLI ```bash opentrain proposals list --job-id --sort screening-score --limit 100 ``` ### MCP Call `opentrain_list_proposals`: ```json { "jobId": "", "sort": "screening_score", "limit": 100 } ``` The REST equivalent is `GET /api/public/v1/jobs/{id}/proposals?sort=screening_score`. How the ranking works: - Order: highest finalized percentage first, then fully graded attempts without a percentage, then submissions awaiting grades, then enrollments not yet submitted, then proposals with no enrollment. - One request returns one ranked page of up to `limit` proposals (default 25, maximum 100). There is no cursor with this sort. - The ranking covers the newest 500 proposals on the job. `ranking.rankedCount` is how many proposals it covered, `ranking.scoredCount` is how many have a finalized percentage, and `ranking.complete` is `false` when the job has more proposals than the ranking covers. `hasMore` is `true` when more ranked rows exist than `limit`, or when the ranking is incomplete. - If screening scores cannot be read, the request fails with a retryable `503` rather than returning a recency order. Retry, or use `sort=recent`, where each row then reports `screeningRead: "FAILED"`. Every proposal row, and `proposals get` (MCP: `opentrain_get_proposal`), carries a `screening` block (`enrollmentKey`, `status`, `waitingOn`, `courseVersionNumber`, `courseVersionCurrent`, `gradeState`, `earnedPoints`, `possiblePoints`, `percentage`, `passed`, `submittedAt`, `released`) and `screeningRead` (`OK` or `FAILED`). `screening` is `null` when the job has no screening course or the proposal is not enrolled. When `screeningRead` is `FAILED`, a `null` block means unknown, not "not enrolled". `percentage` and `passed` appear only once every submitted quiz is fully graded. To see which questions applicants miss, read per-question results for the applicant audience. Names are masked as first name and last initial. ### CLI ```bash opentrain lms question-results --job --audience APPLICANT --course ``` ### MCP Call `opentrain_lms_question_results` with `{ "jobId": "", "audience": "APPLICANT", "courseId": "" }`. ## Step 8: Backfill proposals submitted before the link If the job was published before the screening link, or an enrollment did not complete when a proposal was submitted, the proposal has no enrollment. Find them in Step 5 (`binding: null`, counted by `bindingGaps`), then enroll each one: ### CLI ```bash opentrain lms screening assign \ --job \ --proposal \ --confirm-assign \ --key screening-assign- ``` ### MCP Call `opentrain_lms_screening_assign`: ```json { "jobId": "", "jobOfferId": "", "confirmAssign": true, "idempotencyKey": "screening-assign-" } ``` The backfilled enrollment is identical to one created at submission and uses the link's current published course version. The call is idempotent per proposal: `created: false` means the proposal was already enrolled, and retrying the same key replays the receipt. Typed refusals: | Status | Code | Meaning | | --- | --- | --- | | `409` | `SCREENING_NOT_CONFIGURED` | The job has no active screening link. | | `409` | `SCREENING_COURSE_UNPUBLISHED` | The linked course has no published version. | | `400` | `SCREENING_BINDING_INVALID` | The ID is not a proposal on this job, or it is an invite rather than a proposal. | ## Change or remove the screening course - **New course version:** publish the course again. New enrollments, including backfills, use the new version. Applicants already enrolled keep their version and count toward `staleVersionCount`. - **Stop screening:** run `opentrain lms training unlink --job --course --confirm-unlink --key ` (MCP: `opentrain_lms_unlink_job_course`). The link is retired with its history kept. Applicants who have not finished then see that the screening course is no longer available. ## REST endpoints | Method | Endpoint | Purpose | | --- | --- | --- | | `POST` | `/api/public/v1/lms/assessments` | Create the quiz draft. | | `POST` | `/api/public/v1/lms/assessments/{formId}/publish` | Publish the quiz. | | `POST` | `/api/public/v1/lms/courses` | Create the course draft. | | `POST` | `/api/public/v1/lms/courses/{courseId}/publish` | Publish the course. | | `POST` | `/api/public/v1/lms/jobs/{jobId}/courses/link` | Link the course with `stage: "SCREENING"` and `screeningMaxAttempts`. | | `GET` | `/api/public/v1/lms/jobs/{jobId}/screening` | Read the link and every proposal's enrollment. | | `POST` | `/api/public/v1/lms/jobs/{jobId}/screening/assign` | Backfill one proposal (`jobOfferId`, `confirmAssign: true`). | | `GET` | `/api/public/v1/lms/assignments/{assignmentId}/review` | Review one enrollment; `assignmentId` is `proposal:`. | | `POST` | `/api/public/v1/lms/assignments/{assignmentId}/grades` | Record manual grades. | | `POST` | `/api/public/v1/lms/assignments/{assignmentId}/release-result` | Release a held result. | | `POST` | `/api/public/v1/lms/assignments/{assignmentId}/decision` | Record a final pass or fail. | | `GET` | `/api/public/v1/jobs/{id}/proposals?sort=screening_score` | Rank proposals by screening score. | | `GET` | `/api/public/v1/lms/jobs/{jobId}/question-results?audience=APPLICANT` | Per-question applicant results. | Every write requires an `Idempotency-Key` header. See [List Job Proposals](https://www.opentrain.ai/docs/developers/api-reference/jobs/list-proposals/) for the ranking response fields. ## Completion checklist Before you report that screening is set up: - the human agreed to screen applicants with a quiz, and the course has a published version; - `lms screening status` shows the link, with the attempt ceiling you intended; - the AI interview requirement matches what the human wants (both screens, or the quiz alone); - the link was created before publishing, or `bindingGaps` is `0` after backfilling earlier proposals; - you shared how to review results: the **Screening quiz** column in the job's **Proposals** tab, or a ranked `proposals list --sort screening-score`. ## Related [Application screening quiz](https://www.opentrain.ai/docs/employers/application-screening-quiz/) The employer view: what applicants see and how results appear in proposal review. [Post a job](https://www.opentrain.ai/docs/developers/guides/post-a-job/) Draft and publish the job; link the screening course before Step 3. [Evaluate candidates](https://www.opentrain.ai/docs/developers/guides/evaluate-candidates/) Combine screening scores with interview scores, profiles, and pre-hire messages. [CLI: Project To-dos and quizzes](https://www.opentrain.ai/docs/developers/cli/project-todos/) Assign quizzes to AI trainers you have already hired.