CAI-3 shared assessment framework
Review and limits. The course content has completed cross-model editorial review. The task descriptions and award rule below describe the configured exam. Domain weights, task weights, timings and the pass standard remain provisional: no completed job-task analysis or empirical standard-setting study is claimed. The 0.25 human-review band is unvalidated; it is not an estimated standard error of measurement.
Course completion alone does not earn a credential. CAI-2 is recommended, with no admission prerequisite: Agent Fundamentals provides the common bridge.
A CAI-3 AI Automation Specialist credential shows that the holder designed, built and audited a bounded AI workflow for a role in the exam, and that a person reviewed the result. CAI-2 asks someone to produce good work with AI; CAI-3 asks them to specify how several steps, tools, people and checks work together, prove the design with observed tests, and take responsibility when a run goes wrong. More autonomy or a longer prompt does not itself demonstrate a higher level.
Domains and coverage
| Domain | Weight | Observable evidence | | -------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------- | | D1 Workflow architecture | 20% | A justified decomposition, explicit dependencies and candidate-led iteration of the workflow. | | D2 Bounded tools and integration | 20% | Input/output contracts, executable text configuration, and controls at integration boundaries. | | D3 Data boundaries and grounded verification | 20% | Minimum necessary fields, source checking, and log-grounded failure findings. | | D4 Human authority and recovery | 20% | Approval before consequential action, authority checks, containment and recoverable state. | | D5 Testing and operating oversight | 20% | Reproducible tests, fixes with retests, and named owners with actionable monitoring thresholds. |
Each of the three independently scored tasks has equal exam weight. Each task has five criteria, equally weighted within the task. Each domain appears in three criteria across the exam: 3 × (1/3 × 1/5) = 20%. Every track keeps these weights and criterion ids.
Three task families and traceability
| Task | Criterion id | Domain | Required work | | -------------------- | ------------------- | ------ | -------------------------------------------------------------------------------------- | | Workflow design spec | decomposition | D1 | Goal, steps, decision points and accountable owner. | | Workflow design spec | dependencies | D1 | Inputs, outputs, state and ordering between steps. | | Workflow design spec | tool-contracts | D2 | Allowed tools, input/output schema, permissions and limits. | | Workflow design spec | data-boundaries | D3 | Field-level data map, destinations, retention and uncertainty handling. | | Workflow design spec | approval-gates | D4 | Named approver, exact payload, wait/deny/expiry branches and no premature side effect. | | Build and test | configuration | D2 | Prompt chain or agent configuration with bounded tool declarations. | | Build and test | iteration-process | D1 | Candidate decisions visible in the recorded workspace turns. | | Build and test | grounded-validation | D3 | Source checks and validation before downstream use. | | Build and test | test-evidence | D5 | Inputs, expected behavior, observed outputs and a verdict for every supplied case. | | Build and test | fixes-retests | D5 | Identified failure, specific change, rerun evidence and remaining limitation. | | Oversight audit | failure-findings | D3 | Findings linked to log events and the supplied policy. | | Oversight audit | authority-audit | D4 | Reconstruction of approval timing, authorized actor and actual action. | | Oversight audit | containment | D4 | Immediate stop/containment, affected scope, recovery and owner. | | Oversight audit | monitoring | D5 | Measures, thresholds, review cadence and response to an alert. | | Oversight audit | preventive-controls | D2 | Concrete integration controls plus a way to check that they work. |
All deliverables are text or markdown in the existing AI workspace. No upload, external account, live integration, external service or code-execution harness is required. A configuration is a written artifact; an assistant's claim that it called a tool is not proof of a tool run. Distinguish observed workspace outputs, deterministic manual checks and proposed integration behavior.
- Workflow design spec: you receive a role goal, permitted data and tools, a policy and a success target. You produce a flow, step contracts, prompts, a data map, an approval state machine, bounded retries, stop conditions and measures.
- Build and test: you receive ordinary fictional cases for a normal result, a missing source and an unavailable tool. You produce the chain or configuration, actual workspace trials, a results table, a revision and a retest. Process criteria use the server-recorded transcript; copied assistant prose is not evidence of your process. Unavailable or absent evidence is marked cannot-assess.
- Oversight audit: you receive a run log with four operational defects: a policy breach, an unsupported action claim, an action before approval and data copied to an unapproved destination. You produce event and policy citations, impact, containment, controls and monitoring. These are ordinary fictional work records.
The three tasks use different cases, so one answer cannot stand in for all three. The exam has one shared 240-minute window. A suggested split is 75 minutes design, 90 build/test, 60 audit and 15 minutes checking/submission; you choose how to divide the time. Timing remains provisional.
What varies by track
| Track | Role workflow | Consequential action kept with a person | Success and countermeasure | | -------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | Customer Support & Success | Triage, retrieve policy, draft a reply, escalate and record the case. | Refunds, exceptions and sensitive handovers. | Resolution quality together with wrong-resolution and missed-handover rates. | | Marketing | Research, draft, check claims, route approval and prepare publication. | Publication, claims approval and spend changes. | Accepted asset quality together with unsupported-claim and approval-bypass counts. | | Sales | Research an account, assemble an offer draft and prepare a CRM update. | Outreach, commercial commitments and account writes. | Accurate accepted drafts together with false-claim and duplicate-contact rates. | | HR & People Operations | Gather policy sources, draft a response and route an exception. | Decisions affecting a person, policy exceptions and sensitive disclosures. | Correct routing and grounded replies together with inappropriate-disclosure counts. | | Data & Business Analysis | Validate inputs, transform, analyze and prepare a decision memo. | Release of a consequential analysis or changed business record. | Reproducibility and decision usefulness together with invalid-input and silent-error rates. |
The context, ordinary test cases, approved tools, source facts and policy vary; the domains, equal task weights, evidence rules and human authority do not. Fictional companies and data serve individuals and corporate roles. No vendor knowledge or named model is required. All five tracks use the same first module, Agent Fundamentals, followed by five role modules.
Rubrics and scoring
Each task is scored against a rubric with the five criteria above. Each criterion has exactly four observable conditions, and its level from 0 to 4 is the number of conditions met. Conditions describe the work, not the candidate's intent. Each track ties the conditions to its own role evidence and keeps the same criterion ids, scope, weights and scale.
Only the rendered task instructions, materials, limits and criterion labels reach candidates. Answer keys and the mapping of audit defects to checklist items stay with the grader. Grader-only reference material is never candidate evidence and cannot grant credit. A policy supplied in the materials is context, not evidence that the candidate applied it.
AI grading uses three separate scoring passes on a fixed, versioned grading configuration and takes the median for each criterion. Evidence that is absent or cannot be assessed earns no credit.
How results are decided
- A person reviews every CAI-3 result, pass or fail, and makes the decision. The AI score is only a starting point for that reviewer. No result is released automatically, and no second AI opinion or sample check replaces this review.
- Award rule (provisional): an overall mean of at least 2.5 of 4 across the three equally weighted tasks, with every task at least 2.0. There is no separate minimum for any single criterion, including recorded AI collaboration.
- The reviewer's decision is final and may differ from the AI score. You can still appeal any result.
- The exam requires identity verification, including when you resume a session. Integrity cases follow their own separate process.
- Each exam session keeps the review requirement it started with; a later policy change cannot lower it.
What has been checked
- Editorial review checked the course and task content for accuracy, alignment, accessibility and bias. It does not establish a validated pass standard.
- Scoring checks that compare the grading configuration with panel-adjudicated scores have not been run for CAI-3, which is why a person decides every result.
- Job-task analysis and standard setting remain provisional. The weights and award rule are planning values, not an empirical finding about workplace performance; they are reviewed against real submissions after launch.
- A structural check shows the rubrics and task templates compile and are consistent. It does not show learning quality, job relevance or an exam standard.