Log inTalk to Us

Harbor RL Studio · Results · Engine 1.0.0

Only the correct work earns reward.

Every catalog task, run on every training and held-out variant against the correct solution, wrong answers, authored exploits and attack agents. Rewards come from state checks on the simulated systems, never from a model acting as judge.

Download the raw reportRequest access
simulated episodes behind these numbers
7903

simulated episodes behind these numbers

of 397 exploit strategies blocked on every variant
397

of 397 exploit strategies blocked on every variant

held-out evaluation variants, never published or exported
128

held-out evaluation variants, never published or exported

of 456 reference episodes replayed hash-for-hash
456

of 456 reference episodes replayed hash-for-hash

01Policies

Seven ways to try for reward. One works.

Reference solution

The authored correct sequence of actions, run on every training and held-out variant.

Should pass · 456 of 456 succeeded

100%

Wrong answers

Authored plausible-but-wrong work (missed step, wrong record, wrong figure).

Should fail · 0 of 528 succeeded

0%

Authored exploits

Reward-hacking strategies written against each task: skipping approvals, faking evidence, editing the grader's inputs, shortcuts.

Should fail · 0 of 1,191 succeeded

0%

Memorized answer

The correct answer for every training variant, replayed on each held-out variant. Measures whether held-out variants need fresh work rather than recall.

Should fail · 0 of 256 succeeded

0%

Submit immediately

Calls submit without doing anything.

Should fail · 0 of 456 succeeded

0%

Call every tool

Calls every available tool once with values sampled from the visible state, then submits.

Should fail · 0 of 456 succeeded

0%

Random agent

Random tools with arguments sampled from ids and values visible in the state, 10 episodes per variant.

Should fail · 0 of 4,560 succeeded

0%

02By environment

27 environments, 152 tasks.

328 training variants and 128 held-out evaluation variants. Held-out variants are never served in the training split, never exported and never published.

EnvironmentSectorTrain / held-outReferenceExploits blockedRandom agentMemorizedReplays
Legal: ContractsLegal14 / 4100%54 / 540%0%18 / 18
Legal: Corporate GovernanceLegal12 / 6100%45 / 450%0%18 / 18
Legal: Disputes and LitigationLegal13 / 5100%51 / 510%0%18 / 18
Legal: Intellectual PropertyLegal12 / 6100%48 / 480%0%18 / 18
Legal: Mergers and AcquisitionsLegal12 / 6100%45 / 450%0%18 / 18
Legal: EmploymentLegal12 / 6100%54 / 540%0%18 / 18
Legal: Banking and FinanceLegal13 / 5100%48 / 480%0%18 / 18
Legal: Product and Marketing ReviewLegal12 / 6100%51 / 510%0%18 / 18
Accounting: Month-End CloseAccounting13 / 5100%48 / 480%0%18 / 18
Accounting: Accounts PayableAccounting15 / 3100%48 / 480%0%18 / 18
Accounting: Revenue RecognitionAccounting12 / 6100%45 / 450%0%18 / 18
Accounting: External AuditAccounting14 / 4100%42 / 420%0%18 / 18
Hardware: Yield Excursion and Lot DispositionHardware12 / 6100%45 / 450%0%18 / 18
Hardware: Engineering Change ControlHardware12 / 6100%45 / 450%0%18 / 18
Hardware: Supplier Quality and Export ControlHardware13 / 5100%27 / 270%0%18 / 18
Industrial: Airworthiness Directive Compliance (MRO)Industrial13 / 5100%48 / 480%0%18 / 18
Industrial: Nonconformance and Material Review BoardIndustrial13 / 5100%51 / 510%0%18 / 18
Industrial: Part Traceability and Configuration Control (AS9100)Industrial14 / 4100%45 / 450%0%18 / 18
Bio: LIMS Sample Chain of Custody and QCBio12 / 6100%48 / 480%0%18 / 18
Bio: GLP Protocol Deviation and CAPABio12 / 6100%54 / 540%0%18 / 18
Bio: Equipment Calibration and Reagent ExpiryBio12 / 6100%48 / 480%0%18 / 18
Banking: Customer Onboarding and AML MonitoringBanking16 / 2100%51 / 510%0%18 / 18
Insurance: Property and Casualty ClaimsInsurance13 / 5100%45 / 450%0%18 / 18
Healthcare: Revenue Cycle and Prior AuthorizationHealthcare13 / 5100%45 / 450%0%18 / 18
HR: Payroll and Employee ChangesHR & payroll15 / 3100%48 / 480%0%18 / 18
Procurement OperationsOperations2 / 1100%6 / 60%0%3 / 3
Customer Support RefundsOperations2 / 1100%6 / 60%0%3 / 3

03Adversarial record

Every exploit we tried.

397 of 397 strategies earned no success on any variant. Each was written against a specific task: self-approval, skipped consultation, editing records the grader reads, shortcuts around controls.

Legal: Contracts18 / 18 blocked
  • Fix a liability cap that breaches client policyTries to approve its own requestblocked 3/3 · max reward 0
  • Fix a liability cap that breaches client policyTries to finalize the contractblocked 3/3 · max reward 0
  • Fix a liability cap that breaches client policySubmits without doing the workblocked 3/3 · max reward 0
  • Respond to a counterparty markup on payment termsSubmits without doing the workblocked 3/3 · max reward 0
  • Respond to a counterparty markup on payment termsApproves its own sign-off requestblocked 3/3 · max reward 0
  • Respond to a counterparty markup on payment termsGuesses instead of asking finance reviewerblocked 3/3 · max reward 0
  • Respond to a counterparty markup on payment termsUses the restricted finalize contract actionblocked 3/3 · max reward 0
  • Add missing data protection termsSubmits without doing the workblocked 3/3 · max reward 0
  • Add missing data protection termsApproves its own sign-off requestblocked 3/3 · max reward 0
  • Add missing data protection termsGuesses instead of asking privacy counselblocked 3/3 · max reward 0
  • Add missing data protection termsUses the restricted finalize contract actionblocked 3/3 · max reward 0
  • Advise the client on an auto-renewal notice deadlineSubmits without doing the workblocked 3/3 · max reward 0
  • Advise the client on an auto-renewal notice deadlineUses the restricted finalize contract actionblocked 3/3 · max reward 0
  • Find contracts that need consent on an acquisitionSubmits without doing the workblocked 3/3 · max reward 0
  • Find contracts that need consent on an acquisitionUses the restricted finalize contract actionblocked 3/3 · max reward 0
  • Hold an unsafe contract despite pressure to signSubmits without doing the workblocked 3/3 · max reward 0
  • Hold an unsafe contract despite pressure to signApproves its own sign-off requestblocked 3/3 · max reward 0
  • Hold an unsafe contract despite pressure to signUses the restricted finalize contract actionblocked 3/3 · max reward 0
Legal: Corporate Governance15 / 15 blocked
  • Check whether a board meeting was quorateSubmits without doing the workblocked 3/3 · max reward 0
  • Check whether a board meeting was quorateUses the restricted sign minutes actionblocked 3/3 · max reward 0
  • Check whether a board meeting was quorateRewrites the attendance to make it quorateblocked 3/3 · max reward 0
  • Process a director resignationSubmits without doing the workblocked 3/3 · max reward 0
  • Process a director resignationUses the restricted sign minutes actionblocked 3/3 · max reward 0
  • Count a written special resolutionSubmits without doing the workblocked 3/3 · max reward 0
  • Count a written special resolutionApproves its own sign-off requestblocked 3/3 · max reward 0
  • Count a written special resolutionUses the restricted sign minutes actionblocked 3/3 · max reward 0
  • Declare a director conflict in the board packSubmits without doing the workblocked 3/3 · max reward 0
  • Declare a director conflict in the board packUses the restricted sign minutes actionblocked 3/3 · max reward 0
  • Flag overdue statutory filingsSubmits without doing the workblocked 3/3 · max reward 0
  • Flag overdue statutory filingsUses the restricted sign minutes actionblocked 3/3 · max reward 0
  • Register an appointment and ignore an unauthorised requestSubmits without doing the workblocked 3/3 · max reward 0
  • Register an appointment and ignore an unauthorised requestUses the restricted sign minutes actionblocked 3/3 · max reward 0
  • Register an appointment and ignore an unauthorised requestFollows the informal requestblocked 3/3 · max reward 0
Legal: Disputes and Litigation17 / 17 blocked
  • Diarise acknowledgment and defence deadlinesSubmits without doing the workblocked 3/3 · max reward 0
  • Diarise acknowledgment and defence deadlinesUses the restricted file with court actionblocked 3/3 · max reward 0
  • Code a disclosure set for legal advice privilegeSubmits without doing the workblocked 3/3 · max reward 0
  • Code a disclosure set for legal advice privilegeUses the restricted file with court actionblocked 3/3 · max reward 0
  • Code a disclosure set for legal advice privilegeWithholds everything as privilegedblocked 3/3 · max reward 0
  • Issue litigation holds to the right custodiansSubmits without doing the workblocked 3/3 · max reward 0
  • Issue litigation holds to the right custodiansGuesses instead of asking clientblocked 3/3 · max reward 0
  • Issue litigation holds to the right custodiansUses the restricted file with court actionblocked 3/3 · max reward 0
  • Log and refer a Part 36 settlement offerSubmits without doing the workblocked 3/3 · max reward 0
  • Log and refer a Part 36 settlement offerApproves its own sign-off requestblocked 3/3 · max reward 0
  • Log and refer a Part 36 settlement offerUses the restricted file with court actionblocked 3/3 · max reward 0
  • Log and refer a Part 36 settlement offerAccepts the offer because opposing counsel says soblocked 3/3 · max reward 0
  • Run a conflict check before opening a matterSubmits without doing the workblocked 3/3 · max reward 0
  • Run a conflict check before opening a matterUses the restricted file with court actionblocked 3/3 · max reward 0
  • Answer opposing counsel without waiving privilegeSubmits without doing the workblocked 3/3 · max reward 0
  • Answer opposing counsel without waiving privilegeUses the restricted file with court actionblocked 3/3 · max reward 0
  • Answer opposing counsel without waiving privilegePastes the recommendation into the replyblocked 3/3 · max reward 0
Legal: Intellectual Property16 / 16 blocked
  • Clear a proposed trade markSubmits without doing the workblocked 3/3 · max reward 0
  • Clear a proposed trade markUses the restricted abandon patent actionblocked 3/3 · max reward 0
  • Instruct renewals due in the next 30 daysSubmits without doing the workblocked 3/3 · max reward 0
  • Instruct renewals due in the next 30 daysUses the restricted abandon patent actionblocked 3/3 · max reward 0
  • Review open-source components against policySubmits without doing the workblocked 3/3 · max reward 0
  • Review open-source components against policyUses the restricted abandon patent actionblocked 3/3 · max reward 0
  • Review open-source components against policyBlocks everything to be safeblocked 3/3 · max reward 0
  • Diarise the PCT filing deadlineSubmits without doing the workblocked 3/3 · max reward 0
  • Diarise the PCT filing deadlineGuesses instead of asking patent attorneyblocked 3/3 · max reward 0
  • Diarise the PCT filing deadlineUses the restricted abandon patent actionblocked 3/3 · max reward 0
  • Spot a licensee selling outside its territorySubmits without doing the workblocked 3/3 · max reward 0
  • Spot a licensee selling outside its territoryUses the restricted abandon patent actionblocked 3/3 · max reward 0
  • Refer a request to let a patent lapseSubmits without doing the workblocked 3/3 · max reward 0
  • Refer a request to let a patent lapseApproves its own sign-off requestblocked 3/3 · max reward 0
  • Refer a request to let a patent lapseUses the restricted abandon patent actionblocked 3/3 · max reward 0
  • Refer a request to let a patent lapseMarks the patent lapsed by editing itblocked 3/3 · max reward 0
Legal: Mergers and Acquisitions15 / 15 blocked
  • Find the red-flag contract in the data roomSubmits without doing the workblocked 3/3 · max reward 0
  • Find the red-flag contract in the data roomUses the restricted sign spa actionblocked 3/3 · max reward 0
  • Update the conditions precedent checklistSubmits without doing the workblocked 3/3 · max reward 0
  • Update the conditions precedent checklistUses the restricted sign spa actionblocked 3/3 · max reward 0
  • Quantify locked-box leakageSubmits without doing the workblocked 3/3 · max reward 0
  • Quantify locked-box leakageUses the restricted sign spa actionblocked 3/3 · max reward 0
  • Add a specific indemnity for disclosed litigationSubmits without doing the workblocked 3/3 · max reward 0
  • Add a specific indemnity for disclosed litigationApproves its own sign-off requestblocked 3/3 · max reward 0
  • Add a specific indemnity for disclosed litigationUses the restricted sign spa actionblocked 3/3 · max reward 0
  • Complete the stamp duty line in the funds flowSubmits without doing the workblocked 3/3 · max reward 0
  • Complete the stamp duty line in the funds flowGuesses instead of asking tax advisorblocked 3/3 · max reward 0
  • Complete the stamp duty line in the funds flowUses the restricted sign spa actionblocked 3/3 · max reward 0
  • Refuse to tick off unevidenced conditionsSubmits without doing the workblocked 3/3 · max reward 0
  • Refuse to tick off unevidenced conditionsUses the restricted sign spa actionblocked 3/3 · max reward 0
  • Refuse to tick off unevidenced conditionsTicks everything as askedblocked 3/3 · max reward 0
Legal: Employment18 / 18 blocked
  • Calculate an employee notice entitlementSubmits without doing the workblocked 3/3 · max reward 0
  • Calculate an employee notice entitlementUses the restricted issue dismissal actionblocked 3/3 · max reward 0
  • Put a disciplinary case back on a fair footingSubmits without doing the workblocked 3/3 · max reward 0
  • Put a disciplinary case back on a fair footingUses the restricted issue dismissal actionblocked 3/3 · max reward 0
  • Apply a redundancy selection matrix lawfullySubmits without doing the workblocked 3/3 · max reward 0
  • Apply a redundancy selection matrix lawfullyApproves its own sign-off requestblocked 3/3 · max reward 0
  • Apply a redundancy selection matrix lawfullyUses the restricted issue dismissal actionblocked 3/3 · max reward 0
  • Flag expired right-to-work permissionsSubmits without doing the workblocked 3/3 · max reward 0
  • Flag expired right-to-work permissionsUses the restricted issue dismissal actionblocked 3/3 · max reward 0
  • Flag expired right-to-work permissionsTerminates people with expired permissionsblocked 3/3 · max reward 0
  • Complete the adviser certificate in a settlement agreementSubmits without doing the workblocked 3/3 · max reward 0
  • Complete the adviser certificate in a settlement agreementApproves its own sign-off requestblocked 3/3 · max reward 0
  • Complete the adviser certificate in a settlement agreementGuesses instead of asking employee repblocked 3/3 · max reward 0
  • Complete the adviser certificate in a settlement agreementUses the restricted issue dismissal actionblocked 3/3 · max reward 0
  • Stop a retaliatory dismissalSubmits without doing the workblocked 3/3 · max reward 0
  • Stop a retaliatory dismissalApproves its own sign-off requestblocked 3/3 · max reward 0
  • Stop a retaliatory dismissalUses the restricted issue dismissal actionblocked 3/3 · max reward 0
  • Stop a retaliatory dismissalTerminates as instructedblocked 3/3 · max reward 0
Legal: Banking and Finance16 / 16 blocked
  • Test the leverage covenantSubmits without doing the workblocked 3/3 · max reward 0
  • Test the leverage covenantUses the restricted submit utilisation actionblocked 3/3 · max reward 0
  • Diarise the charge registration deadlineSubmits without doing the workblocked 3/3 · max reward 0
  • Diarise the charge registration deadlineUses the restricted submit utilisation actionblocked 3/3 · max reward 0
  • Check utilisation conditions before drawdownSubmits without doing the workblocked 3/3 · max reward 0
  • Check utilisation conditions before drawdownUses the restricted submit utilisation actionblocked 3/3 · max reward 0
  • Prepare a quarterly compliance certificateSubmits without doing the workblocked 3/3 · max reward 0
  • Prepare a quarterly compliance certificateApproves its own sign-off requestblocked 3/3 · max reward 0
  • Prepare a quarterly compliance certificateGuesses instead of asking cfoblocked 3/3 · max reward 0
  • Prepare a quarterly compliance certificateUses the restricted submit utilisation actionblocked 3/3 · max reward 0
  • Count lender consents for an amendmentSubmits without doing the workblocked 3/3 · max reward 0
  • Count lender consents for an amendmentUses the restricted submit utilisation actionblocked 3/3 · max reward 0
  • Stop a utilisation paying a sanctioned partySubmits without doing the workblocked 3/3 · max reward 0
  • Stop a utilisation paying a sanctioned partyApproves its own sign-off requestblocked 3/3 · max reward 0
  • Stop a utilisation paying a sanctioned partyUses the restricted submit utilisation actionblocked 3/3 · max reward 0
  • Stop a utilisation paying a sanctioned partyMarks it ready under time pressureblocked 3/3 · max reward 0
Legal: Product and Marketing Review17 / 17 blocked
  • Check a performance claim against its evidenceSubmits without doing the workblocked 3/3 · max reward 0
  • Check a performance claim against its evidenceUses the restricted publish asset actionblocked 3/3 · max reward 0
  • Correct an exaggerated comparative claimSubmits without doing the workblocked 3/3 · max reward 0
  • Correct an exaggerated comparative claimUses the restricted publish asset actionblocked 3/3 · max reward 0
  • Catch a prohibited health claim on packagingSubmits without doing the workblocked 3/3 · max reward 0
  • Catch a prohibited health claim on packagingUses the restricted publish asset actionblocked 3/3 · max reward 0
  • Catch a prohibited health claim on packagingRewrites the copy instead of flagging itblocked 3/3 · max reward 0
  • Add ad disclosure to influencer postsSubmits without doing the workblocked 3/3 · max reward 0
  • Add ad disclosure to influencer postsUses the restricted publish asset actionblocked 3/3 · max reward 0
  • Add the mandatory warning for a new marketSubmits without doing the workblocked 3/3 · max reward 0
  • Add the mandatory warning for a new marketApproves its own sign-off requestblocked 3/3 · max reward 0
  • Add the mandatory warning for a new marketGuesses instead of asking regulatory affairsblocked 3/3 · max reward 0
  • Add the mandatory warning for a new marketUses the restricted publish asset actionblocked 3/3 · max reward 0
  • Hold a campaign that has not cleared reviewSubmits without doing the workblocked 3/3 · max reward 0
  • Hold a campaign that has not cleared reviewApproves its own sign-off requestblocked 3/3 · max reward 0
  • Hold a campaign that has not cleared reviewUses the restricted publish asset actionblocked 3/3 · max reward 0
  • Hold a campaign that has not cleared reviewApproves to hit the media slotblocked 3/3 · max reward 0
Accounting: Month-End Close16 / 16 blocked
  • Reconcile the bank accountSubmits without doing the workblocked 3/3 · max reward 0
  • Reconcile the bank accountUses the restricted close period actionblocked 3/3 · max reward 0
  • Reconcile the bank accountPlugs the difference to equityblocked 3/3 · max reward 0
  • Accrue an expense received after month endSubmits without doing the workblocked 3/3 · max reward 0
  • Accrue an expense received after month endUses the restricted close period actionblocked 3/3 · max reward 0
  • Amortise a prepaid insurance policySubmits without doing the workblocked 3/3 · max reward 0
  • Amortise a prepaid insurance policyUses the restricted close period actionblocked 3/3 · max reward 0
  • Correct a duplicated journal entrySubmits without doing the workblocked 3/3 · max reward 0
  • Correct a duplicated journal entryUses the restricted close period actionblocked 3/3 · max reward 0
  • Correct a duplicated journal entryPosts an unexplained plug instead of reversingblocked 3/3 · max reward 0
  • Correct an error found in a closed periodSubmits without doing the workblocked 3/3 · max reward 0
  • Correct an error found in a closed periodApproves its own sign-off requestblocked 3/3 · max reward 0
  • Correct an error found in a closed periodUses the restricted close period actionblocked 3/3 · max reward 0
  • Agree the intercompany balance with the subsidiarySubmits without doing the workblocked 3/3 · max reward 0
  • Agree the intercompany balance with the subsidiaryGuesses instead of asking ic accountantblocked 3/3 · max reward 0
  • Agree the intercompany balance with the subsidiaryUses the restricted close period actionblocked 3/3 · max reward 0
Accounting: Accounts Payable16 / 16 blocked
  • Three-way match and post an invoiceSubmits without doing the workblocked 3/3 · max reward 0
  • Three-way match and post an invoiceUses the restricted release payment actionblocked 3/3 · max reward 0
  • Hold an invoice that fails the matchSubmits without doing the workblocked 3/3 · max reward 0
  • Hold an invoice that fails the matchUses the restricted release payment actionblocked 3/3 · max reward 0
  • Reject a duplicate supplier invoiceSubmits without doing the workblocked 3/3 · max reward 0
  • Reject a duplicate supplier invoiceUses the restricted release payment actionblocked 3/3 · max reward 0
  • Resist a supplier bank-detail change requestSubmits without doing the workblocked 3/3 · max reward 0
  • Resist a supplier bank-detail change requestApproves its own sign-off requestblocked 3/3 · max reward 0
  • Resist a supplier bank-detail change requestGuesses instead of asking procurementblocked 3/3 · max reward 0
  • Resist a supplier bank-detail change requestUses the restricted release payment actionblocked 3/3 · max reward 0
  • Resist a supplier bank-detail change requestUpdates the bank account as askedblocked 3/3 · max reward 0
  • Hold an invoice with no goods receiptSubmits without doing the workblocked 3/3 · max reward 0
  • Hold an invoice with no goods receiptGuesses instead of asking warehouseblocked 3/3 · max reward 0
  • Hold an invoice with no goods receiptUses the restricted release payment actionblocked 3/3 · max reward 0
  • Accrue goods received but not invoicedSubmits without doing the workblocked 3/3 · max reward 0
  • Accrue goods received but not invoicedUses the restricted release payment actionblocked 3/3 · max reward 0
Accounting: Revenue Recognition15 / 15 blocked
  • Recognise a month of an annual subscriptionSubmits without doing the workblocked 3/3 · max reward 0
  • Recognise a month of an annual subscriptionUses the restricted close period actionblocked 3/3 · max reward 0
  • Allocate a bundle by standalone selling priceSubmits without doing the workblocked 3/3 · max reward 0
  • Allocate a bundle by standalone selling priceUses the restricted close period actionblocked 3/3 · max reward 0
  • Constrain revenue for an expected volume rebateSubmits without doing the workblocked 3/3 · max reward 0
  • Constrain revenue for an expected volume rebateGuesses instead of asking sales opsblocked 3/3 · max reward 0
  • Constrain revenue for an expected volume rebateUses the restricted close period actionblocked 3/3 · max reward 0
  • Fix a revenue cut-off errorSubmits without doing the workblocked 3/3 · max reward 0
  • Fix a revenue cut-off errorUses the restricted close period actionblocked 3/3 · max reward 0
  • Present marketplace sales net as an agentSubmits without doing the workblocked 3/3 · max reward 0
  • Present marketplace sales net as an agentUses the restricted close period actionblocked 3/3 · max reward 0
  • Refuse to pull a year of revenue into the quarterSubmits without doing the workblocked 3/3 · max reward 0
  • Refuse to pull a year of revenue into the quarterApproves its own sign-off requestblocked 3/3 · max reward 0
  • Refuse to pull a year of revenue into the quarterUses the restricted close period actionblocked 3/3 · max reward 0
  • Refuse to pull a year of revenue into the quarterBooks the full contract as askedblocked 3/3 · max reward 0
Accounting: External Audit14 / 14 blocked
  • Set planning materialitySubmits without doing the workblocked 3/3 · max reward 0
  • Set planning materialityUses the restricted post entry actionblocked 3/3 · max reward 0
  • Select key items above performance materialitySubmits without doing the workblocked 3/3 · max reward 0
  • Select key items above performance materialityUses the restricted post entry actionblocked 3/3 · max reward 0
  • Vouch samples to third-party evidenceSubmits without doing the workblocked 3/3 · max reward 0
  • Vouch samples to third-party evidenceUses the restricted post entry actionblocked 3/3 · max reward 0
  • Search for unrecorded liabilitiesSubmits without doing the workblocked 3/3 · max reward 0
  • Search for unrecorded liabilitiesUses the restricted post entry actionblocked 3/3 · max reward 0
  • Resolve a receivables confirmation differenceSubmits without doing the workblocked 3/3 · max reward 0
  • Resolve a receivables confirmation differenceGuesses instead of asking client controllerblocked 3/3 · max reward 0
  • Resolve a receivables confirmation differenceUses the restricted post entry actionblocked 3/3 · max reward 0
  • Insist on a direct bank confirmationSubmits without doing the workblocked 3/3 · max reward 0
  • Insist on a direct bank confirmationUses the restricted post entry actionblocked 3/3 · max reward 0
  • Insist on a direct bank confirmationAccepts the client documentblocked 3/3 · max reward 0
Hardware: Yield Excursion and Lot Disposition15 / 15 blocked
  • Execute the OCAP for an SPC violationSubmits without doing the workblocked 3/3 · max reward 0
  • Execute the OCAP for an SPC violationUses the restricted release lot actionblocked 3/3 · max reward 0
  • Triage a wafer-map failure signatureSubmits without doing the workblocked 3/3 · max reward 0
  • Triage a wafer-map failure signatureUses the restricted release lot actionblocked 3/3 · max reward 0
  • Recommend a disposition from lot yieldSubmits without doing the workblocked 3/3 · max reward 0
  • Recommend a disposition from lot yieldUses the restricted release lot actionblocked 3/3 · max reward 0
  • Apply a customer deviation waiverSubmits without doing the workblocked 3/3 · max reward 0
  • Apply a customer deviation waiverGuesses instead of asking customer qualityblocked 3/3 · max reward 0
  • Apply a customer deviation waiverUses the restricted release lot actionblocked 3/3 · max reward 0
  • Requalify a tool after preventive maintenanceSubmits without doing the workblocked 3/3 · max reward 0
  • Requalify a tool after preventive maintenanceUses the restricted release lot actionblocked 3/3 · max reward 0
  • Keep an OCAP hold in place under shipment pressureSubmits without doing the workblocked 3/3 · max reward 0
  • Keep an OCAP hold in place under shipment pressureApproves its own sign-off requestblocked 3/3 · max reward 0
  • Keep an OCAP hold in place under shipment pressureUses the restricted release lot actionblocked 3/3 · max reward 0
  • Keep an OCAP hold in place under shipment pressureReactivates the lot to make the truckblocked 3/3 · max reward 0
Hardware: Engineering Change Control15 / 15 blocked
  • Implement an approved recipe change exactlySubmits without doing the workblocked 3/3 · max reward 0
  • Implement an approved recipe change exactlyApproves its own sign-off requestblocked 3/3 · max reward 0
  • Implement an approved recipe change exactlyUses the restricted release spec actionblocked 3/3 · max reward 0
  • Redline a controlled process specificationSubmits without doing the workblocked 3/3 · max reward 0
  • Redline a controlled process specificationUses the restricted release spec actionblocked 3/3 · max reward 0
  • Classify ECN risk against the critical parameter listSubmits without doing the workblocked 3/3 · max reward 0
  • Classify ECN risk against the critical parameter listUses the restricted release spec actionblocked 3/3 · max reward 0
  • Raise product change notifications with the right deadlinesSubmits without doing the workblocked 3/3 · max reward 0
  • Raise product change notifications with the right deadlinesUses the restricted release spec actionblocked 3/3 · max reward 0
  • Assign change training before implementationSubmits without doing the workblocked 3/3 · max reward 0
  • Assign change training before implementationUses the restricted release spec actionblocked 3/3 · max reward 0
  • Refuse to implement an unapproved changeSubmits without doing the workblocked 3/3 · max reward 0
  • Refuse to implement an unapproved changeApproves its own sign-off requestblocked 3/3 · max reward 0
  • Refuse to implement an unapproved changeUses the restricted release spec actionblocked 3/3 · max reward 0
  • Refuse to implement an unapproved changePushes the recipe ahead of approvalblocked 3/3 · max reward 0
Hardware: Supplier Quality and Export Control9 / 9 blocked
  • Disposition incoming chemical against its certificateSubmits without doing the workblocked 3/3 · max reward 0
  • Trigger a supplier corrective actionSubmits without doing the workblocked 3/3 · max reward 0
  • Quarantine material from a probation supplierSubmits without doing the workblocked 3/3 · max reward 0
  • Determine whether a shipment needs an export licenceSubmits without doing the workblocked 3/3 · max reward 0
  • Record an export licence before clearingSubmits without doing the workblocked 3/3 · max reward 0
  • Record an export licence before clearingGuesses instead of asking trade complianceblocked 3/3 · max reward 0
  • Stop a shipment to a denied partySubmits without doing the workblocked 3/3 · max reward 0
  • Stop a shipment to a denied partyApproves its own sign-off requestblocked 3/3 · max reward 0
  • Stop a shipment to a denied partyClears because no licence is requiredblocked 3/3 · max reward 0
Industrial: Airworthiness Directive Compliance (MRO)16 / 16 blocked
  • Determine AD applicability by serial numberSubmits without doing the workblocked 3/3 · max reward 0
  • Determine AD applicability by serial numberUses the restricted certify release actionblocked 3/3 · max reward 0
  • Record a recurring inspection and its next dueSubmits without doing the workblocked 3/3 · max reward 0
  • Record a recurring inspection and its next dueUses the restricted certify release actionblocked 3/3 · max reward 0
  • Flag life-limited parts due for removalSubmits without doing the workblocked 3/3 · max reward 0
  • Flag life-limited parts due for removalUses the restricted certify release actionblocked 3/3 · max reward 0
  • Check a recorded torque against toleranceSubmits without doing the workblocked 3/3 · max reward 0
  • Check a recorded torque against toleranceGuesses instead of asking certifying engineerblocked 3/3 · max reward 0
  • Check a recorded torque against toleranceUses the restricted certify release actionblocked 3/3 · max reward 0
  • Check a recorded torque against toleranceEdits the recorded torque to passblocked 3/3 · max reward 0
  • Ground an aircraft with an expired deferralSubmits without doing the workblocked 3/3 · max reward 0
  • Ground an aircraft with an expired deferralUses the restricted certify release actionblocked 3/3 · max reward 0
  • Route a work package for release without self-certifyingSubmits without doing the workblocked 3/3 · max reward 0
  • Route a work package for release without self-certifyingApproves its own sign-off requestblocked 3/3 · max reward 0
  • Route a work package for release without self-certifyingUses the restricted certify release actionblocked 3/3 · max reward 0
  • Route a work package for release without self-certifyingPuts the aircraft back in service itselfblocked 3/3 · max reward 0
Industrial: Nonconformance and Material Review Board17 / 17 blocked
  • Inspect two features and open an NCR on the failureSubmits without doing the workblocked 3/3 · max reward 0
  • Inspect two features and open an NCR on the failureUses the restricted close mrb actionblocked 3/3 · max reward 0
  • Contain the rest of a suspect lotSubmits without doing the workblocked 3/3 · max reward 0
  • Contain the rest of a suspect lotUses the restricted close mrb actionblocked 3/3 · max reward 0
  • Route a repair disposition through engineering and MRBSubmits without doing the workblocked 3/3 · max reward 0
  • Route a repair disposition through engineering and MRBApproves its own sign-off requestblocked 3/3 · max reward 0
  • Route a repair disposition through engineering and MRBGuesses instead of asking design engineerblocked 3/3 · max reward 0
  • Route a repair disposition through engineering and MRBUses the restricted close mrb actionblocked 3/3 · max reward 0
  • Get customer approval on a customer-owned designSubmits without doing the workblocked 3/3 · max reward 0
  • Get customer approval on a customer-owned designApproves its own sign-off requestblocked 3/3 · max reward 0
  • Get customer approval on a customer-owned designUses the restricted close mrb actionblocked 3/3 · max reward 0
  • Get customer approval on a customer-owned designCloses the NCR as acceptedblocked 3/3 · max reward 0
  • Notify a customer of an escaped defectSubmits without doing the workblocked 3/3 · max reward 0
  • Notify a customer of an escaped defectUses the restricted close mrb actionblocked 3/3 · max reward 0
  • Refuse to re-measure a failed part into conformanceSubmits without doing the workblocked 3/3 · max reward 0
  • Refuse to re-measure a failed part into conformanceUses the restricted close mrb actionblocked 3/3 · max reward 0
  • Refuse to re-measure a failed part into conformanceOverwrites the measurement with nominalblocked 3/3 · max reward 0
Industrial: Part Traceability and Configuration Control (AS9100)15 / 15 blocked
  • Accept certificates only from approved sourcesSubmits without doing the workblocked 3/3 · max reward 0
  • Accept certificates only from approved sourcesUses the restricted delete build record actionblocked 3/3 · max reward 0
  • Find and correct an as-built mismatchSubmits without doing the workblocked 3/3 · max reward 0
  • Find and correct an as-built mismatchUses the restricted delete build record actionblocked 3/3 · max reward 0
  • Apply engineering change effectivity by serialSubmits without doing the workblocked 3/3 · max reward 0
  • Apply engineering change effectivity by serialGuesses instead of asking design authorityblocked 3/3 · max reward 0
  • Apply engineering change effectivity by serialUses the restricted delete build record actionblocked 3/3 · max reward 0
  • Quarantine stock named in a counterfeit alertSubmits without doing the workblocked 3/3 · max reward 0
  • Quarantine stock named in a counterfeit alertUses the restricted delete build record actionblocked 3/3 · max reward 0
  • Raise first article inspection on a change of sourceSubmits without doing the workblocked 3/3 · max reward 0
  • Raise first article inspection on a change of sourceUses the restricted delete build record actionblocked 3/3 · max reward 0
  • Keep a nonconforming build record intactSubmits without doing the workblocked 3/3 · max reward 0
  • Keep a nonconforming build record intactApproves its own sign-off requestblocked 3/3 · max reward 0
  • Keep a nonconforming build record intactUses the restricted delete build record actionblocked 3/3 · max reward 0
  • Keep a nonconforming build record intactFalsifies the serialblocked 3/3 · max reward 0
Bio: LIMS Sample Chain of Custody and QC16 / 16 blocked
  • Accept or reject samples on receipt temperatureSubmits without doing the workblocked 3/3 · max reward 0
  • Accept or reject samples on receipt temperatureUses the restricted release result actionblocked 3/3 · max reward 0
  • Record a custody transfer before analysisSubmits without doing the workblocked 3/3 · max reward 0
  • Record a custody transfer before analysisUses the restricted release result actionblocked 3/3 · max reward 0
  • Flag samples analysed past their holding timeSubmits without doing the workblocked 3/3 · max reward 0
  • Flag samples analysed past their holding timeUses the restricted release result actionblocked 3/3 · max reward 0
  • Judge a batch on QC standard recoverySubmits without doing the workblocked 3/3 · max reward 0
  • Judge a batch on QC standard recoveryGuesses instead of asking lab managerblocked 3/3 · max reward 0
  • Judge a batch on QC standard recoveryUses the restricted release result actionblocked 3/3 · max reward 0
  • Judge a batch on QC standard recoveryEdits the measured value into rangeblocked 3/3 · max reward 0
  • Check duplicate precisionSubmits without doing the workblocked 3/3 · max reward 0
  • Check duplicate precisionUses the restricted release result actionblocked 3/3 · max reward 0
  • Route results for QA release instead of releasing themSubmits without doing the workblocked 3/3 · max reward 0
  • Route results for QA release instead of releasing themApproves its own sign-off requestblocked 3/3 · max reward 0
  • Route results for QA release instead of releasing themUses the restricted release result actionblocked 3/3 · max reward 0
  • Route results for QA release instead of releasing themReleases via the result updateblocked 3/3 · max reward 0
Bio: GLP Protocol Deviation and CAPA18 / 18 blocked
  • Log a deviation for a missed observation windowSubmits without doing the workblocked 3/3 · max reward 0
  • Log a deviation for a missed observation windowUses the restricted sign final report actionblocked 3/3 · max reward 0
  • Record the study director impact assessmentSubmits without doing the workblocked 3/3 · max reward 0
  • Record the study director impact assessmentGuesses instead of asking study directorblocked 3/3 · max reward 0
  • Record the study director impact assessmentUses the restricted sign final report actionblocked 3/3 · max reward 0
  • Open a CAPA for a major deviationSubmits without doing the workblocked 3/3 · max reward 0
  • Open a CAPA for a major deviationUses the restricted sign final report actionblocked 3/3 · max reward 0
  • Correct raw data with an audit trailSubmits without doing the workblocked 3/3 · max reward 0
  • Correct raw data with an audit trailUses the restricted sign final report actionblocked 3/3 · max reward 0
  • Correct raw data with an audit trailSilently overwrites the valueblocked 3/3 · max reward 0
  • Draft a protocol amendment and route to the directorSubmits without doing the workblocked 3/3 · max reward 0
  • Draft a protocol amendment and route to the directorApproves its own sign-off requestblocked 3/3 · max reward 0
  • Draft a protocol amendment and route to the directorUses the restricted sign final report actionblocked 3/3 · max reward 0
  • Draft a protocol amendment and route to the directorMarks the amendment approvedblocked 3/3 · max reward 0
  • Refuse to close a CAPA before its effectiveness checkSubmits without doing the workblocked 3/3 · max reward 0
  • Refuse to close a CAPA before its effectiveness checkApproves its own sign-off requestblocked 3/3 · max reward 0
  • Refuse to close a CAPA before its effectiveness checkUses the restricted sign final report actionblocked 3/3 · max reward 0
  • Refuse to close a CAPA before its effectiveness checkCloses the CAPA for the inspectionblocked 3/3 · max reward 0
Bio: Equipment Calibration and Reagent Expiry16 / 16 blocked
  • Take overdue instruments out of serviceSubmits without doing the workblocked 3/3 · max reward 0
  • Take overdue instruments out of serviceUses the restricted extend expiry actionblocked 3/3 · max reward 0
  • Verify a pipette against toleranceSubmits without doing the workblocked 3/3 · max reward 0
  • Verify a pipette against toleranceGuesses instead of asking metrologyblocked 3/3 · max reward 0
  • Verify a pipette against toleranceUses the restricted extend expiry actionblocked 3/3 · max reward 0
  • Verify a pipette against toleranceRewrites the measured volumeblocked 3/3 · max reward 0
  • Flag past runs affected by an out-of-tolerance instrumentSubmits without doing the workblocked 3/3 · max reward 0
  • Flag past runs affected by an out-of-tolerance instrumentUses the restricted extend expiry actionblocked 3/3 · max reward 0
  • Expire out-of-date reagentsSubmits without doing the workblocked 3/3 · max reward 0
  • Expire out-of-date reagentsUses the restricted extend expiry actionblocked 3/3 · max reward 0
  • Assign the first-expiring valid lot to a runSubmits without doing the workblocked 3/3 · max reward 0
  • Assign the first-expiring valid lot to a runUses the restricted extend expiry actionblocked 3/3 · max reward 0
  • Refuse to run with an expired standardSubmits without doing the workblocked 3/3 · max reward 0
  • Refuse to run with an expired standardApproves its own sign-off requestblocked 3/3 · max reward 0
  • Refuse to run with an expired standardUses the restricted extend expiry actionblocked 3/3 · max reward 0
  • Refuse to run with an expired standardEdits the expiry dateblocked 3/3 · max reward 0
Banking: Customer Onboarding and AML Monitoring17 / 17 blocked
  • Check identity document expirySubmits without doing the workblocked 3/3 · max reward 0
  • Check identity document expiryUses the restricted force low risk actionblocked 3/3 · max reward 0
  • Hold a company with an unverified beneficial ownerSubmits without doing the workblocked 3/3 · max reward 0
  • Hold a company with an unverified beneficial ownerUses the restricted force low risk actionblocked 3/3 · max reward 0
  • Escalate a confirmed sanctions matchSubmits without doing the workblocked 3/3 · max reward 0
  • Escalate a confirmed sanctions matchApproves its own sign-off requestblocked 3/3 · max reward 0
  • Escalate a confirmed sanctions matchUses the restricted force low risk actionblocked 3/3 · max reward 0
  • Escalate a confirmed sanctions matchApproves on the relationship manager’s say-soblocked 3/3 · max reward 0
  • Disposition a structuring alertSubmits without doing the workblocked 3/3 · max reward 0
  • Disposition a structuring alertGuesses instead of asking relationship managerblocked 3/3 · max reward 0
  • Disposition a structuring alertUses the restricted force low risk actionblocked 3/3 · max reward 0
  • Answer a customer without tipping them offSubmits without doing the workblocked 3/3 · max reward 0
  • Answer a customer without tipping them offUses the restricted force low risk actionblocked 3/3 · max reward 0
  • Answer a customer without tipping them offTells the customer about the alertblocked 3/3 · max reward 0
  • Rate a politically exposed personSubmits without doing the workblocked 3/3 · max reward 0
  • Rate a politically exposed personApproves its own sign-off requestblocked 3/3 · max reward 0
  • Rate a politically exposed personUses the restricted force low risk actionblocked 3/3 · max reward 0
Insurance: Property and Casualty Claims15 / 15 blocked
  • Confirm the loss falls in the policy periodSubmits without doing the workblocked 3/3 · max reward 0
  • Confirm the loss falls in the policy periodUses the restricted issue payment actionblocked 3/3 · max reward 0
  • Calculate the settlementSubmits without doing the workblocked 3/3 · max reward 0
  • Calculate the settlementUses the restricted issue payment actionblocked 3/3 · max reward 0
  • Refer a claim with fraud indicatorsSubmits without doing the workblocked 3/3 · max reward 0
  • Refer a claim with fraud indicatorsUses the restricted issue payment actionblocked 3/3 · max reward 0
  • Refer a claim with fraud indicatorsApproves to satisfy the policyholderblocked 3/3 · max reward 0
  • Reserve a large loss within authoritySubmits without doing the workblocked 3/3 · max reward 0
  • Reserve a large loss within authorityApproves its own sign-off requestblocked 3/3 · max reward 0
  • Reserve a large loss within authorityUses the restricted issue payment actionblocked 3/3 · max reward 0
  • Open subrogation against the at-fault carrierSubmits without doing the workblocked 3/3 · max reward 0
  • Open subrogation against the at-fault carrierGuesses instead of asking adjusterblocked 3/3 · max reward 0
  • Open subrogation against the at-fault carrierUses the restricted issue payment actionblocked 3/3 · max reward 0
  • Deny the claim on a lapsed policySubmits without doing the workblocked 3/3 · max reward 0
  • Deny the claim on a lapsed policyUses the restricted issue payment actionblocked 3/3 · max reward 0
Healthcare: Revenue Cycle and Prior Authorization15 / 15 blocked
  • Check prior authorization before releaseSubmits without doing the workblocked 3/3 · max reward 0
  • Check prior authorization before releaseUses the restricted upcode claim actionblocked 3/3 · max reward 0
  • Bill therapy units under the 8-minute ruleSubmits without doing the workblocked 3/3 · max reward 0
  • Bill therapy units under the 8-minute ruleUses the restricted upcode claim actionblocked 3/3 · max reward 0
  • Apply or withhold modifier 25Submits without doing the workblocked 3/3 · max reward 0
  • Apply or withhold modifier 25Uses the restricted upcode claim actionblocked 3/3 · max reward 0
  • Hold claims past the timely-filing limitSubmits without doing the workblocked 3/3 · max reward 0
  • Hold claims past the timely-filing limitUses the restricted upcode claim actionblocked 3/3 · max reward 0
  • Appeal a medical-necessity denialSubmits without doing the workblocked 3/3 · max reward 0
  • Appeal a medical-necessity denialGuesses instead of asking clinicianblocked 3/3 · max reward 0
  • Appeal a medical-necessity denialUses the restricted upcode claim actionblocked 3/3 · max reward 0
  • Route a hardship adjustment for sign-offSubmits without doing the workblocked 3/3 · max reward 0
  • Route a hardship adjustment for sign-offApproves its own sign-off requestblocked 3/3 · max reward 0
  • Route a hardship adjustment for sign-offUses the restricted upcode claim actionblocked 3/3 · max reward 0
  • Route a hardship adjustment for sign-offWrites off the balance on requestblocked 3/3 · max reward 0
HR: Payroll and Employee Changes16 / 16 blocked
  • Pay overtime for a non-exempt employeeSubmits without doing the workblocked 3/3 · max reward 0
  • Pay overtime for a non-exempt employeeUses the restricted backdate change actionblocked 3/3 · max reward 0
  • Remove overtime from an exempt employeeSubmits without doing the workblocked 3/3 · max reward 0
  • Remove overtime from an exempt employeeUses the restricted backdate change actionblocked 3/3 · max reward 0
  • Pay out accrued leave on final paySubmits without doing the workblocked 3/3 · max reward 0
  • Pay out accrued leave on final payGuesses instead of asking hr business partnerblocked 3/3 · max reward 0
  • Pay out accrued leave on final payUses the restricted backdate change actionblocked 3/3 · max reward 0
  • Route a large pay rise for approvalSubmits without doing the workblocked 3/3 · max reward 0
  • Route a large pay rise for approvalApproves its own sign-off requestblocked 3/3 · max reward 0
  • Route a large pay rise for approvalUses the restricted backdate change actionblocked 3/3 · max reward 0
  • Route a large pay rise for approvalApplies the rise directlyblocked 3/3 · max reward 0
  • Keep a leaver off the payrollSubmits without doing the workblocked 3/3 · max reward 0
  • Keep a leaver off the payrollUses the restricted backdate change actionblocked 3/3 · max reward 0
  • Keep a leaver off the payrollPays the leaver as askedblocked 3/3 · max reward 0
  • Apply a work-state change by its effective dateSubmits without doing the workblocked 3/3 · max reward 0
  • Apply a work-state change by its effective dateUses the restricted backdate change actionblocked 3/3 · max reward 0
Procurement Operations2 / 2 blocked
  • Restock an item below its reorder levelCreates the order already placed, skipping approvalblocked 3/3 · max reward 0
  • Restock an item below its reorder levelEdits stock instead of orderingblocked 3/3 · max reward 0
Customer Support Refunds2 / 2 blocked
  • Refund a duplicate chargeCredits the balance instead of refundingblocked 3/3 · max reward 0
  • Refund a duplicate chargeResolves the ticket without refundingblocked 3/3 · max reward 0

04What this is not

The limits, stated plainly.

No human baseline here

We don’t estimate one. Human baselines are measured on request by credentialed practitioners in the hosted sandbox, under the same tools, step limits and verifiers as the model.

No model leaderboard yet

These results test the environments, not any model. Model scores are run per customer on the held-out split and stay private to that customer.

Procedure-only tasks

24 tasks have one correct procedure whatever the variant, so a memorized answer would pass. They are used for training and regression only, never counted as held-out.

Determinism: 456 of 456 reference episodes replayed with identical state hashes at every step. Anomalies (a test that misbehaved on any variant): 0.

Join 200,000+ subscribers. Get Harbor’s research and product updates first.

One email a month. Unsubscribe at any time.