What technical steps should we take in the first week with a pilot sample?

Treat the first working session with a sample as an integration test of the material, not just a first evaluation run. A sensible order is below.

1. **Validate the schema.** Load every table, check types, and confirm that each foreign key resolves. Report orphans back to the provider. 2. **Check the keys.** Solve a subset by hand and record disagreements with their case identifiers. 3. **Map tool actions.** Connect actions such as `lookup_party`, `check_completeness`, `request_information` and `route_case` to your harness, or to a thin wrapper over the case tables. Keep the action names stable. 4. **Confirm reset.** Run one task twice from the same `snapshot_id` and seed, then compare a hash of the starting tables and of the final state. They should match when the agent's actions match. 5. **Run a baseline.** Use a trivial agent, such as always routing to the most common queue, and a simple rule-based one. Weak scores confirm that the pack is not trivially solvable. 6. **Run your agent.** Log traces in JSONL and grade outcome, process and notes separately. 7. **Record versions.** Store the release identifier, schema version and grader version with every result.

Share the baseline and the disagreements with the provider. That gives both sides a concrete basis for revising the sample and deciding whether to widen the scope.