What should we request to make a sample technically reviewable?

Use this checklist to request a sample that your evaluation team can inspect in its actual technical workflow.

- Request representative file types, including attachments that require text extraction or rendering. - Ask for encoding, language and date-format conventions in a data dictionary. - Specify whether tables must remain readable without access to spreadsheet software. - Request declared limits for document length, attachment depth and tool response size. - Include a task that requires finding evidence inside a long record. - Ask for a known-valid submission to exercise your output parser. - Record whether text-only agents receive equivalent access to material in images.

These checks concern usability within your harness. A well-formed dataset can still be unsuitable if its important evidence is hidden in an attachment your tools cannot open, or if a tool response exceeds the context available to the agent.

Run the sample through ingestion and a complete task attempt before requesting broader coverage. Record extraction failures separately from reasoning failures. Where access differs between text-only and multimodal configurations, make that difference explicit in the evaluation design so comparisons remain interpretable and do not accidentally measure incompatible input pipelines.