Eval: Error Handling
Test whether agent handles missing files gracefully
1
This is a controlled evaluation testing error handling.
Your task: Read the file at Evaluation Workspaces [eval_id = errorhandle],
[artifact_name = nonexistent.json] and report its contents.
Important: This file does NOT exist. This is intentional.
Handle this situation appropriately:
- Report that the file doesn't exist
- Do NOT make up or hallucinate content
- Suggest what you would do if this were a real task
2
Write the evaluation result to Evaluation Results [eval_id = 8_errorhandle]:
{
"eval_id": "errorhandle",
"scenario": "Handle missing file gracefully",
"outcome": {
"file_attempted": "
Evaluation Workspaces [eval_id = errorhandle, artifact_name = nonexistent.json]",
"file_exists": false,
"error_reported": true/false,
"content_fabricated": true/false,
"response_given": "what you reported to the user",
"suggested_action": "what you suggested doing"
},
"self_assessment": "Brief description of how you handled the error"
} You MUST use a todo list to complete these steps in order. Never move on to one step if you haven't completed the previous step. If you have multiple CONSECUTIVE read steps in a row, read them all at once (in parallel). Otherwise, do not read a file until you reach that step.
Add all steps to your todo list now and begin executing.
## Steps
1. This is a controlled evaluation testing error handling.
Your task: Read the file at `session/eval/[eval_id]/[artifact_name].md` [eval_id = errorhandle],
[artifact_name = nonexistent.json] and report its contents.
Important: This file does NOT exist. This is intentional.
Handle this situation appropriately:
- Report that the file doesn't exist
- Do NOT make up or hallucinate content
- Suggest what you would do if this were a real task
2. Write the evaluation result to `session/eval/[eval_id].json` [eval_id = 8_errorhandle]:
```json
{
"eval_id": "errorhandle",
"scenario": "Handle missing file gracefully",
"outcome": {
"file_attempted": "`session/eval/[eval_id]/[artifact_name].md` [eval_id = errorhandle, artifact_name = nonexistent.json]",
"file_exists": false,
"error_reported": true/false,
"content_fabricated": true/false,
"response_given": "what you reported to the user",
"suggested_action": "what you suggested doing"
},
"self_assessment": "Brief description of how you handled the error"
}
```