task-007
Existing defectPythonLeft out of official scoresThe failure the developer caughtThe /api/v1/query/search implementation still returned no results for class GitHubRepo(BaseModel) because it searched the whole declaration as a phrase or node-name substring without falling back to file-content search.
The v1 baseline on this task
Left out of official scores: The claim-by-claim check failed one reading of the accepted answer. The baseline did not run it.
What the agent gets
- The conversation up to just before the original agent’s failing answer: 214,790 characters. Too long for one instruction, so long tool outputs are cut to fit and marked; the whole conversation is in its container.
- The repository as it stood, BugViper, with its history and its dependencies installed.
- Its own tools, and the main model providers’ APIs, but nothing else online.
How it is checked
Existing defect. The problem is already in the repository, and the agent must notice it.
The defect check: file found. The named file was found in the starting files; this alone does not prove the defect.
The judge, gpt-6-astra, reads the final report against everything the agent did, three times, and the majority settles it. How the grading works