Masking: How an AI Tool's Own Feedback Pipeline Manufactures Content-Empty Bug Reports That Naive Quality Metrics Cannot See
Abstract
On Anthropic's public anthropics/claude-code issue tracker, ~22.7% of new issues in a one-week window arrive via the product's own in-app /bug pipeline, which auto-fills environment/error fields and auto-generates the title with an LLM. When the human writes no bug content, the template still yields a fully-formed artifact that a scheduled bot later closes NOT_PLANNED. ~1.5% (95% CI 0.8-2.8) carry no human bug content yet pass every naive quality check - "masking". Two control claims are kept strictly separate: isolation (detectable metric: claude-code 1.5% vs controls 0%) and masking (naive metric: claude-code 0.0% vs vscode 0.8%). De-identified data and code are released for full reproduction; raw identifier-bearing data is withheld to protect individuals.
Fields of Study & Subjects
Software Engineering Research
Ethics and Social Impacts of AI
Scientific Computing and Data Management
Publication details
- Journal / source
- OSF Preprints (OSF Preprints)
- Publication date
- 1 January 2026
- Document type
- Preprint