Masking: How an AI Tool's Own Feedback Pipeline Manufactures Content-Empty Bug Reports That Naive Quality Metrics Cannot See

Read full text
Authors: H. Tamba
Cited by: 2

Abstract

On Anthropic's public anthropics/claude-code issue tracker, ~22.7% of new issues in a one-week window arrive via the product's own in-app /bug pipeline, which auto-fills environment/error fields and auto-generates the title with an LLM. When the human writes no bug content, the template still yields a fully-formed artifact that a scheduled bot later closes NOT_PLANNED. ~1.5% (95% CI 0.8-2.8) carry no human bug content yet pass every naive quality check - "masking". Two control claims are kept strictly separate: isolation (detectable metric: claude-code 1.5% vs controls 0%) and masking (naive metric: claude-code 0.0% vs vscode 0.8%). De-identified data and code are released for full reproduction; raw identifier-bearing data is withheld to protect individuals.

Fields of Study & Subjects

Software Engineering Research Ethics and Social Impacts of AI Scientific Computing and Data Management

Publication details

Journal / source
OSF Preprints (OSF Preprints)
Publication date
1 January 2026
Document type
Preprint