Learning to Detect Hardcoded Secrets

Many tools exist to detect secrets in source code, but rule-based approaches often produce high false positives. Our study shows that for non-generative models, performance depends heavily on the chosen features, with contextual code information being the most effective. Among these, CodeBERT performed best, achieving 88% MCC and 89% F1, and along with Random Forest remained strong on unseen data (~65% MCC). Generative models showed moderate accuracy and more false positives, as they tend to over-flag potential secrets to encourage secure coding practices.