LLMs for Qualitative Data Analysis Fail on Security-specific Comments in Human Experiments

[Background:] Thematic analysis of free-text justifications in human experiments provides significant qualitative insights. Yet, it is costly because reliable annotations require multiple domain experts. Large language models (LLMs) seem ideal candidates to replace human annotators. [Problem:] Coding security-specific aspects (code identifiers mentioned, lines-of-code-mentioned, security keywords mentioned) may require deeper contextual understanding than sentiment classification. [Objective:] explore whether LLMs can act as automated annotators for technical security comments by human subjects. [Method:] We prompt four best LLMs on LiveBench to detect nine security-relevant codes in free-text comments by human subjects analyzing vulnerable code snippets. Outputs are compared to the human annotators along with Cohen’s Kappa (chance-corrected accuracy). We test different prompts mimicking annotation best practices: emerging codes, a detailed codebook with examples, and conflicting examples. [Negative Results:] We observed marked improvements only with the code descriptions, but they are not uniform across codes and not sufficient to reliably replace a human annotator. [Limitations:] Additional studies with more LLMs and annotation tasks are needed.