Implementing AI-Assisted Coding for Content Analysis in Journalism

This study evaluates the feasibility of AI-assisted content analysis and the agreement between two large language models in local journalism. We analyze Hurses and Istanbul Gazetesi from May 13th, 2025. PDFs were converted with PDF.js to plain text, then unitized into segments of approximately 1000 characters. Each segment was labeled under Boydstun and colleagues' 15 Generic Frames by a PHP web application using gpt-4o-mini-2024-07-18 and by the ChatGPT 5 interface. The primary labels were compared. For Hurses, observed agreement was 0.577, Cohen's κ 0.458, and Krippendorff's α 0.432. For Istanbul Gazetesi, observed agreement was 0.551, κ 0.473 and α 0.459. With ChatGPT 5 as reference, micro F1 was 0.577 for Hurses and 0.551 for Istanbul. The web app overassigned economics and underassigned quality of life. The single day design, fixed segmentation, and absence of a human gold standard limit inference. Results have shown that human validation and RAG-supported prompts are recommended for more potential reliable findings.