AxPic/poem-int-eval: v1.0.0 - Paper Publication Release

This release archives the complete research repository for the paper "Interpretation, Argument, Evaluation: A Workflow for Assessing LLM-Generated Interpretations of Poetry." Paper DOI: https://doi.org/10.48694/jcls.4312 This repository contains all resources, data, and code used to examine how large language models can generate and be evaluated on literary interpretations of poetry according to standards from literary studies. Contents: Jupyter notebooks for interpretation generation and evaluation (Interpretation_Generation.ipynb, Analysis_Notebook.ipynb) Annotated data files with human annotations and Likert scale ratings Methods for calculating Inter-Annotator Agreement metrics (Cohen's Kappa, Percent Agreement) Statistical analysis tools for evaluating generated interpretations The workflow developed in this research draws on reference data from literary studies and argumentative structures to generate interpretations that can be evaluated for both argumentative coherence and adherence to literary scholarship standards. Citation: If you use this repository in your research, please cite the associated paper: "Interpretation, Argument, Evaluation: A Workflow for Assessing LLM-Generated Interpretations of Poetry"