ArvinRastegar/i-adopt-llm-based-service: V1.2-Experiment

Results configuration-results.csv — every configuration we ran, with its scores. Plain CSV, no setup, GitHub renders it as a sortable table. 298 rows, sorted by Close F1 descending. Each row is one model configuration scored over the same 97 variables: model_id, provider, prompt, shots, temperature, reasoning, then precision / recall / F1 in both scoring modes (Close = similarity ≥ 0.8, Exact = identical text). What this release adds iadopt-lab/ — a self-contained experiment system that measures how well language models decompose scientific variable definitions into I-ADOPT components (property, object of interest, matrix, context object, statistical modifier, constraints). What we did Took the 102 variables of Corpus v2.0.1, kept 5 as worked examples, evaluated on the other 97. Asked each model for a structured JSON decomposition. Invalid JSON is sent back with the exact errors, up to 3 tries. Swept a grid: 3 prompt styles × 0/1/3/5 examples × temperature 0/0.5/1, over 9 model deployments on two providers (PSNC self-hosted, OpenRouter). Scored every answer against the gold decomposition, micro-averaged over the population. Three campaigns, 38,412 tasks, almost $30 total (PSNC was not billed). Every prompt, raw response, retry, prediction and score was kept in PostgreSQL, so each number traces back to the evidence that produced it. What we found Best: z-ai/glm-5.2, matrix-decomposition prompt, 5 examples, T=0.5, reasoning on — Close F1 0.4206. The number of examples matters most, and it has not flattened at 5 (0 shots → 0.20, 5 shots → 0.30). Prompt style is a weak effect; temperature barely matters. Cheap models stay close. ministral-8b reaches 86% of the best score for 1/36 of the cost. Hardest fields: matrix and context object. Easiest: property and object of interest. Worth knowing Scores are low in absolute terms (best ≈ 0.42 Close F1). This is a hard task, and output still needs human review. The CSV holds the 298 configurations that completed all 97 variables. A further 98 lost variables to provider rate limits or output truncation (even with 3 retries) and have no comparable score.