Evaluating multimodal commercial and open-source large language models for dynamical astronomy: a benchmark study of resonant behavior classification
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
We present a systematic evaluation of modern multimodal large language models (LLMs) for the classification of mean-motion and secular resonances from images of resonant arguments. Four benchmark datasets (RB-TEST, RB-PILOT, RB-SMALL, RB-FULL) were constructed to cover clear, ambiguous, and transient cases, with both binary and three-class outputs. Using standardized prompts (a full prompt for large models and a simplified variant for small models that cannot process complex instructions), we tested flagship commercial models, large open-source models, and small locally runnable models. Commercial LLMs reach [Formula: see text] on simple cases and up to [Formula: see text] on the three-class RB-SMALL dataset, while the best open-source models also reach [Formula: see text] on unambiguous cases and [Formula: see text] on the complex ones. On the full binary benchmark, open-source models approach commercial performance ([Formula: see text]-[Formula: see text]). Most errors occur in transient and resonance-sticking regimes. The results show that LLMs can perform resonance classification at levels comparable to those of classical or machine-learning methods without training or fine-tuning, and that even small open-source models achieve practically useful accuracy. The released benchmarks establish a reproducible standard for evaluating LLMs on dynamical astronomy tasks.
Fields of Study & Subjects
Stellar
planetary
and galactic studies
Astro and Planetary Science
Topic Modeling
Publication details
- Journal / source
- Scientific Reports
- Volume
- 16
- Issue
- 1
- Publication date
- 28 March 2026
- Language
- EN