Image Ranker: Fast Open-Source Pairwise Ranking for Human Preference Learning using TrueSkill
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
Image Ranker is an open-source tool for efficiently ranking large collections of images through pairwise comparisons using a Bayesian TrueSkill-based ranking algorithm. It enables fast and scalable human-in-the-loop preference estimation without requiring exhaustive comparisons. The system maintains probabilistic ratings for each image, modeling both estimated quality (μ) and uncertainty (σ), and updates rankings globally after each comparison. This allows accurate ranking from incomplete and noisy comparison data. To further accelerate ranking on large datasets, Image Ranker includes several optimization strategies: Sequential elimination, reducing ranking complexity from O(N²) to O(N) Uncertainty-driven sampling (smart shuffle) to prioritize informative comparisons Auto-shuffling for continuous ranking efficiency The tool provides a lightweight web-based interface that allows users to: Compare images directly from local directories without uploading data Track and export rankings (CSV) with resume capability Annotate exclusions with structured reasons Attach contextual metadata to images Image Ranker is particularly useful for: Human preference data collection for reinforcement learning from human feedback (RLHF) Evaluation and selection of generative model outputs (e.g. diffusion models) Dataset curation and ranking under limited labeling budgets By combining probabilistic ranking with efficient comparison strategies, Image Ranker enables scalable preference estimation workflows for modern machine learning and data-centric applications.
Fields of Study & Subjects
Recommender Systems and Techniques
Information Retrieval and Search Behavior
Machine Learning and Data Classification
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 27 March 2026
- Language
- EN