Image Ranker: Fast Open-Source Pairwise Ranking for Human Preference Learning using TrueSkill

Read full text Summarise with VIRAT
Authors: Quentin Wach
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Image Ranker is an open-source tool for efficiently ranking large collections of images through pairwise comparisons using a Bayesian TrueSkill-based ranking algorithm. It enables fast and scalable human-in-the-loop preference estimation without requiring exhaustive comparisons. The system maintains probabilistic ratings for each image, modeling both estimated quality (μ) and uncertainty (σ), and updates rankings globally after each comparison. This allows accurate ranking from incomplete and noisy comparison data. To further accelerate ranking on large datasets, Image Ranker includes several optimization strategies: Sequential elimination, reducing ranking complexity from O(N²) to O(N) Uncertainty-driven sampling (smart shuffle) to prioritize informative comparisons Auto-shuffling for continuous ranking efficiency The tool provides a lightweight web-based interface that allows users to: Compare images directly from local directories without uploading data Track and export rankings (CSV) with resume capability Annotate exclusions with structured reasons Attach contextual metadata to images Image Ranker is particularly useful for: Human preference data collection for reinforcement learning from human feedback (RLHF) Evaluation and selection of generative model outputs (e.g. diffusion models) Dataset curation and ranking under limited labeling budgets By combining probabilistic ranking with efficient comparison strategies, Image Ranker enables scalable preference estimation workflows for modern machine learning and data-centric applications.

Fields of Study & Subjects

Recommender Systems and Techniques Information Retrieval and Search Behavior Machine Learning and Data Classification

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
27 March 2026
Language
EN