Pact Benchmark: ICPC World Finals — Contract-First Multi-Agent vs Single-Agent Code Generation

Read full text
Authors: Jeremy McEntire
Cited by: 0

Abstract

Benchmark comparing Pact (contract-first multi-agent framework) against Claude Code on 5 ICPC World Finals competitive programming problems (212 test cases). Pact achieves 100% (212/212) vs Claude Code single-shot 79% (167/212) and iterative 92% (196/212). All conditions use Claude Opus 4.6. Includes test data, baseline results, full Pact state for both conditions (research and base), and reproduction scripts. The decisive problem is Trailing Digits (2020 World Finals): Claude Code scores 31/47 even with 5 retry iterations — the naive algorithm times out. Pact's interview and decomposition phases force upfront mathematical analysis, producing the correct O(log n) approach on the first attempt.

Fields of Study & Subjects

Teaching and Learning Programming Software Engineering Research Multi-Agent Systems and Negotiation

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
4 March 2026
Language
EN
Document type
Dataset