BiO-HMC: Dynamic Human-Machine Collaboration for Consensus Decision-Making via Bilevel Optimization

Consensus decision-making uses crowd responses (usually non-experts) to questions to reach a consensus answer based on human-machine collaboration. The crucial point is dynamic, which should not only enable rapid self-iteration toward the correct answer through crowd workers' responses but also adaptively suggest the next most valuable question(s) to speed up the answer integration. However, existing methods reach consensus answers based on either offline data or fixed search structures of questions, where the dynamic nature is largely left aside. In response, we propose a bilevel optimization-based human-machine collaboration (BiO-HMC), which explores an inner & outer-level optimization to enable effective answer integration and efficient question selection. The resulting optimization problem is intractable because there is no closed form expression in the inner-level optimization. We employ a gradient-based method and guarantee the method's theoretical convergence. Experimental results on synthetic and real-world data sets demonstrate the effectiveness and efficiency of the BiO-HMC model, i.e., achieving the highest confidence in the correct answer with the lowest labor cost.