Exploring humans and LLMs collaboration in the data science pipeline

The modern data science pipeline — spanning data preparation, exploration, feature engineering, modeling, and evaluation — has traditionally relied on human expertise to ensure high-quality, context-aware insights. However, the proliferation of unstructured and multimodal data, along with the potential of Large Language Models (LLMs) for automating complex tasks requiring reasoning and contextual understanding, has outpaced the scalability of human-driven workflows, necessitating new forms of intelligent automation. This work envisions a human-LLM collaborative framework in which both humans and LLMs are treated as noisy oracles — agents with uncertain, context-sensitive, and variable-cost responses. We propose a multi-agent orchestration framework that integrates these oracles to explore trade-offs in cost and output quality throughout the pipeline. Our overarching goal is to maximize end-to-end pipeline utility through principled coordination of humans and LLMs, paving the way for a new generation of data science pipelines.