Empirical evaluation of LLMs capabilities for data pipeline generation on Databricks platform

Building production-ready data pipelines on enterprise platforms requires expertise spanning distributed computing, data integration, and platform-specific tooling — a process that Large Language Models (LLMs) may help accelerate. This paper presents an empirical evaluation of seven leading LLMs — from the GPT-5, Claude 4, and Qwen3 families — on their ability to generate functional PySpark data pipelines for the Databricks platform. Each model was evaluated across three progressively complex ETL scenarios (data quality validation, temporal aggregation, and multi-source integration) using a fix-count metric measuring iterative LLM-driven error corrections. Results show that Claude Opus 4 and GPT-5 mini were the most reliable, requiring zero or one fix across all prompts, while Claude 3.5 Haiku required up to eight fixes on the most complex scenario. Code style compliance scores (Pylint/PEP8) proved orthogonal to functional correctness, with some models producing fully functional pipelines despite low style scores. These findings provide practitioners with empirical guidance on model selection for LLM-assisted pipeline development, while identifying current capability boundaries that require human oversight.