An Airflow-Orchestrated AI Pipeline for Podcast Transcription, Topic Modeling, and Recommendation System

This study presents a production-ready AI pipeline for audio content processing, implemented within the Youth Radio platform, which serves as an extension of the European School Radio initiative. The system uses a multi-server architecture: an AI Server that runs batch/offline jobs, orchestrated by Apache Airflow, and two Web Servers that deliver all the Backend as well as the Frontend applications, configured with load balancing and redundancy to ensure high availability and fault tolerance. The implemented AI Pipeline includes tasks such as preprocessing, transcription, audio classification and topic modeling. Processed Podcasts are indexed in a Qdrant vector database to facilitate both dense and sparse retrieval while a recommendation system enriches the user’s experience. We summarize design choices and report system-level metrics and task-level indicators (ASR quality after correction, retrieval effectiveness) to guide similar deployments.