Scalable Big Data Infrastructure for Robust AI Data Onboarding

AI systems process massive datasets (TB-PB) for training and inference, creating a scalability bottleneck across preprocessing, analytics, and online feature serving.We address the question: How can we scale data infrastructure linearly with data volume while maintaining low latency for AI workloads?We evaluate three complementary methods along the pipeline: (1) MapReduce for parallel preprocessing, (2) Hive with partitioning for fast analytics, and (3) NoSQL databases for low-latency feature serving.Experiments show near-linear MapReduce scalability (up to 833 MB/s at 64 nodes), 15× query speedup from Hive partition pruning, and 3-5× lower p99 read latency from NoSQL (Cassandra/MongoDB) compared to a traditional RDBMS baseline.These results establish practical design principles for production-grade AI data onboarding.