Modernizing Data Quality: Evolving Dimensions for Unstructured Text Data in Big Data, AI and Ethical Contexts

The exponential growth of unstructured text data – spanning social media, customer feedback, scientific publications, and digital communications – has created transformative opportunities for AI-driven analysis. Yet, ensuring high-quality text data at scale remains a pressing challenge. Traditional data quality dimensions such as accuracy, completeness, and consistency inadequately capture the complexities of text in diverse AI contexts. This chapter advances the discussion by reframing data quality through the lens of large-scale textual datasets and introducing frameworks and metrics specifically tailored to text-centric AI applications, focusing on making these approaches accessible to lay users. The novel contribution of this chapter is threefold: First, we extend existing theoretical frameworks by mapping four underexplored challenges – format heterogeneity, semantic ambiguity, syntactic variability, and scale – directly to the evolving needs of natural language processing and large language model training. Second, we propose and illustrate – such as semantic consistency, interpretability, contextual validity, and fairness – that go beyond traditional dimensions. Third, we bridge theory and practice by benchmarking quality assurance approaches across five critical U.S. sectors – healthcare, finance, media & journalism, retail & e-commerce, and government & policy – highlighting compliance, licensing, and technological constraints as well as strategies for handling noisy, subjective, and large-scale text data. By combining conceptual advances with practical engineering illustrations, this chapter provides a new, domain-grounded roadmap for text data quality assurance. We call for updated, scalable, and collaborative standards that embed quality into pipelines from collection to deployment, enabling the responsible and effective use of unstructured text in AI.