Neural Information Retrieval for AI and Beyond

Information retrieval (IR) has long been a cornerstone of artificial intelligence (AI), with search engines designed to efficiently access and deliver information for human users. For decades, these systems have focused on helping people find relevant information to support their decisions. However, with the rise of large language models (LLMs) like ChatGPT, we are witnessing a shift in how IR can enable AI to retrieve knowledge and complete tasks for humans. In this thesis, we focus on neural information retrieval and explore how retrieval systems should be designed and applied in this new context. We begin by examining the core principles of existing neural information retrieval systems, including their architecture, learning methods, and related work. This is followed by an exploration of our efforts in developing effective retrieval models. We then address the design of retrieval systems optimized for AI rather than humans, emphasizing the key differences in this shift. Next, we discuss the challenges of managing retrieval indices in large-scale data stores as data volume continues to grow across various contexts. We also examine the extension of retrieval methods to handle multi-modal data, such as images and videos, alongside traditional documents. Finally, we conclude by outlining future work and directions for the field.