Introduction to Vector Databases: Embedding, Similarity, and Indexing

As digital technologies have grown quickly, about 90% of the world's data is now unstructured. Unlike traditional databases, unstructured data lacks standardised formats, making it hard to analyse. This paper examines how vector databases help solve this problem by enabling systems to find and use information based on meaning rather than just keywords. This paper explains how vector embeddings work, focusing on the Word2Vec model, which turns text into high-dimensional vectors. It also covers how cosine similarity measures how closely related texts are. Using a Python example with movie plot summaries, the paper shows how to turn text into vectors and search by meaning. It also discusses how these tools affect large language models through retrieval-augmented generation and examines the trade-offs between speed, complexity, and accuracy in approximate nearest neighbour search.