Transformer Models

Transformer is a machine learning architecture introduced in 2017 that quickly became very popular, surpassing the state of the art in many areas, including language modeling, machine translation, question answering, and chatbots. This chapter describes the main components of the Transformer architecture, such as tokenization, embeddings, and self-attention, followed by the basics of training Transformer models and generating texts. It introduces some of the most well-known pre-trained language models (e.g., BERT and GPT), highlighting their impact and the rise of large language models (LLMs). It discusses how these pre-trained models can be tailored to specific tasks such as text classification, named entity recognition, text generation, and sentence embeddings for similarity analysis. The chapter introduces principles of in-context learning, prompt engineering, chain-of-thought prompting, and fine-tuning; it also discusses computational resources and available cloud platforms. Finally, it underscores the significance of responsible AI development, emphasizing the need to mitigate biases and ensure the ethical use of these powerful language models.