Abstract: This paper reviews the evolution of Natural Language Processing (NLP) models, concentrating on the distillation techniques used to create efficient and compact versions of large models.