Uncover This Tech Term: Foundation Model
Kyu-Hwan Jung
Abstract
dataset.The self-supervision dataset for the LLMs comprises a set of sentences with masked words generated by a computer without human involvement.The model is then trained to predict the masked words by looking at the unmasked part of the text.By utilizing vast amounts of self-supervised dataset, the model learns a semantically meaningful 'representation' (often colloquially referred to as feature) of the original dataset.These representations can be used for downstream tasks after fine-tuning with additional annotated datasets [3] or for generating new content in generative AI models [4].'Transformers,' a special form of deep learning model, which can deal with a large volume of unstructured dataset [5] have been the architectural choice of many FMs.Transformers were originally developed for natural language processing to overcome the limitations of recurrent neural networks in solving sequence-to-sequence tasks.However, the expressivity and scalability of transformers based on self-attention mechanisms have rapidly expanded their application to other domains.Transformers applied specifically to computer vision are called vision transformers (ViTs) [6], and have shown remarkable success in improving the performance of convolutional neural networks. Characteristics of FMFMs demonstrate emergent abilities, which refer to the AI's capability to perform certain skills that were not explicitly intended during training, and these abilities improve as the scale of the model increases [7].This is especially advantageous as it provides the potential for zero-shot or few-shot predictions in a novel task without additional training using an annotated dataset, which significantly reduces dataset construction efforts.Another characteristic of FMs is their ability to proficiently encode