The text discusses foundational aspects of large language models, focusing on their role in enabling universal models to tackle diverse issues through large-scale language modeling. The document covers pre-training, generative models, prompting, and alignment methods, aimed at machine learning and natural language processing backgrounds. It is self-contained, offering a flexible learning path for in-depth exploration or comprehensive understanding. Key concepts include variables, functions, and models in machine learning and statistics, with notations covering probability, loss functions, and attention mechanisms in sequential models. Terms like max, arg max, input and output tokens, model parameters, and hidden states are defined. The Softmax function and KL divergence are also mentioned. The text delves into pre-training and generative models in natural language processing, discussing unsupervised, supervised, and self-supervised pre-training methods, adapting pre-trained models, and self-supervised pre-training tasks like decoder-only, encoder-only, and encoder-decoder pre-training. The BERT model is highlighted, including its standard model, more training and larger models, more efficient models, and multi-lingual models. The text also explores applying BERT models and their fine-tuning. In generative models, the text introduces large language models (LLMs), their training, fine-tuning, aligning with the world, and prompting techniques. It discusses training at scale, data preparation, model modifications, distributed training, and scaling laws. Long sequence modeling is covered from HPC perspectives, efficient architectures, cache and memory management, and sharing across heads and layers. The text covers AI prompting and alignment, including general design, advanced methods like chain of thought, problem decomposition, self-refinement, ensembling, and RAG. It also discusses learning to prompt, optimization, and reduction techniques. The alignment section explores LLM alignment, instruction alignment through supervised fine-tuning, data acquisition, and generalization, human preference alignment using reinforcement learning, and improved methods for better reward modeling. The text discusses methods for direct preference optimization, automatic preference data generation, step-by-step alignment, and inference-time alignment. It concludes with a summary and a bibliography. Pre-training in NLP, using self-supervised learning, enables universal language understanding and generation. This approach, involving large-scale unsupervised training on unlabeled data, creates foundation models adaptable to various tasks through fine-tuning or prompting. Inspired by early deep learning efforts, pre-training has seen resurgence, particularly with language models like BERT and GPT, trained to predict masked words in vast text datasets. These models, pre-trained on general tasks, excel in diverse NLP problems, often surpassing supervised systems. Recent advancements in large language models indicate promising future applications in AI. Pre-training involves training a neural network model on diverse tasks without specific downstream tasks assumed. The model aims to generalize across various tasks. Two main approaches are unsupervised and supervised pre-training. Unsupervised pre-training optimizes model parameters using criteria unrelated to specific tasks, aiding in discovering better local minima and adding regularization. Supervised pre-training uses labeled data for initial training. The model is then fine-tuned for specific tasks using labeled data or task descriptions. This approach reduces reliance on task-specific labeled data, enabling the development of more general models. Pre-training NLP models involves creating a text classification system by stacking a classifier on an encoder. The model, initially not optimized for classification, is fine-tuned using a labeled dataset, adapting its parameters for the task. Alternatively, encoder parameters can be frozen to maintain their pre-trained state, focusing on optimizing the classifier. Fine-tuning uses less data than pre-training, making adaptation efficient. This process allows pre-trained models to be adapted for various tasks, such as question answering and machine translation, without additional modules. Pre-training large language models on extensive data enables them to excel in token prediction, transforming numerous NLP problems into text generation tasks. This approach allows for simple prompting, where a task is framed by concatenating input text with a specific instruction, such as predicting the polarity of a text. Large language models can perform complex tasks through such instructions, demonstrating zero-shot learning capabilities. Few-shot learning, achieved through in-context learning with demonstrations, further enhances these models' abilities by teaching them to perform tasks with limited data. Self-supervised pre-training tasks involve identifying text polarity and using decoder-only, encoder-only, or encoder-decoder architectures, focusing on Transformers. Decoder-only models predict token distributions given preceding tokens, using the log-scale cross-entropy loss for training. Pre-training involves optimizing parameters for a language model using a loss function based on log-scale cross-entropy between predicted and actual sequences. The goal is to minimize loss across a dataset, mathematically equivalent to maximum likelihood estimation. Optimized parameters enable computing sequence probabilities. Encoder-only pre-training combines an encoder with output layers for supervision, using a Softmax layer on top of the Transformer encoder to predict probability distributions for each sequence position. Self-supervised pre-training involves training a model to reconstruct masked tokens using a masked language modeling approach, like in BERT. This contrasts with standard language models where predictions are made based on the left context. In pre-training, the entire sequence is observed, enabling bidirectional prediction. The pre-trained encoder, without the softmax layer, is then combined with a prediction network for specific tasks, often requiring fine-tuning with labeled data. Pre-training involves masking tokens in sequences for a model to predict, optimizing it to maximize reconstruction probability. This autoencoding-like process focuses on masked tokens, simplifying training. The objective is to maximize masked token prediction probabilities, using either maximum likelihood estimation or cross-entropy loss. Once trained, the model can be fine-tuned for specific tasks or directly applied. Permuted language modeling addresses discrepancies in self-supervised pre-training by allowing predictions in any order, unlike causal language modeling. This method enables tokens to be conditioned on a broader context, improving prediction accuracy. Transformers easily implement this by setting appropriate self-attention masks, enhancing model performance. The text outlines three self-supervised pre-training tasks: causal language modeling, masked language modeling, and permuted language modeling. Causal language modeling predicts the next word in a sequence given previous words. Masked language modeling predicts a masked word given context. Permuted language modeling predicts words in a shuffled sequence. These tasks generate training samples by comparing actual consecutive sentences (positive samples) with randomly sampled sentences (negative samples). NSP, used as an additional training loss, assesses sentence order understanding. The ELECTRA model uses a masked language model to generate altered sequences, which are then classified by a discriminator to distinguish original tokens from altered ones. This process trains a Transformer encoder to identify token alterations, pre-training it for downstream tasks. The generator is optimized with maximum likelihood estimation, while the discriminator uses classification-based loss. The model combines these losses for joint training, with an alternative using generative adversarial networks. Post-training, the generator is discarded, and the discriminator's encoding part is used for various tasks. Encoder-decoder models in NLP can be adapted for various tasks by treating text as input and output. This approach enables a single text-to-text system for diverse NLP problems. Pre-training involves training an encoder-decoder model on self-supervised tasks to gain general language knowledge. The T5 model by Raffel et al. frames multiple tasks as text-to-text, using a format with a task description, input, and response. Examples include translation, simplification, and scoring translations, showcasing the model's versatility in handling different NLP tasks. Pre-training transforms the scoring problem into text generation, aiming to create text that represents numerical values. This method unifies various tasks, training a single model capable of multiple functions. Fine-tuning adapts the model for specific tasks, with instructions in text form aiding in learning general knowledge and enabling zero-shot learning. Pre-trained encoder-decoder models can be trained using self-supervised learning, focusing on predicting subsequent sequences given a prefix. For multi-lingual tasks, models require training with multi-lingual data to learn shared representations across languages. Self-supervised pre-training involves tasks like denoising autoencoding, where noise is added to input data, and the model aims to reconstruct the original. Key methods include token masking, token deletion, and span masking. Token masking replaces selected tokens with [MASK], while token deletion removes them. Span masking involves masking non-overlapping spans with [MASK], introducing challenges for the model to predict span lengths, akin to fertility modeling in machine translation. The BART model employs two pre-training methods: Sentence Reordering, which randomizes sentence sequence to teach reordering, and Document Rotation, aiming to identify sequence start tokens by rotating the sequence. Pre-training with multiple corruption methods enhances model robustness. Tasks are categorized based on training objectives, including Language Modeling for sequence prediction and Masked Language Modeling for token prediction in masked sequences. BERT models, popular in NLP, use masked language modeling, discriminative training, and denoising autoencoding for pre-training. These tasks involve predicting masked tokens, training classifiers, and reconstructing corrupted sequences, respectively. BERT, introduced by Devlin et al. (2019), combines masked language modeling and next sentence prediction tasks, optimizing parameters through minimizing a loss function that sums these task losses. BERT-style pre-training involves masking tokens in sentences for prediction, using techniques like causal, prefix, and masked language modeling, and permutation. It also includes next sentence prediction, token classification, reordering, deletion, and masking. The process selects random training samples, accumulates loss, and updates model parameters via gradient descent. BERT models, trained on these tasks, are versatile for various language understanding problems. BERT uses token masking, random replacement, and unchanged tokens to train models to predict masked or random tokens, enhancing context understanding. The loss function, LossMLM, calculates the probability of predicting a token given the modified sequence. For next sentence prediction, samples are classified into 'IsNext' or 'NotNext' labels. BERT models, based on Transformer architecture, use embeddings for token, position, and segment identification. Each input token is represented by a vector sum of these embeddings. Models consist of multiple Transformer layers, each with self-attention and FFN sub-layers, employing post-norm architecture. Key aspects in BERT development include vocabulary size, embedding and hidden dimensions, number of heads, and FFN hidden size. Larger models feature significantly larger FFN hidden layers. BERT, a pivotal NLP model, has inspired advancements through scaling, with RoBERTa and larger models. Scaling involves increased data, compute, and data removal for improved performance. However, this introduces training challenges, especially with very large models. Efforts aim to create more efficient BERT models, focusing on knowledge distillation and model compression. Techniques like training smaller models with larger ones' knowledge and pruning layers are used to reduce size and improve efficiency. BERT models are optimized through pre-training, pruning, quantization, and dynamic network adaptation. Pre-training involves fine-tuning on specific tasks or using a percentage of parameters. Pruning reduces model size by removing heads in multi-head attention models or layers in deep networks. Quantization compresses models by representing parameters as low-precision numbers. Dynamic networks adapt by dynamically choosing layers for processing, optimizing for efficiency. Parameter sharing across layers reduces model size, enabling reuse in multi-layer networks. Multi-lingual BERT models, trained on diverse languages, offer universal representation and cross-lingual learning capabilities. Improvements include bilingual data in pre-training, enhancing cross-lingual transfer abilities. BERT pre-training involves treating the model as an encoder, maximizing probabilities of masked tokens in bilingual data. This enables learning representations for multiple languages and their correspondences, making the model adept at handling code-switching. Multi-lingual pre-trained models inherently manage code-switching without language identification, using a shared vocabulary. Factors influencing the result include vocabulary size, language samples, and model architecture. Larger models and vocabularies are beneficial, especially for low-resource languages, but can lead to interference with extended training. The text discusses BERT models, focusing on pre-training and application. BERT models, initially pre-trained on large datasets, require fine-tuning for specific tasks. This involves aligning the model's output with the task's requirements through a prediction network. The model is then fine-tuned using a set of labeled samples to optimize performance for the targeted task. Key points include the importance of stopping pre-training early to avoid performance degradation and the necessity of fine-tuning for task adaptation. BERT models are fine-tuned for tasks like text classification, both single and pair texts. In single text classification, BERT encodes input sequences into vectors, using the first output for text representation. This vector is input to a prediction network for label prediction. For pair text classification, texts are concatenated, and the hcls vector is used for prediction. The prediction network can be any classification model, and the entire model is trained or fine-tuned similarly to standard classification models. BERT supports text pair classification, regression for similarity assessment, and sequence labeling for tasks like POS tagging and NER. It processes inputs like "Text 1" and "Text 2", generating embeddings and outputs for classification, regression, or sequence labeling tasks. The text discusses BERT models, focusing on pre-training and application. BERT models, initially pre-trained on large datasets, require fine-tuning for specific tasks. This involves aligning the model's output with the task's requirements through a prediction network. The model is then fine-tuned using a set of labeled samples to optimize performance for the targeted task. Key points include the importance of stopping pre-training early to avoid performance degradation and the necessity of fine-tuning for task adaptation. BERT models are fine-tuned for tasks like text classification, both single and pair texts. In single text classification, BERT encodes input sequences into vectors, using the first output for text representation. This vector is input to a prediction network for label prediction. For pair text classification, texts are concatenated, and the hcls vector is used for prediction. The prediction network can be any classification model, and the entire model is trained or fine-tuned similarly to standard classification models. BERT models are fine-tuned for tasks like text classification, both single and pair texts. In single text classification, BERT encodes input sequences into vectors, using the first output for text representation. This vector is input to a prediction network for label prediction. For pair text classification, texts are concatenated, and the hcls vector is used for prediction. The prediction network can be any classification model, and the entire model is trained or fine-tuned similarly to standard classification models. BERT models are fine-tuned for tasks like text classification, both single and pair texts. In single text classification, BERT encodes input sequences into vectors, using the first output for text representation. This vector is input to a prediction network for label prediction. For pair text classification, texts are concatenated, and the hcls vector is used for prediction. The prediction network can be any classification model, and the entire model is trained or fine-tuned similarly to standard classification models. The text discusses BERT models, focusing on pre-training and application. BERT models, initially pre-trained on large datasets, require fine-tuning for specific tasks. This involves aligning the model's output with the task's requirements through a prediction network. The model is then fine-tuned using a set of labeled samples to optimize performance for the targeted task. Key points include the importance of stopping pre-training early to avoid performance degradation and the necessity of fine-tuning for task adaptation. BERT models are fine-tuned for tasks like text classification, both single and pair texts. In single text classification, BERT encodes input sequences into vectors, using the first output for text representation. This vector is input to a prediction network for label prediction. For pair text classification, texts are concatenated, and the hcls vector is used for prediction. The prediction network can be any classification model, and the entire model is trained or fine-tuned similarly to standard classification models. BERT models are fine-tuned for tasks like text classification, both single and pair texts. In single text classification, BERT encodes input sequences into vectors, using the first output for text representation. This vector is input to a prediction network for label prediction. For pair text classification, texts are concatenated, and the hcls vector is used for prediction. The prediction network can be any classification model, and the entire model is trained or fine-tuned similarly to standard classification models. The text discusses BERT models, focusing on pre-training and application. BERT models, initially pre-trained on large datasets, require fine-tuning for specific tasks. This involves aligning the model's output with the task's requirements through a prediction network. The model is then fine-tuned using a set of labeled samples to optimize performance for the targeted task. Key points include the importance of stopping pre-training early to avoid performance degradation and the necessity of fine-tuning for task adaptation. BERT models are fine-tuned for tasks like text classification, both single and pair texts. In single text classification, BERT encodes input sequences into vectors, using the first output for text representation. This vector is input to a prediction network for label prediction. For pair text classification, texts are concatenated, and the hcls vector is used for prediction. The prediction network can be any classification model, and the entire model is trained or fine-tuned similarly to standard classification models. The text discusses BERT models, focusing on pre-training and application. BERT models, initially pre-trained on large datasets, require fine-tuning for specific tasks. This involves aligning the model's output with the task's requirements through a prediction network. The model is then fine-tuned using a set of labeled samples to optimize performance for the targeted task. Key points include the importance of stopping pre-training early to avoid performance degradation and the necessity of fine-tuning for task adaptation. BERT models are fine-tuned for tasks like text classification, both single and pair texts. In single text classification, BERT encodes input sequences into vectors, using the first output for text representation. This vector is input to a prediction network for label prediction. For pair text classification, texts are concatenated, and the hcls vector is used for prediction. The prediction network can be any classification model, and the entire model is trained or fine-tuned similarly to standard classification models. The text discusses BERT models, focusing on pre-training and application. BERT models, initially pre-trained on large datasets, require fine-tuning for specific tasks. This involves aligning the model's output with the task's requirements through a prediction network. The model is then fine-tuned using a set of labeled samples to optimize performance for the targeted task. Key points include the importance of stopping pre-training early to avoid performance degradation and the necessity of fine-tuning for task adaptation. BERT models are fine-tuned for tasks like text classification, both single and pair texts. In single text classification, BERT encodes input sequences into vectors, using the first output for text representation. This vector is input to a prediction network for label prediction. For pair text classification, texts are concatenated, and the hcls vector is used for prediction. The prediction network can be any classification model, and the entire model is trained or fine-tuned similarly to standard classification models. The text discusses BERT models, focusing on pre-training and application. BERT models, initially pre-trained on large datasets, require fine-tuning for specific tasks. This involves aligning the model's output with the task's requirements through a prediction network. The model is then fine-tuned using a set of labeled samples to optimize performance for the targeted task. Key points include the importance of stopping pre-training early to avoid performance degradation and the necessity of fine-tuning for task adaptation. BERT models are fine-tuned for tasks like text classification, both single and pair texts. In single text classification, BERT encodes input sequences into vectors, using the first output for text representation. This vector is input to a prediction network for label prediction