What is 'Self-Attention' in the context of Transformer models?
Self-attention computes dynamic relevance scores between every pair of words within the same input sequence, allowing contextual representation of words based on all other words in the sentence.
In Transformer Decoders, what is the purpose of 'Causal Masking' (Look-Ahead Masking) in the self-attention layer?
Causal masking sets attention scores of future positions to -infinity before softmax, ensuring predictions for position i depend only on known tokens up to i.
What are 'Encoder-Decoder' Transformer models (such as original T5 and BART) uniquely suited for?
Encoder-decoder models (like BART and T5) excel at sequence-to-sequence tasks like translation and summarization. Both (A) and (B) are correct.
In modern Large Language Models, what does 'Context Window' refer to?
The context window is the maximum sequence length (e.g., 4k, 32k, 128k, 1M tokens) that the transformer's attention mechanism can evaluate simultaneously.
What does 'Temperature' control during stochastic text generation from an LLM's output token probability distribution?
Dividing logits by temperature T before softmax controls entropy: T -> 0 approaches argmax (greedy deterministic sampling); higher T flattens probabilities, yielding diverse outputs.