View source for T5 (language model)

{{Short description|Series of large language models developed by Google AI}}
{{Infobox software
| name = Text-to-Text Transfer Transformer (T5)
| logo = 
| screenshot = 
| screenshot size = 
| caption = 
| author = [[Google AI]]
| developer = 
| released = {{Start date and age|23 October 2019}}
| latest release version = T5X {{url|https://github.com/google-research/t5x}}
| latest release date = 
| repo = https://github.com/google-research/text-to-text-transfer-transformer
| programming language = 
| operating system = 
| replaces = 
| replaced_by = 
| genre = {{ indented plainlist |
*[[Large language model]]
*[[Transformer (deep learning architecture)]]
}}
| license = [[Apache-2.0]]
| website = {{url|https://blog.research.google/2020/02/exploring-transfer-learning-with-t5.html}}
}}

'''T5 (Text-to-Text Transfer Transformer)''' is a series of [[Large language model|large language models]] developed by [[Google AI]]. Introduced in 2019,<ref name=":0">{{Cite journal |last1=Raffel |first1=Colin |last2=Shazeer |first2=Noam |last3=Roberts |first3=Adam |last4=Lee |first4=Katherine |last5=Narang |first5=Sharan |last6=Matena |first6=Michael |last7=Zhou |first7=Yanqi |last8=Li |first8=Wei |last9=Liu |first9=Peter J. |date=2020 |title=Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer |url=http://jmlr.org/papers/v21/20-074.html |journal=Journal of Machine Learning Research |volume=21 |issue=140 |pages=1–67 |issn=1533-7928}}</ref><ref name=":6">{{Citation |title=google-research/text-to-text-transfer-transformer |date=2024-08-21 |url=https://github.com/google-research/text-to-text-transfer-transformer |access-date=2024-08-21 |publisher=Google Research}}</ref> T5 models are trained on a massive dataset of text and code using a text-to-text framework. The T5 models are capable of performing the text-based tasks that they were pretrained for. They can also be finetuned to perform other tasks. They have been employed in various applications, including chatbots, machine translation systems, text summarization tools, code generation, and robotics.<ref>{{Cite arXiv |last1=Jiang |first1=Yunfan |last2=Gupta |first2=Agrim |last3=Zhang |first3=Zichen |last4=Wang |first4=Guanzhi |last5=Dou |first5=Yongqiang |last6=Chen |first6=Yanjun |last7=Fei-Fei |first7=Li |last8=Anandkumar |first8=Anima |last9=Zhu |first9=Yuke |date=2022-10-06 |title=VIMA: General Robot Manipulation with Multimodal Prompts |class=cs.RO |eprint=2210.03094 |language=en}}</ref>

Like the original Transformer model,<ref>{{Cite journal |last1=Vaswani |first1=Ashish |last2=Shazeer |first2=Noam |last3=Parmar |first3=Niki |last4=Uszkoreit |first4=Jakob |last5=Jones |first5=Llion |last6=Gomez |first6=Aidan N |last7=Kaiser |first7=Łukasz |last8=Polosukhin |first8=Illia |date=2017 |title=Attention is All you Need |url=https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=30}}</ref> T5 models are [[Transformer (deep learning architecture)|encoder-decoder Transformers]], where the encoder processes the input text, and the decoder generates the output text.

== Training ==
The original T5 models are pre-trained on the [[Colossal Clean Crawled Corpus]] (C4), containing text and code [[Web crawler|scraped from the internet]]. This pre-training process enables the models to learn general language understanding and generation abilities. T5 models can then be fine-tuned on specific downstream tasks, adapting their knowledge to perform well in various applications.

The T5 models were pretrained on many tasks, all in the format of <code><input text></code> -> <code><output text></code>.
[[File:T5-finetune-summarization.svg|thumb|How a T5 can be finetuned for a summarization task.<ref name=":7">{{Cite book |last1=Zhang |first1=Aston |title=Dive into deep learning |last2=Lipton |first2=Zachary |last3=Li |first3=Mu |last4=Smola |first4=Alexander J. |date=2024 |publisher=Cambridge University Press |isbn=978-1-009-38943-3 |location=Cambridge New York Port Melbourne New Delhi Singapore |chapter=11.9. Large-Scale Pretraining with Transformers |chapter-url=https://d2l.ai/chapter_attention-mechanisms-and-transformers/large-pretraining-transformers.html}}</ref>]]
Some examples are:

* restoring corrupted text: <code>Thank you <X> me to your party <Y> week.</code> -> <code><X> for inviting <Y> last <Z></code>, where the <code><Z></code> means "end of output", and the  <code><X></code> and  <code><Y></code> denote blanks to be filled, called "sentinels" in the original report.
* translation: <code>translate English to German: That is good.</code> -> <code>Das ist gut.</code>.
* judging the grammatical acceptability of a sentence ([[Corpus of Linguistic Acceptability|CoLA]] sentence): <code>The course is jumping well.</code> -> <code>not acceptable</code> .

== Architecture ==
[[File:T5_encoder-decoder_structure.svg|thumb|T5 encoder-decoder structure, showing the attention structure. In the encoder self-attention (lower square), all input tokens attend to each other; In the encoder–decoder cross-attention (upper rectangle), each target token attends to all input tokens; In the decoder self-attention (upper triangle), each target token attends to present and past target tokens only (causal).<ref name=":7" />]]
The T5 series encompasses several models with varying sizes and capabilities, all [[Transformer (deep learning architecture)|encoder-decoder Transformers]], where the encoder processes the input text, and the decoder generates the output text.

These models are often distinguished by their parameter count, which indicates the complexity and potential capacity of the model. The original paper<ref name=":0" /> reported the following 5 models:

{| class="wikitable"
|-
! Model !! Parameters !! <math>n_{layer}</math>!! <math>d_{model}</math> !! <math>d_{ff}</math> !! <math>d_{kv}</math> !! <math>n_{head}</math>
|-
| Small || 60M || 6 || 512 || 2048 || 64 || 8
|-
| Base || 220M || 12 || 768 || 3072 || 64 || 12
|-
| Large || 770M || 24 || 1024 || 4096 || 64 || 16
|-
| 3B || 3B || 24 || 1024 || 16384 || 128 || 32
|-
| 11B || 11B || 24 || 1024 || 65536 || 128 || 128
|}
<sup>*These describe the encoder's shape. The decoder has the same shape. So for exampel the T5-small has 6 layers in the encoder and 6 layers in the decoder.</sup>

In the above table,

* <math>n_{layer}</math>: Number of layers in the encoder; also, number of layers in the decoder. They always have the same number of layers.
* <math>n_{head}</math>: Number of attention heads in each attention block.
* <math>d_{model}</math>: Dimension of the embedding vectors.
* <math>d_{ff}</math>: Dimension of the feedforward network within each encoder and decoder layer.
* '''<math>d_{kv}</math>''': Dimension of the key and value vectors used in the self-attention mechanism.
Note that unlike typical Transformers, the 3B and 11B models do not satisfy <math>d_{model} = d_{kv} n_{head}</math>.<ref>{{Cite web |date=2020-04-24 |title=config.json · google-t5/t5-11b at main |url=https://huggingface.co/google-t5/t5-11b/blob/main/config.json |access-date=2024-09-17 |website=huggingface.co}}</ref>

Compared to the original Transformer, it uses a few minor modifications: layer normalization with no additive bias; placing the layer normalization outside the residual path; relative positional embedding.<ref>{{Citation |last1=Shaw |first1=Peter |title=Self-Attention with Relative Position Representations |date=2018-04-12 |url=https://arxiv.org/abs/1803.02155 |access-date=2024-09-17 |arxiv=1803.02155 |last2=Uszkoreit |first2=Jakob |last3=Vaswani |first3=Ashish}}</ref>

== Variants ==
Several subsequent models used the T5 architecture, with non-standardized naming conventions used to differentiate them. This section attempts to collect the main ones. An exhaustive list of the variants released by Google Brain is on the GitHub repo for T5X.<ref name=":5">{{Cite web |title=t5x/docs/models.md at main · google-research/t5x |url=https://github.com/google-research/t5x/blob/main/docs/models.md |access-date=2024-08-05 |website=GitHub |language=en}}</ref>

Some models are trained from scratch while others are trained by starting with a previous trained model. By default, each model is trained from scratch, except otherwise noted.

* ''T5'' small, base, large, 3B, 11B (2019): The original models.<ref name=":0" /> 
* ''T5 1.1'' small, base, large, XL, XXL: Improved versions of the original T5 series. These have roughly equal parameters. The 3B and the 11B were changed to "XL" and "XXL", and their shapes are changed:<ref name=":5" /><ref>{{Cite web |date=2020-11-19 |title=config.json · google/t5-v1_1-xl at main |url=https://huggingface.co/google/t5-v1_1-xl/blob/main/config.json |access-date=2024-09-17 |website=huggingface.co}}</ref><ref>{{Cite web |date=2020-11-19 |title=config.json · google/t5-v1_1-xxl at main |url=https://huggingface.co/google/t5-v1_1-xxl/blob/main/config.json |access-date=2024-09-17 |website=huggingface.co}}</ref>

{| class="wikitable"
|-
! Model !! Parameters !! <math>n_{layer}</math> !! <math>d_{model}</math> !! <math>d_{ff}</math> !! <math>d_{kv}</math> !! <math>n_{head}</math>
|-
| Small || 60M || 8 || 512 || 1024 || 64 || 6
|-
| Base || 220M || 12 || 768 || 2048 || 64 || 12
|-
| Large || 770M || 24 || 1024 || 2816 || 64 || 16
|-
| XL || 3B || 24 || 2048 || 5120 || 64 || 32
|-
| XXL || 11B || 24 || 4096 || 10240 || 64 || 64
|}

* ''LM-adapted T5'' (2021): a series of models (from small to XXL) that started from checkpoints of the ''T5'' series, but trained further on 100B additional tokens from C4.<ref>{{Citation |last1=Lester |first1=Brian |title=The Power of Scale for Parameter-Efficient Prompt Tuning |date=2021-09-02 |url=https://arxiv.org/abs/2104.08691 |access-date=2024-08-21 |arxiv=2104.08691 |last2=Al-Rfou |first2=Rami |last3=Constant |first3=Noah}}</ref>
* Switch Transformer (2021): a [[Mixture of experts|mixture-of-experts]] variant of T5, by replacing the feedforward layers in the encoder and decoder blocks with mixture of expert feedforward layers.<ref>{{Citation |last1=Fedus |first1=William |title=Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity |date=2022-06-16 |url=http://arxiv.org/abs/2101.03961 |access-date=2024-08-05 |arxiv=2101.03961 |last2=Zoph |first2=Barret |last3=Shazeer |first3=Noam}}</ref><ref>{{Cite web |title=SwitchTransformers |url=https://huggingface.co/docs/transformers/en/model_doc/switch_transformers |access-date=2024-08-05 |website=huggingface.co}}</ref>
* ''T0'' 3B, 11B (2021): a series of models that started from checkpoints of ''LM-adapted T5'', and further trained to perform tasks based only on task instruction ([[Zero-shot learning|zero-shot]]).<ref>{{Citation |last1=Sanh |first1=Victor |title=Multitask Prompted Training Enables Zero-Shot Task Generalization |date=2022-03-17 |url=http://arxiv.org/abs/2110.08207 |access-date=2024-08-05 |arxiv=2110.08207 |last2=Webson |first2=Albert |last3=Raffel |first3=Colin |last4=Bach |first4=Stephen H. |last5=Sutawika |first5=Lintang |last6=Alyafeai |first6=Zaid |last7=Chaffin |first7=Antoine |last8=Stiegler |first8=Arnaud |last9=Scao |first9=Teven Le}}</ref> Different entries in the series uses different finetuning data.<ref>{{Cite web |date=2024-03-04 |title=bigscience/T0 · Hugging Face |url=https://huggingface.co/bigscience/T0 |access-date=2024-08-21 |website=huggingface.co}}</ref>
* ''ByT5'' (2021): a byte-level version of T5, trained on mC4 (multilingual C4) dataset.<ref>{{Cite journal |last1=Xue |first1=Linting |last2=Barua |first2=Aditya |last3=Constant |first3=Noah |last4=Al-Rfou |first4=Rami |last5=Narang |first5=Sharan |last6=Kale |first6=Mihir |last7=Roberts |first7=Adam |last8=Raffel |first8=Colin |date=2022-03-25 |title=ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models |url=https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00461/110049/ByT5-Towards-a-Token-Free-Future-with-Pre-trained |journal=Transactions of the Association for Computational Linguistics |language=en |volume=10 |pages=291–306 |doi=10.1162/tacl_a_00461 |issn=2307-387X|arxiv=2105.13626 }}</ref> It operates on text encoded as [[UTF-8]] bytes, without tokenizers.
* ''Flan-T5-XL'' (2022): a model that started with a checkpoint of ''T5 XL'', then [[Instruction tuning|instruction-tuned]] on the FLAN dataset.<ref>{{Cite journal |last1=Chung |first1=Hyung Won |last2=Hou |first2=Le |last3=Longpre |first3=Shayne |last4=Zoph |first4=Barret |last5=Tay |first5=Yi |last6=Fedus |first6=William |last7=Li |first7=Yunxuan |last8=Wang |first8=Xuezhi |last9=Dehghani |first9=Mostafa |last10=Brahma |first10=Siddhartha |last11=Webson |first11=Albert |last12=Gu |first12=Shixiang Shane |last13=Dai |first13=Zhuyun |last14=Suzgun |first14=Mirac |last15=Chen |first15=Xinyun |date=2024 |title=Scaling Instruction-Finetuned Language Models |url=http://jmlr.org/papers/v25/23-0870.html |journal=Journal of Machine Learning Research |volume=25 |issue=70 |pages=1–53 |issn=1533-7928}}</ref><ref>{{Cite journal |last1=Longpre |first1=Shayne |last2=Hou |first2=Le |last3=Vu |first3=Tu |last4=Webson |first4=Albert |last5=Chung |first5=Hyung Won |last6=Tay |first6=Yi |last7=Zhou |first7=Denny |last8=Le |first8=Quoc V. |last9=Zoph |first9=Barret |last10=Wei |first10=Jason |last11=Roberts |first11=Adam |date=2023-07-03 |title=The Flan Collection: Designing Data and Methods for Effective Instruction Tuning |url=https://proceedings.mlr.press/v202/longpre23a.html |journal=Proceedings of the 40th International Conference on Machine Learning |language=en |publisher=PMLR |pages=22631–22648}}</ref><ref name=":2">{{Citation |title=google-research/FLAN |date=2024-08-03 |url=https://github.com/google-research/FLAN |access-date=2024-08-05 |publisher=Google Research}}</ref><ref>{{Cite web |date=2024-01-04 |title=google/flan-t5-xl · Hugging Face |url=https://huggingface.co/google/flan-t5-xl |access-date=2024-08-05 |website=huggingface.co}}</ref>
* ''T5X'' (2022): a [[Google JAX|JAX]]-based re-implementation of the original ''T5'' codebase. It is '''not''' a model.<ref name=":1">{{Cite journal |last1=Roberts |first1=Adam |last2=Chung |first2=Hyung Won |last3=Mishra |first3=Gaurav |last4=Levskaya |first4=Anselm |last5=Bradbury |first5=James |last6=Andor |first6=Daniel |last7=Narang |first7=Sharan |last8=Lester |first8=Brian |last9=Gaffney |first9=Colin |last10=Mohiuddin |first10=Afroz |last11=Hawthorne |first11=Curtis |last12=Lewkowycz |first12=Aitor |last13=Salcianu |first13=Alex |last14=Zee |first14=Marc van |last15=Austin |first15=Jacob |date=2023 |title=Scaling Up Models and Data with t5x and seqio |url=http://jmlr.org/papers/v24/23-0795.html |journal=Journal of Machine Learning Research |volume=24 |issue=377 |pages=1–8 |issn=1533-7928}}</ref> The original T5 codebase was implemented in [[TensorFlow]] with MeshTF.<ref name=":6" />
* ''UL2'' 20B (2022): a model with the same architecture as the ''T5'' series, but scaled up to 20B, and trained with "mixture of denoisers" objective on the C4.<ref name=":3">{{Citation |last1=Tay |first1=Yi |title=UL2: Unifying Language Learning Paradigms |date=2023-02-28 |url=http://arxiv.org/abs/2205.05131 |access-date=2024-08-05 |arxiv=2205.05131 |last2=Dehghani |first2=Mostafa |last3=Tran |first3=Vinh Q. |last4=Garcia |first4=Xavier |last5=Wei |first5=Jason |last6=Wang |first6=Xuezhi |last7=Chung |first7=Hyung Won |last8=Shakeri |first8=Siamak |last9=Bahri |first9=Dara}}</ref>
* ''Flan-UL2'' 20B (2022): ''UL2'' 20B [[Instruction tuning|instruction-finetuned]] on the FLAN dataset.<ref name=":3" /><ref name=":2" />
* ''Pile-T5'' (2024): has the same architecture of ''T5'', except it used the [[Llama (language model)|Llama]] tokenizer. It was trained on [[The Pile (dataset)|The Pile]]. It came in sizes of base, large, XL, XXL.<ref name=":4">{{Cite web |last1=Sutawika |first1=Lintang |last2=Komatsuzaki |first2=Aran |last3=Raffel |first3=Colin |date=2024-04-15 |title=Pile-T5 |url=https://blog.eleuther.ai/pile-t5/ |access-date=2024-05-05 |website=EleutherAI Blog |language=en}}</ref>

== Applications ==
The T5 model itself is an encoder-decoder model, allowing it to be used for instruction following. The encoder encodes the instruction, and the decoder autoregressively generates the reply.

The T5 encoder can be used as a text encoder, much like BERT. It encodes a text into a sequence of real-number vectors, which can be used for downstream applications. For example, Google Imagen<ref>{{Cite web |title=Imagen: Text-to-Image Diffusion Models |url=https://imagen.research.google/ |access-date=2024-08-23 |website=imagen.research.google}}</ref> uses T5-XXL as text encoder, and the encoded text vectors are used as conditioning on a [[diffusion model]]. As another example, the AuraFlow diffusion model<ref>{{Cite web |title=AuraFlow |url=https://huggingface.co/docs/diffusers/main/en/api/pipelines/aura_flow |access-date=2024-08-23 |website=huggingface.co}}</ref> uses ''Pile-T5-XL''.

== References ==
<references />


{{Google AI}}
{{Google LLC}}
{{Natural Language Processing}}
{{Differentiable computing}}

[[Category:Google software]]
[[Category:Large language models]]
[[Category:Software using the Apache license]]