What Is a Transformer?

Also called transformer architecture

Short Answer

A transformer is the neural network architecture behind almost every modern large language model. Its key idea is attention: for each word, the model looks at all the other words in the text and weighs which ones matter most. Introduced by Google researchers in 2017, it replaced older sequence models.

  • Taught in lesson 80
2.1%In AI Job Ads2.1% of technical job ads at AI companies ask for Transformers (number 38 of the skills ai2hired tracks).Read the full skill guide: what employers expect, how deep to go and interview questions

Why It Matters

Transformers are the engine of modern AI: language models, many image models and speech models use them. Understanding attention explains practical facts engineers deal with daily, such as why long prompts cost more, why context windows have limits and why models are run on GPUs.

How It Works

The paper that introduced it, "Attention Is All You Need" (2017), showed that attention alone, without the recurrent networks used before, could read whole sequences in parallel. That made it possible to train far bigger models on far more data using GPUs.

The T in GPT stands for transformer: generative pre-trained transformer. Transformers now also power image, audio and protein models, not only text.

Before transformers, language models read text one word at a time, which was slow and made it hard to connect words far apart. Attention lets every token look at every other token at once, so the model can link a pronoun to a name three sentences earlier and can be trained in parallel on huge datasets. The cost is that comparing every token with every other one grows quickly with length, which is why long contexts are expensive.

Transformer in Practice

Everyday Example

In "The trophy did not fit in the suitcase because it was too big", attention helps the model link "it" to "the trophy", not to "the suitcase".

Industry Example: Pharma

Protein language models use the transformer architecture on amino acid sequences instead of words, helping scientists predict protein structure and function.

Where You Meet It

  • Explanations of how LLMs work.
  • The Hugging Face Transformers library in code.
  • Research and ML engineer job ads.
  • Model cards that describe a model's architecture and context length.

Key Ideas Around It

Attention
Weighing how much each token should use the others.
Encoder
The part that reads and represents text.
Decoder
The part that generates text token by token.
Parallel training
Processing a whole sequence at once, which made large models practical.

A Common Mix-Up

A transformer is an architecture, not a product. ChatGPT, Claude and Gemini are products built on very large transformer models; small transformers are also used quietly for search and classification.

Check YourselfWhy do longer prompts cost more to process in a transformer?

Because attention compares tokens with each other, so the work grows quickly as the input gets longer.

Learn It on ai2hired

S7 · #80Transformers Explained

Coming in Season 7, Deep Learning and Transformers. Every ai2hired lesson is free on YouTube.

Questions People Ask

What does attention mean in a transformer?

It is a learned way of scoring how relevant every other token is to the current one, so the model can combine the most useful information.

Do I need to build a transformer to get an AI job?

Most AI engineering roles need you to understand how it works, not to code one. Research and some ML engineering roles go deeper.

Learn AI From Zero to Hired

Free lessons on YouTube, each one built from what real AI job ads ask for.