The Skill Explained
What Are Transformers?
A transformer is the neural network architecture behind almost every modern large language model. Its key idea is attention: for each word, the model looks at all the other words in the text and weighs which ones matter most. Introduced by Google researchers in 2017, it replaced older sequence models.
Why AI Employers Ask for Them
Almost every recent AI breakthrough in language uses Transformers, so roles that train, fine-tune or optimise models need to understand them. Knowing how they work explains practical limits such as context length, cost growing with input size, and why models handle some tasks well and others poorly. In job ads it also often refers to the Hugging Face Transformers library.
How It Works
Text is split into tokens and each token becomes a vector of numbers (an embedding), plus information about its position. In each layer, self-attention compares every token with every other one and mixes in information from the tokens that matter most. A small feed-forward network then processes each position. Stacking many such layers builds rich representations.
There are three main families. Encoder models (such as BERT) read a whole text and are good for classification and search. Decoder models (such as GPT-style models) generate text one token at a time and power chat assistants. Encoder-decoder models translate or summarise from one text to another.
Attention compares all pairs of tokens, so cost grows quickly with input length. Much engineering work, such as caching, efficient attention and quantization, exists to make Transformers faster and cheaper.
What You Do With It at Work
- Fine-tune a pre-trained Transformer for classification or extraction.
- Choose between an encoder for search and a decoder for generation.
- Use the Hugging Face Transformers library to load and run models.
- Profile memory and speed, and reduce them for deployment.
- Explain model limits, such as context length, to product teams.
Tools and Terms You Will Meet
- Attention
- A mechanism that weighs how much each token should use information from the others.
- Encoder and decoder
- Parts that read and understand text, and parts that generate it.
- BERT
- A well-known encoder model, widely used for classification and search.
- Hugging Face
- A platform and library hub for sharing and running pre-trained models.
- Positional encoding
- Information added so the model knows the order of tokens.
Industry Example: Pharma
A pharmacovigilance team fine-tunes an encoder Transformer to spot adverse event mentions in free-text reports, so safety reviewers see the most likely cases first.
A Common Mix-Up
A Transformer is the architecture; an LLM is a very large Transformer trained on vast text. Small Transformers are used every day for tasks like classification and need no chat interface.
How Deep Do You Need to Go?
- Know itExplain tokens, embeddings and attention in plain words.
- Use itLoad, run and fine-tune pre-trained Transformers with the Hugging Face libraries.
- Master itImplement attention, modify architectures and optimise Transformers for long inputs.