LLM and hardware

Overview¶

Computations in LLMs
LLM on super-comupters

Computations in LLMs¶

Neural networks¶

Learn patterns by adjusting parameters (weights);
Training = prediction → differentiation → update;
So far: mini-batch & optimizer & big → good.

Neural networks are building blocks of modern machine learning applications, the principle is simple, just like your regular gradient descent, you:

compute the output of your model;
compute the loss function according to reference output
compute the gradient of loss with respect to your parameters;
update you parameters slightly in the direction that reduces the loss the most;

Neural networks are however:

easy to differentiate: thanks to automatic differentiation;
easy to parallelize: matrix multiplications can be done in parallel;
easy to scale: training is done on small subsets of data per step.

which makes it extermely easy to scale up, and show great performance when scaled up. [^1]

Note that during training, we need to store multiple copies of all model parameters (for gradients, optimizer states, etc.), which multiplies memory needs.

PS: If you are suprised that large models work better, you are not alone; see double descent

Transformer¶

Transformer computes relationships between tokens (attention);
tokens can be processed in parallel

Training of LLMs¶

Just neural networkes that can be parallelized more efficiently;

Fine-tuninig of LLMs¶

With specialized data (instruct, chat, etc);
Less memory usage by "freezing parameters"

Inference of LLMs¶

GPT-style inference: pre-filling and decoding;
Pre-filling: process the input prompt in parallel;
Decoding: generate new tokens one-by-one, using cached results.

Optimize caches for inference¶

KV cache:
paged attention: indexed blockes of caches;
flash attention: fuse operations to reduce caches;

more in-depth discussion of the technique where that visualization is from: paged attention from first principles.

Key takeaway¶

LLMs/NNs benefit from massive parallelization;
Need for different tasks:
training: memory + compute + data throughput;
fine-tuninig: similar to training, cheaper;
pre-filling: compute;
decoding: memory;

LLM on HPC clusters¶

LLM on general computers¶

Mostly about inference;
Quantization;
CPU offloading;
Memory-mapped file formats;

HPC clusters¶

Racked computer nodes;
Parallel network storage;
Infiniband/RoCE networking;

Alvis hardware - compute¶

Data type	A100	A40	V100	T4
FP64	9.7 \| 19.5	0.58	7.8	0.25
FP32	19.5	37.4	15.7	8.1
TF32	156	74.8	N/A	N/A
FP16	312	149.7	125	65
BF16	312	149.7	N/A	N/A
Int8	624	299.3	64	130
Int4	1248	598.7	N/A	260

Alvis hardware - network & storage¶

Fast storage: WEKA file system;
Infiniband: 100Gbit (A100 nodes);
Ethernet: 25Gbit (most other nodes);

Running LLMs on supercomputers¶

Most common bottleneck: memory
Quantized models to fit larger models;
Parallelize the model across GPUs or nodes;

Supercomputers allow us to run larger LLMs because of not only the more powerful nodes, but also the nodes are connected with fast internet connection. But we still have the same issue as most high performance computing tasks, the compute needs to access the memory and the data need to transferred.

Most commonly, when you try to run a large model (70B and 400B) parameters, you need to split the models to many GPUs. You can choose to quantize the model, so that the model use less memory, and you need less GPUs and less parallelization issues. To this end, you will need to look up the compatibility between the model, hardware and implementation.

You will also need to think about how do you parallelize the models, you have the options:

Tensor parallelism;
Pipeline parallelism;
Data parallelism

So next we will introduce the different formats in LLMs, to give you an idea of what it means when you download a certain model, and tomorrow we will walk through the

Tools to gain information¶

grafana (network utilization, temp disk);
nvtop, htop (CPU/GPU utilization, power draw);
nvidia nsight (advanced debugging and tracing);

See details in C3SE documentation.

Summary¶

Take home messages¶

LLMs/neural networks benefit from massive parallelization;
Same issue of memeory vs. compute-bound;
Some optimization strategies;
Be aware of the troubleshooting tools!

Useful links¶

nanotron has some in-depth discussion about the efficiency of model training; (The Ultra-Scale Playbook), as well as a prediction memory estimation tool;
Alvis hardware specifications;
Alvis monitoring tools;