An original interactive lab
Eight machines. One brain. Watch a real neural network learn while its workers pass gradients around a ring, then find out exactly when adding more machines stops helping.
Computed live from the hardware model on the left. Drag the model size or bandwidth and watch the curves bend.
Trains a small neural network the way large models are trained: the batch is split across workers, each computes a gradient on its slice, and the workers sum their gradients before everyone takes the same step. Every number on this page is computed live, in your browser, from real maths.
Sending every gradient to one server makes that server's link carry N copies. A ring has each worker talk only to a neighbour, so traffic per worker stays about twice the model size however many workers join. That is the idea behind NCCL and Horovod.
Each step the lab recomputes the gradient on one machine and compares. The error stays near machine precision and the replicas never drift apart, so you can see that parallelism changes the speed, not the answer.
Frontier models cannot fit or finish on one device. Knowing where communication, stragglers and batch size eat your scaling is the difference between a cluster that is fast and one that is merely expensive.