Tensor parallelism

Splitting each layer's weight matrices across several GPUs, so every GPU holds a slice of every layer and they exchange partial results at each layer. It lets a model too large for one GPU run, and it needs fast links between the GPUs.

All terms, A to Z