Pipeline parallelism

Splitting a model by layers, so different GPUs hold different whole layers and each passes its output to the next. It also lets a model too large for one GPU run, usually with less communication between GPUs than tensor parallelism needs.

All terms, A to Z