Skip to main content
GGenerative AI
    • Start here
    • Learning paths5 paths
    • Modules11 modules
    • Projects5 projects
    • Quizzes
    • Flashcards
    • Homework
    • Interview prep8 question banks
    • Cheat sheets11 sheets
    • Glossary239 terms
    • Code snippets
    • Ask the mentor
    • My notes
    • My progress
SearchStart here
Menu
    • Start here
    • Learning paths5 paths
    • Modules11 modules
    • Projects5 projects
    • Quizzes
    • Flashcards
    • Homework
    • Interview prep8 question banks
    • Cheat sheets11 sheets
    • Glossary239 terms
    • Code snippets
    • Ask the mentor
    • My notes
    • My progress
  1. Reference
  2. Throughput

Glossary

Throughput

The total work a system completes per unit time, such as tokens or requests per second.

Where it is explained

  • LLM inference and servingProduction: cost, serving and observability

See also

Terms explained in the same lessons.

  • Chunked prefill
  • Continuous batching
  • FlashAttention
  • Goodput
  • Inter-token latency
  • Memory-bandwidth-bound
Previous termTest-time computeNext termTime to first token (TTFT)

All terms, A to Z

GGenerative AI

Learn generative AI from the first token to production systems.

  • Privacy
  • Search