Cheat sheet: Multimodal and document AI

Quick answers#

• Images cost tokens: a VLM turns image patches into input tokens, so more or larger images means more cost and latency. Downscaling saves tokens but can blur small print.
• Four ways to read a page: text layer, OCR plus layout analysis, VLM page reading, page-image retrieval. Start with the cheapest that passes your evaluation; route hard pages up.
• Parsing caps quality: reading order, tables, units, charts and scans are the usual failures. If the fact never reaches the chunk, retrieval and generation cannot recover it.
• Isolate the layer: wrong chunk content is parsing, right chunk not retrieved is retrieval, right chunk but wrong answer is generation.
• Chunking parsed documents: keep tables whole (or repeat the header), store column names and units, add page and section metadata, flag model-written captions.
• Cite pages: an answer that names a page and table can be checked quickly.
• Verify numbers in code: totals add up, units match, low-confidence fields go to a human.
• Privacy: page images expose the whole page to a hosted API; check retention terms or self-host.
• Evaluate: field-level accuracy on a labelled sample, page-level retrieval recall, faithfulness to the cited page.