Multimodal GenAI and document AI

Story

Hide story

At the border station a baggage manifest arrives as a scan. Its table runs across two sheets, and a round purple stamp sits squarely over one column.

Merrow stamps a plausible total FILED and moves on.

Later the total is wrong, and only one sheet’s figures are anywhere in the Records Car. Where do you begin looking: at the reading, the finding, or the answering?

Concept#

Multimodal models accept (and sometimes produce) more than text: images, documents, audio, video. A common design for image input has three parts. A vision encoder turns the image into patch embeddings. A projection maps them into the language model’s token space. The language model then reasons over text tokens and image tokens together.

Document AI is the applied problem: turning real documents (layouts, tables, charts, scans) into reliable answers and structured data. The hard part is often not the model. It is getting the right content, in the right order, in front of it.

Why It Matters#

Most enterprise knowledge lives in PDFs, slides, spreadsheets and scans, not in clean text. When a document RAG system gives a wrong answer, the cause is often upstream of the model: the fact never reached a chunk, or it reached the chunk scrambled. Retrieval and generation cannot recover a fact that parsingParsingTurning a source file such as a PDF into clean text and structure. It is the usual cap on document RAG, because a fact lost here cannot be recovered later.Open the glossary entryClose lost.

This is also where you make a real engineering choice with real costs. Reading a page as an image is richer but slower and dearer than reading its text layer. You will be asked to justify that choice.

Mental Model#

Think of a page as a photograph of a page of text, a spreadsheet and a chart that someone taped together. You have two ways to get information out of it.

• Transcribe first. A cheap clerk copies out what they can read (text extraction, OCROCRShort for optical character recognition: software that reads text from images of pages.Open the glossary entryClose), and you work from the copy. Fast and cheap, but whatever the clerk drops or shuffles is gone.
• Look directly. An expensive expert studies the photo (a vision-language modelVision-language model (VLM)A model that takes images as well as text. A vision encoder turns patches of the image into tokens the language model can read, so more images cost more.Open the glossary entryClose). They see layout and charts, but they take longer, cost more, and can confidently misread a number.

Most production systems mix both: transcribe by default, send only the hard pages to the expert, and always keep a link back to the photo so a human can check.

Technical Deep Dive#

How a model reads a page image#

A vision encoder cuts the image into small square patches and turns each into a vector. Each patch becomes roughly one “visual token” the language model attends to, next to your text tokens. So an image is not free metadata. It is a block of input tokens whose size depends on the image.

This is separate from contrastive image-text embeddings (CLIP-style), which train an image encoder and a text encoder so a picture and its caption land close together. Those are useful for image search. A vision-language model (VLM) adds a language decoder so it can answer open questions about the image.

Cost follows from this. More pixels means more patches, up to a cap, and more images means more tokens, latency and spend. Providers differ in detail, so read the current docs for the model you use. As a dated example (checked 2026-10-09), Anthropic documents 28x28-pixel patches, a cost of about ceil(width/28) x ceil(height/28) visual tokens, and larger images being downscaled to a per-model cap (1,568 visual tokens on its standard tier, up to 4,784 on its high-resolution tier). OpenAI documents a different scheme (32x32-pixel patches for some models, 512-pixel tiles for others) and a detail setting (low, high, original, auto) that trades fidelity for tokens. The principle holds either way: downscaling saves tokens but can make small print unreadable, so check the image the model actually receives.

Four ways to read a page#

Layout-aware open-source parsers exist for the second row. For example, the Docling technical report describes a pipeline built from a page-layout detector and a table-structure model (TableFormer). Tools change quickly, so pick on your own documents, not on a feature list.

The ColPali paper (ICLR 2025) argues that text-extraction pipelines lose visual cues, and proposes producing multi-vector embeddings straight from page images, evaluated on its ViDoRe benchmark. In effect it files photographs of pages by how they look, rather than filing a transcript. Treat it as a strong option to test, not a default.

How do you choose? Start with the cheapest approach that passes your evaluation on your own documents. Use text-layer extraction when the layer is clean. Add OCR and layout analysis for scans. Route only complex pages (charts, forms, dense tables) to a VLM. Consider page-image retrieval when parsing keeps failing on visually rich pages.

Where parsing breaks#

Learn the failure patterns, because they tell you where to look.

• Reading order. A two-column page read left to right across both columns produces sentences that never existed.
• Tables. Merged and spanning cells, multi-line headers and missing borders flatten into a string of numbers with no column names.
• Units and footnotes. “(in thousands)” in a header, or a footnote that changes a figure, gets dropped, so a correct number is wrong by a factor of 1,000.
• Charts. The values are in pixels, not text. A VLM can describe a trend and still misread an exact value.
• Scans. Low resolution, skew, stamps and handwriting raise OCR error rates.
• Hallucinated figures. A VLM that cannot read a cell may produce a plausible number.

When users report wrong numbers, isolate the layer first. Open the chunk the retriever returned. If the fact is missing or scrambled in the chunk, it is a parsing failure. If the correct chunk exists but was not retrieved, it is retrieval. If the right chunk was in the prompt and the answer is still wrong, it is generation. Fix the earliest broken layer, because later layers cannot repair it.

Chunking parsed documents#

The chunking lesson covers generic chunking. Parsed documents add specific rules.

• Keep a table whole. Splitting a table by token count separates the header from the rows. Keep small tables in one chunk. For large ones, split by row groups and repeat the header in every chunk.
• Store tables in a structure the model can read. Use Markdown, HTML or one record per row with its column names and unit. A bare string of numbers loses meaning.
• Attach metadata. Page number, section path, element type (paragraph, table, figure) and, where available, the bounding box. This is what makes citations and click-through to the source possible.
• Treat figures separately. Store a caption or a model-written summary of a chart, and flag it as model-generated so nobody mistakes it for the document’s own words.
• Keep the link to the source page. An answer that cites “page 14, Table 3” can be checked in seconds. An answer without it cannot.

The common mistake is to extract text, concatenate it and embed it. That works on prose and fails on the tables and figures users often ask about.

Cost, latency and privacy#

• Cost. A VLM reads every page as image tokens, and some providers also feed in the extracted text. As a dated example (checked 2026-10-09), Anthropic’s PDF documentation says each page is converted to an image with its text extracted alongside, and typically uses about 1,500 to 3,000 text tokens per page on top of the image tokens. Multiply by your page count before running a VLM over everything.
• Latency. Image tokens add input processing time, and a VLM pass per page is slower than reading a text layer. Do the work at ingestion time, not at question time, wherever you can.
• Request limits. Providers cap pages, images and request size, and dense pages can exhaust the context window first. Limits change, so check the docs and split large files.
• Privacy. Sending page images to a hosted API sends the whole page, including content you did not mean to share. Check the provider’s retention and training terms, redact where you can, or run a self-hosted parser and model for sensitive documents.
• Accuracy. More capable is not the same as exact. Providers list limitations such as errors on low-quality, rotated or very small images and approximate counting, and advise human review for high-stakes use.

Evaluating the pipeline#

Evaluate each layer separately, on documents from your own domain, with a labelled set.

• Extraction: field-level accuracyField-level accuracyThe share of individual extracted fields, such as a total or a date, that match labelled values. It is a core metric for document extraction.Open the glossary entryClose (exact or normalized match per field), plus table-structure checks where tables matter.
• Retrieval: recall at the page or section level. Did the right page come back?
• Answers: faithfulness to the cited page, and citation correctness.
• Numbers: programmatic checks. Do line items add to the total? Are units consistent? Does a date parse? A failed check routes the document to review.
• Confidence and review: set a threshold below which a human checks the field, and measure how often that happens. That rate is a cost.

Without a labelled set you cannot tell whether switching from OCR to a VLM helped. For general evaluation see evaluating LLM systems.

Diagram#

Diagram: Documents are routed by type: pages with a clean text layer go through a text and table parser, scans go through OCR and layout analysis,...

Real-World Application#

A finance Q&A system ingests annual reports. Tables are parsed into rows with column names and units. Charts are summarized by a VLM into captions flagged “model-generated”. Every answer cites the page and table. Before display, a script checks that parsed line items add up to the stated total, and any mismatch goes to a human review queue. The team tracks field accuracy on a labelled sample every time the parser or model changes.

Beyond documents: voice#

Voice assistants are the other common multimodal product. The usual design is a cascade (speech-to-text, then an LLM, then text-to-speech) or an end-to-end speech model. The main engineering issue is the latency budget: stream every stage, handle barge-in (the user interrupting) and endpointing (deciding the user has finished), and keep prompts small. This lesson does not teach voice in depth.

Diagram: Speech flows through streaming speech-to-text, a streaming LLM and streaming text-to-speech to the speaker, and a barge-in signal from the...

Interview Perspective#

Expect “build RAG over PDFs with tables”, “how do you handle scanned documents” and “users report wrong numbers, what do you do”. Strong answers name the layer to isolate first (parsing), pick an approach with its trade-offs, keep page citations, and validate numbers in code. Common mistakes are assuming PDF text extraction is solved, ignoring tables, trusting VLM numbers without checks, and ignoring cost and privacy when sending every page to a hosted model.

Provenance: Common industry pattern. These are standard topics for document-AI and RAG roles, not a verified list for any named company.

Practice Questions#

These are generated practice questions, not reported from any company.

• Basic: What is CLIP-style contrastive embedding used for, and how does a VLM differ?
• Intermediate: Text extraction versus VLM page reading: what are the trade-offs?
• Advanced: Users report wrong numbers from tables. Locate and fix the failure.
• Product-Company: Design ingestion for 100,000 scanned invoices with a human review path for low-confidence fields.

Activity#

Take one real PDF with a table and a chart. Extract it two ways (for example the text layer and a VLM or layout parser). For each method, list the facts it lost or distorted, and write down the cost in time and money. Then choose a pipeline and say which pages you would route to the expensive method.

Homework#

Design the ingestion and evaluation plan for 100,000 scanned invoices into structured fields. Include the approach per stage, the labelled sample you would build, the field-level metric, the number checks, the confidence threshold for human review, and the privacy controls.

Quick Revision#

• A VLM turns image patches into tokens; more or larger images means more tokens, latency and cost.
• Parsing quality caps document RAG quality; reading order, tables, charts and scans are the usual culprits.
• Options: text layer, OCR plus layout, VLM page reading, page-image retrieval. Start cheap, route hard pages up.
• Keep tables whole, attach page and section metadata, cite pages.
• Isolate parsing, retrieval and generation; validate numbers in code; send low-confidence fields to a human.

Check yourself#

• Checkpoint question: Your RAG gives wrong answers to questions answered by tables in PDFs. What do you check first, and why?
• Common misconception: “Just embed the extracted text.” It scrambles reading order, drops table structure and loses figures.

Related#

• RAG architecture and chunking
• All RAG types
• Production deployment of RAG