Supercharge your OCR Pipelines with Open Models

TL;DR: The rise of powerful vision-language models has transformed document AI. Each model comes with unique strengths, making it tricky to choose the right one. Open-weight models offer better cost efficiency and privacy. To help you get started with them, we’ve put together this guide.

In this guide, you’ll learn:

• The landscape of current models and their capabilities
• When to fine-tune models vs. use models out-of-the-box
• Key factors to consider when selecting a model for your use case
• How to move beyond OCR with multimodal retrieval and document QA

By the end, you’ll know how to choose the right OCR model, start building with it, and gain deeper insights into document AI. Let’s go!

• Supercharge your OCR Pipelines with Open Models

Optical Character Recognition (OCR) is one of the earliest and longest running challenges in computer vision. Many of AI’s first practical applications focused on turning printed text into digital form.

With the surge of vision-language models (VLMs), OCR has advanced significantly. Recently, many OCR models have been developed by fine-tuning existing VLMs. But today’s capabilities extend far beyond OCR: you can retrieve documents by query or answer questions about them directly. Thanks to stronger vision features, these models can also handle low-quality scans, interpret complex elements like tables, charts, and images, and fuse text with visuals to answer open-ended questions across documents.

Recent models transcribe texts into a machine-readable format.

The input can include:

• Handwritten text
• Various scripts like Latin, Arabic, and Japanese characters
• Mathematical expressions
• Chemical formulas
• Image/Layout/Page number tags

OCR models convert them into machine-readable text that comes in many different formats like HTML, Markdown and more.

On top of text, some models can also recognize:

• Images
• Charts
• Tables

Some models know where images are inside the document, extract their coordinates, and insert them appropriately between texts. Other models generate captions for images and insert them where they appear. This is especially useful if you are feeding the machine-readable output into an LLM. Example models are OlmOCR by AllenAI, or PaddleOCR-VL by PaddlePaddle.

Models use different machine-readable output formats, such as DocTags, HTML or Markdown (explained in the next section Output Formats). The way a model handles tables and charts often depends on the output format they are using. Some models treat charts like images: they are kept as is. Other models convert charts into markdown tables or JSON, e.g., a bar chart can be converted as follows.

Similarly for tables, cells are converted into a machine-readable format while retaining context from headings and columns.

Different OCR models have different output formats. Briefly, here are the common output formats used by modern models.DocTag: DocTag is an XML-like format for documents that expresses location, text format, component-level information, and more. Below is an illustration of a paper parsed into DocTags. This format is employed by the open Docling models.

**HTML:**HTML is one of the most popular output formats used for document parsing as it properly encodes structure and hierarchical information.**Markdown:**Markdown is the most human-readable format. It’s simpler than HTML but not as expressive. For example, it can’t represent split-column tables.**JSON:**JSON is not a format that models use for the entire output, but it can be used to represent information in tables or charts.

The right model depends on how you plan to use its outputs:

Digital reconstruction: To reconstruct documents digitally, choose a model with a layout-preserving format (e.g., DocTags or HTML).LLM input or Q&A: If the use case involves passing outputs to LLM, pick a model that outputs Markdown and image captions, since they’re closer to natural language.Programmatic use: If you want to pass your outputs to a program (like data analysis), opt for a model that generates structured outputs like JSON.

Documents can have complex structures, like multi-column text blocks and floating figures. Older OCR models handled these documents by detecting words and then the layout of pages manually in post-processing to have the text rendered in reading order, which is brittle. Modern OCR models, on the other hand, incorporate layout metadata to help preserve reading order and accuracy. This metadata is called “anchor”, it can come in bounding boxes. This process is also called as “grounding/anchoring” because it helps with reducing hallucination.

OCR models can either take in images and an optional text prompt, this depends on the model architecture and the pre-training setup.

Some OCR models support prompt-based task switching, e.g. granite-docling can parse an entire page with the prompt “Convert this page to Docling” while it can also take prompts like “Convert this formula to LaTeX” along with a page full of formulas.

Other models, however, are trained only for parsing entire pages, and they are conditioned to do this through a system prompt.

For instance, OlmOCR by AllenAI takes a long conditioning prompt. Like many others, OlmOCR is technically an OCR fine-tuned version of a VLM (Qwen2.5VL in this case), so you can prompt for other tasks, but its performance will not be on par with the OCR capabilities.

We’ve seen an incredible wave of new models this past year. Because so much work is happening in the open, these players build on and benefit from each other’s work. A great example is AllenAI’s release of OlmOCR, which not only released a model but also the dataset used to train it. With these, others can build upon them in new directions. The field is incredibly active, but it’s not always obvious which model to use.

To make things a bit easier, we’re putting together a non-exhaustive comparison of some of our current favorite models. All of the models below are layout-aware and can parse tables, charts, and math equations. The full list of languages each model supports are detailed in their model cards, so make sure to check them if you’re interested. All models below have open-source license except for Chandra having OpenRAIL license and Nanonets license being unclear. The average scores are taken from model cards of Chandra, OlmOCR, evaluated on OlmOCR Benchmark, which is English-only. Many of the models in this collection have been fine-tuned from Qwen2.5-VL or Qwen3-VL, so we also provide Qwen3-VL model below as well.

| Model Name | Output formats | Features | Model Size | Multilingual? | |---|---|---|---|---| | Nanonets-OCR2-3B | structured Markdown | Captions images, Signature & watermark extraction, Handles handwriting | 3B | Yes | | PaddleOCR-VL | Markdown, JSON, HTML | Handles handwriting, old documents, Converts tables & charts to HTML | 0.9B | Yes (109 languages) | | dots.ocr | Markdown, JSON | Grounding, Extracts and inserts images, Handles handwriting | 3B | Yes | | OlmOCR-2 | Markdown, HTML, LaTeX | Grounding, Optimized for large-scale batch processing | 8B | English-only | | Granite-Docling-258M | DocTags | Prompt-based task switching, Rich output | 258M | Yes | | DeepSeek-OCR | Markdown, HTML | General visual understanding, Memory-efficient | 3B | Yes (nearly 100 languages) |