SOTA OCR with Core ML and dots.ocr

SOTA OCR with Core ML and dots.ocr

Every year our hardware is a little more powerful, our models a little smarter for each parameter. In 2025, it is more feasible than ever to run truly competitive models on-device. dots.ocr, a 3B parameter OCR model from RedNote, surpasses Gemini 2.5 Pro in OmniDocBench, making OCR a truly no compromises on-device use case. Running models on-device is certainly appealing to developers: no smuggling API keys, zero cost, and no network required. However, if we want these models to run on-device, we need to be mindful of the limited compute and power budgets. Enter the Neural Engine, Apple's custom AI accelerator that has shipped with every Apple device since 2017. This accelerator is designed for high performance whilst sipping battery power. Some of our testing has found the Neural Engine to be 12x more power efficient than CPU, and 4x more power efficient than GPU.

Whilst this all sounds very appealing, unfortunately the Neural Engine is only accessible through Core ML, Apple's closed source ML framework. Furthermore, even just converting a model from PyTorch to Core ML can present some challenges, and without a preconverted model or some knowledge of the sharp edges it can be arduous for developers. Luckily, Apple also offers MLX, a more modern and flexible ML framework that targets the GPU (not the Neural Engine), and can be used in conjunction with Core ML.

In this three part series, we will provide a reasoning trace of how we converted dots.ocr to run on-device, using a combination of CoreML and MLX. This process should be applicable to many other models, and we hope that this will help highlight the ideas and tools needed for developers looking to run their own models on-device.

Converting from PyTorch to CoreML is a two step process:

• Capturing your PyTorch execution graph (via torch.jit.trace or, the more modern approach of torch.export).
• Compiling this converted graph to an .mlpackage using coremltools.

Following the programmers litany of make it work, make it right, make it fast, we will first focus on getting the conversion working on GPU, in FLOAT32, and with static shapes. Once we have this working, we can dial down the precision and try and move to the Neural Engine.

Dots.OCR consists of two key components: A 1.2B parameter vision encoder trained from scratch, based on the NaViT architecture, and a Qwen2.5-1.5B backbone. We will be using CoreML to run the vision encoder, and MLX to run the LM backbone.

In order to convert a model, it's best to understand the structure and function before getting started. The dots vision encoder works on a patch basis, in this case 14x14 patches. We can simplify the model by removing all of the other attention implementations and just focusing on simple sdpa for now.

Using torch.jit.trace is still the most mature method for converting models to CoreML. It is rare that a model will convert first time. Often, you will need to progressively make changes further and further down the execution graph until you reach the final node. Common issues include dtype mismatches in torch.arange, unsupported ops like repeat_interleave for dynamic sequences, and the lack of bool tensor support on the Neural Engine. Most ML compilers do not like dynamic control flow, which requires careful handling of tensor shapes during the trace process.