Information-Driven Design of Imaging Systems

An encoder (optical system) maps objects to noiseless images, which noise corrupts into measurements. Our information estimator uses only these noisy measurements and a noise model to quantify how well measurements distinguish objects.

Many imaging systems produce measurements that humans never see or cannot interpret directly. Your smartphone processes raw sensor data through algorithms before producing the final photo. MRI scanners collect frequency-space measurements that require reconstruction before doctors can view them. Self-driving cars process camera and LiDAR data directly with neural networks.

What matters in these systems is not how measurements look, but how much useful information they contain. AI can extract this information even when it is encoded in ways that humans cannot interpret.

And yet we rarely evaluate information content directly. Traditional metrics like resolution and signal-to-noise ratio assess individual aspects of quality separately, making it difficult to compare systems that trade off between these factors. The common alternative, training neural networks to reconstruct or classify images, conflates the quality of the imaging hardware with the quality of the algorithm.

We developed a framework that enables direct evaluation and optimization of imaging systems based on their information content. In our NeurIPS 2025 paper, we show that this information metric predicts system performance across four imaging domains, and that optimizing it produces designs that match state-of-the-art end-to-end methods while requiring less memory, less compute, and no task-specific decoder design.

Why mutual information?

Mutual information quantifies how much a measurement reduces uncertainty about the object that produced it. Two systems with the same mutual information are equivalent in their ability to distinguish objects, even if their measurements look completely different.

This single number captures the combined effect of resolution, noise, sampling, and all other factors that affect measurement quality. A blurry, noisy image that preserves the features needed to distinguish objects can contain more information than a sharp, clean image that loses those features.

Estimating information from measurements

Estimating mutual information between high-dimensional variables is notoriously difficult. Sample requirements grow exponentially with dimensionality, and estimates suffer from high bias and variance.

However, imaging systems have properties that enable decomposing this hard problem into simpler subproblems. Imaging systems have well-characterized noise. Photon shot noise follows a Poisson distribution. Electronic readout noise is Gaussian. This known noise physics means we can compute the noise entropy directly, leaving only the total measurement entropy to be learned from data.

For the measurement entropy, we fit a probabilistic model (e.g. a transformer or other autoregressive model) to a dataset of measurements. The model learns the distribution of all possible measurements. The approach provides an upper bound on true information; any modeling error can only overestimate, never underestimate.

Validation across four imaging domains

Information estimates should predict decoder performance if they capture what limits real systems. We tested this relationship across four imaging applications: color photography, radio astronomy, lensless imaging, and microscopy. In all cases, higher information meant better downstream performance.

Designing systems with IDEAL

Our Information-Driven Encoder Analysis Learning (IDEAL) method uses gradient ascent on information estimates to optimize imaging system parameters. IDEAL avoids the problems of end-to-end optimization by optimizing the encoder alone. The final result matched end-to-end optimization in both information content and reconstruction quality while avoiding decoder complexity during training.