Scientific Machine Learning: The Big Picture Before the Details

• Tags: scientific-machine-learning inverse-problems PINN neural-operators neural-fields differentiable-physics computational-imaging

Most of the deep learning I learned first was about fitting functions to data. You collect enough labeled examples, pick an architecture, minimize a loss, and hope the network generalizes. It works astonishingly well for natural images and text — domains where data is abundant and the underlying “rules” are too messy to write down anyway.

But a lot of the world is not like that. In physics, optics, fluid dynamics, and medical imaging, we often know the rules. We have governing equations, forward models, and decades of classical theory. Throwing all of that away to learn everything from scratch feels wasteful — and in data-scarce regimes, it simply fails.

This is the gap that Scientific Machine Learning (SciML) tries to close: ML that respects the physics we already know. This post is a map of that landscape, written before I get into any of the specifics. I want to lay out the territory first, so that later posts — on denoising, inverse problems, and specific imaging modalities — have somewhere to anchor.

Why pure data-driven hits a wall

Pure data-driven models are powerful, but in scientific and medical settings they run into a recurring set of problems:

  • Hallucination. A model trained to produce “plausible” outputs can invent detail that looks right but isn’t there. In a medical image, a plausible-but-wrong texture is not a cosmetic issue — it’s a diagnostic hazard.
  • Physics is ignored. A network denoising a CT scan usually knows nothing about photon statistics, the scanner geometry, or how the image was actually formed.
  • Weak out-of-distribution generalization. Train on one scanner or one dose level, and performance can collapse on another.
  • The evaluation gap. Surrogate metrics like PSNR and SSIM correlate poorly with what a clinician actually cares about, so even “good” numbers can be misleading.

Each of these failure modes points in the same direction: the model has no idea how the data was generated. If we could hand the network that knowledge, many of these problems soften. That handover is exactly what SciML is about.

The unifying frame: inverse problems

Before naming techniques, it helps to have one frame that ties most of them together: the inverse problem.

There is some hidden truth $x$ we cannot observe directly — an attenuation field, a tissue structure, a velocity field. A physical process, the forward operator $A(\cdot)$, maps it to what we actually measure:

\[y = A(x) + n\]

where $y$ is the observation and $n$ is noise. The inverse problem is to recover $x$ from $y$, given knowledge of $A$.

The catch is that these problems are almost always ill-posed: the operator $A$ may be ill-conditioned or non-invertible, solutions may not be unique, and noise gets amplified during reconstruction. Classical approaches lean on regularization — adding prior assumptions about what a “reasonable” $x$ looks like.

SciML’s central idea is to replace or augment that hand-designed prior with a learned one, while keeping the forward operator $A$ in the picture. That single move — “keep the physics, learn the prior” — is the thread running through everything below.

The SciML landscape

There isn’t one technique called “scientific ML.” It’s a family. Here are the branches I find most useful to keep straight, roughly organized by what part of the problem they put a neural network into.

Physics-Informed Neural Networks (PINNs)

The idea: embed the governing PDE directly into the loss function. The network is trained not only to match data, but also to satisfy the differential equation that describes the system. The PDE residual acts as a built-in regularizer.

  • Strengths: works with sparse data, mesh-free, elegant when you have a clean governing equation.
  • Weaknesses: training can be stiff and finicky, and scaling to high-dimensional or stiff PDEs remains hard.

Neural Operators

Instead of learning a single function, neural operators learn a mapping between function spaces — for example, from a coefficient field or boundary condition to the full PDE solution. The Fourier Neural Operator (FNO) learns in the spectral domain, which gives it a global receptive field and a degree of resolution invariance.

This branch has been a workhorse for weather, fluid, and climate simulation, where you want to replace a slow numerical solver with a fast learned surrogate.

Neural Fields (Implicit Neural Representations)

A neural field represents a signal as a continuous function of coordinates:

\[f_\theta : (x, y, z) \mapsto \text{value}(x, y, z)\]

Rather than storing a voxel grid, you store the weights of a network that can be queried at any coordinate. That makes the representation continuous, compact, differentiable, and resolution-free.

For inverse problems this is a natural fit: use a coordinate-MLP as the image representation, attach a differentiable forward model, and optimize the network so its synthesized measurements match the observed ones — no paired ground truth required. NeRF popularized the idea for view synthesis; works like NeRP and NAF brought it into sparse-view and limited-data medical reconstruction.

Differentiable Physics

Here the move is to implement the physics simulator itself inside an autograd framework (PyTorch, JAX), so that gradients flow through the physics.

class DifferentiablePhysics(nn.Module):
    def forward(self, params, x):
        # PDE / forward-model steps,
        # all differentiable
        return y

Once the forward model is differentiable, you can solve inverse problems by backpropagation, fuse it naturally with neural networks, and even optimize the system design. This is, to me, one of the most quietly powerful ideas in the field.

Known Operator Learning

Proposed by Andreas Maier (FAU Erlangen) and others, this approach keeps the classical operator (FBP, Radon, NUFFT) where it belongs and inserts a neural component only for the part we don’t know how to model. The whole pipeline stays differentiable and end-to-end trainable.

The payoff is a sweet spot: it generalizes better than pure data-driven reconstruction, and it’s more expressive than pure classical reconstruction. It’s a clean bridge between decades of imaging theory and modern deep learning.

The pattern underneath all of them

Step back from the specific names and the same shape appears every time:

\[\text{physics knowledge (PDE / simulator / classical operator)} \;+\; \text{ML model} \;\longrightarrow\; \text{faster, more accurate, better-generalizing}\]

Whether the physics enters through the loss (PINN), the architecture (neural operator), the representation (neural field), or the forward model (differentiable physics, known operator), the bet is identical: don’t make the network rediscover what we already know.

This is a field-wide trend, not a niche

It’s easy to assume this is a corner of medical imaging. It isn’t — the same paradigm is reshaping multiple sciences:

  • Weather forecasting: GraphCast (DeepMind), FourCastNet (NVIDIA), and Aurora (Microsoft) run orders of magnitude faster than traditional numerical weather prediction.
  • Molecular dynamics and drug discovery: equivariant models like NequIP and MACE, and the AlphaFold line of work.
  • Computational fluid dynamics: MeshGraphNets and companies like Neural Concept bringing learned surrogates into industrial design.
  • Materials discovery: GNoME (DeepMind) proposing large numbers of novel crystal structures.
  • Astrophysics: gravitational-wave detection and galaxy classification.

There’s even a growing industry around it — Neural Concept, NVIDIA Modulus, PhysicsX, Isomorphic Labs — all betting that ML augmenting scientific simulation is a durable direction, not a fad.

A word of honesty: this field is harder to enter

I’ll end with a caveat I wish someone had told me earlier. Compared to mainstream representation-learning ML — computer vision, NLP — SciML has a higher barrier to entry:

  • You often have to implement the forward physics yourself; there’s no off-the-shelf dataset and baseline.
  • It demands domain knowledge — optics, PDEs, signal processing — on top of ML.
  • Real data frequently requires hardware or clinical collaboration, which is slow.

In CV or NLP you can have a baseline running in a week. Here, getting to a credible starting point can take much longer. But that difficulty is also the moat: if you’re already in an optics or medical imaging lab, the environment itself is the asset. The physics you have access to is the thing most ML researchers don’t.

Where this series is going

This post was deliberately a map, not a tutorial. In future posts I want to descend from this big picture into specifics — starting with denoising, where I first ran into the limits of pure data-driven methods, and working toward how inverse-problem thinking reframes the whole task.

The shift I keep coming back to is simple to state and hard to fully internalize: from data-driven to physics-informed. Once you see it, you start noticing the same move everywhere.