Every time you run a program, you're benefiting from a piece of software most people never think about: the compiler. A compiler takes code written in a human-readable language — Python, C++, Rust — and translates it into the raw machine instructions a CPU can actually execute. But it doesn't just translate. It optimizes: reordering, restructuring, and trimming your code so the resulting program runs faster and takes up less space.
For decades, the decisions a compiler makes during optimization were governed by hand-crafted rules — heuristics built up over years by expert engineers. Now, machine learning is beginning to replace those rules. Compilers are starting to learn how to optimize, and the results are measurable. Here's how it works, why it's hard, and what it means for the software you use every day.
What a Compiler Actually Does (and Why Optimization Is Hard)
Think of a compiler as a very sophisticated translator with an editor built in. The translation part is relatively well understood. The editing part — optimization — is where things get complicated.
When a compiler optimizes your code, it makes hundreds of small decisions. Should it copy a small function's body directly into the places that call it (a technique called inlining), or keep it as a separate function? Should it repeat a loop's body multiple times to reduce the overhead of checking the loop condition (called loop unrolling)? Should it rearrange instructions to keep the CPU's pipeline busy?
Each of these decisions interacts with every other decision. A choice that makes one part of your program faster might slow down another part, or increase the total size of the compiled binary. And they must be applied in some order — which optimizations run first affects what the later ones can do.
This sequencing challenge has a formal name. Traditional compiler phase-ordering — deciding in what sequence to apply optimization passes — is an NP-hard combinatorial problem that machine learning approaches attempt to approximate more effectively than fixed rule sets. NP-hard means there's no known efficient algorithm that finds the perfect answer for every possible program. The number of possible orderings grows so fast with the number of optimization passes that exhaustively trying them all is impossible.
So historically, compiler engineers did what engineers do when faced with an intractable problem: they built rules of thumb. If a function is small enough, inline it. If a loop runs enough times, unroll it. These heuristics were carefully tuned and worked well on average — but they couldn't adapt to the enormous variety of real-world code.
Why Hand-Crafted Heuristics Have Limits
A heuristic is essentially a bet. The compiler engineer bets that, for most programs, a function shorter than some threshold is worth inlining. That threshold was probably chosen by testing a large set of programs and finding a value that performed well on average.
The problem is that your specific program might not be average. A function that's technically large might be called in a tight loop that runs millions of times per second, making inlining it dramatically worthwhile. A function that's tiny might rarely be called, making inlining it a waste of binary space. The hand-crafted threshold can't know this — it's a fixed rule applied to a dynamic, context-dependent problem.
As programs have grown more complex, and as CPUs have become more elaborate (with multiple cores, deep caches, and sophisticated branch predictors), the gap between what a fixed heuristic can achieve and what an ideal optimizer would do has widened. This is precisely where machine learning offers something new.
How Machine Learning Enters the Compiler
The core idea is to replace a fixed rule with a trained model — a system that has learned from examples what decisions tend to produce good outcomes, and can apply that learned knowledge to new situations.
The most direct version of this replaces a specific heuristic with a neural network. Instead of asking "is this function shorter than X lines?" the compiler asks a small neural network: "given everything we know about this function, this call site, and this program, should we inline this call?" The network was trained on real programs and real performance measurements, so it has internalized a much richer understanding of what actually leads to faster or smaller code.
This isn't magic — it's a learned approximation. But because a neural network can consider many more factors simultaneously than a simple threshold rule, it can make better-informed decisions in a wider range of situations.
MLGO: Machine Learning Inside LLVM
LLVM is one of the most important pieces of infrastructure in modern software development. It's the foundation under compilers used by Apple, Google, many game engines, and countless other systems. Getting ML-based optimization working inside LLVM, at production scale, is a significant achievement.
LLVM, one of the most widely used compiler infrastructures, has integrated ML-based passes such as Inliner and loop unrolling heuristics through projects like MLGO (Machine Learning Guided Optimization) developed by Google engineers.
MLGO works by training a policy — a model that, given a description of the current state of the compiler and the code being compiled, outputs a decision (inline or don't inline, for example). The training process uses reinforcement learning, a technique where the model learns by trying things and getting feedback on whether those choices led to good outcomes (smaller binaries, faster code).
The results on real code are concrete. Google published the MLGO framework in 2022, demonstrating that a trained policy could reduce binary code size by roughly 3–7% compared to hand-tuned LLVM inlining heuristics on large production codebases.
A 3–7% reduction might not sound dramatic, but applied across the billions of lines of code compiled at a company like Google, it translates into meaningful savings in storage, faster program loading, and better cache utilization. These aren't artificial benchmark improvements — they were measured on real, large-scale production software.
AlphaDev: Discovering New Algorithms, Not Just Tuning Old Ones
MLGO improves how existing optimization rules are applied. A more radical approach asks whether ML can discover new ways of doing things that humans haven't thought of.
DeepMind took exactly this approach with a project called AlphaDev. Rather than optimizing existing high-level code, AlphaDev worked directly at the level of assembly instructions — the raw commands that a CPU executes. It framed the problem as a game: given a task (like sorting three numbers), find a sequence of assembly instructions that accomplishes the task correctly and in as few steps as possible.
Using reinforcement learning — the same family of techniques that powered AlphaGo and AlphaZero in games — AlphaDev explored the vast space of possible instruction sequences. The results were striking. DeepMind's AlphaDev project, published in Nature in 2023, used reinforcement learning to discover new sorting algorithm implementations in assembly that outperformed human-written routines and were subsequently incorporated into the LLVM libc++ standard library.
This matters for a specific reason: sorting small, fixed numbers of elements (like exactly three or four items) is something that happens constantly inside larger programs. The standard library routines that do this have been hand-optimized by expert engineers for decades. The fact that an RL system found improvements those experts had missed is a genuine milestone — and crucially, the improvements were verified to be correct and adopted into real software that runs on real computers.
Optimizing AI Itself: The TVM Compiler and Ansor
There's an interesting recursion happening in this field: machine learning is being used to optimize the software that runs machine learning models. Neural networks need to perform enormous numbers of mathematical operations — multiplying matrices, applying functions across arrays — and the efficiency of those operations depends heavily on how they're scheduled to run on specific hardware.
Different hardware (a laptop CPU, a server GPU, a smartphone chip) has different memory layouts, different numbers of processing cores, and different optimal ways to break up computation. Writing optimal code for each combination by hand is essentially impossible given the variety of hardware and the complexity of modern neural network architectures.
The TVM deep learning compiler, developed originally at the University of Washington, uses an ML-based autotuner called Ansor that searches the space of tensor computation schedules to optimize neural network inference on target hardware.
A tensor is just a multi-dimensional array — the fundamental data structure of deep learning. A computation schedule describes how to break that computation up: which dimensions to process in which order, how to divide the work across CPU cores, how to tile the data to fit efficiently into cache memory. The number of valid schedules for a given computation is astronomical.
Ansor uses a learned model to guide this search — quickly identifying which regions of the schedule space are likely to be efficient, and spending more time exploring those areas rather than wasting time on obviously poor options. The result is that TVM can compile a neural network for a new hardware target and automatically find near-optimal execution strategies, without human engineers having to manually tune for every combination.
What All These Approaches Have in Common
Despite their differences, MLGO, AlphaDev, and TVM's Ansor share a common structure. Each replaces a search or decision process that was previously driven by fixed rules with a learned model trained on real performance data. Each treats compiler optimization as a problem of navigating a huge space of possibilities, and uses ML to navigate it more intelligently than a rule book can.
They also share an important constraint: the output must be correct. A compiler that makes your code run 10% faster but sometimes produces wrong answers is useless — worse than useless, because bugs introduced by the compiler are extraordinarily difficult to diagnose. This is why ML-guided compilers are typically structured so that the ML model makes decisions (inline or not, which schedule to use), but a separate, formally verified process checks that the resulting code is semantically correct. The learning happens in the decision layer; correctness is enforced by traditional methods.
Why This Matters Beyond Performance Numbers
The implications of this shift extend beyond any particular benchmark improvement.
First, it changes what's achievable. Hand-crafted heuristics plateau: once experts have tuned them for a decade, there's little room left. Learned policies can continue improving as more training data is collected and better training methods are developed. The ceiling is higher.
Second, it changes what generalizes. A heuristic tuned on the programs that existed in 2010 may not perform well on AI workloads that barely existed then. A model retrained on current codebases naturally adapts to what programs look like today.
Third, and perhaps most interestingly, AlphaDev demonstrates that ML systems can discover optimization strategies that human experts didn't know existed. This suggests that the accumulated expertise embedded in compilers — impressive as it is — isn't the final word. There are likely better ways of doing things waiting to be found.
The Road Ahead
ML-guided compiler optimization is still a young field. Current deployments tend to focus on specific, well-defined decisions (inlining, loop unrolling, instruction scheduling) rather than end-to-end optimization of entire programs. Training these models requires large amounts of real code and real performance measurements, which creates practical challenges for smaller organizations that don't have Google-scale resources.
There are also open questions about interpretability — when a neural network decides not to inline a particular function, it's difficult to understand why in the way you could inspect a traditional rule. This matters for compiler engineers who need to debug unexpected behavior.
But the direction is clear. The tools that turn human-written code into machine instructions are themselves becoming systems that learn. The compilers shipping in the next decade will increasingly rely on trained models alongside hand-crafted rules, with the balance shifting over time. For developers, this is largely invisible — the compiler just works better. For the field of software, it's a fundamental change in how one of its most critical tools is built and improved.
Sources
Every factual claim in this article was independently verified against the following sources:
- MLGO: A Machine Learning Framework for Compiler Optimization — research.google
- MLGO Framework Brings Machine Learning in Compiler Optimizations - InfoQ — infoq.com
- Faster sorting algorithms discovered using deep reinforcement learning | Nature — nature.com
- Mitigating the Compiler Optimization Phase-Ordering Problem using Machine Learning | Request PDF — researchgate.net
- Introducing TVM Auto-scheduler (a.k.a. Ansor) — tvm.apache.org
