AI Automates Mathematical Proof Of Code Safety

Software verification is entering a new era where artificial intelligence handles the heavy mathematical lifting.

Anthropic has demonstrated that its Claude model can autonomously generate formal proofs for existing software code. By partnering with Theorem, the company used Claude to verify the correctness of 123 open-source libraries from Google's OSS-Fuzz project. In a matter of weeks, the system produced approximately 180 million lines of mathematical proof code, a task that traditionally requires specialized human expertise and significant time.

This development addresses a long-standing bottleneck in the tech industry. Historically, formally verifying software has been prohibitively expensive and labor-intensive. For example, a team of specialists required 20 person-years to verify the seL4 operating system kernel, limiting the practice to critical systems like aviation software. Anthropic’s Frontier Red Team sought to change this dynamic by allowing the AI to take over the proof generation process.

Rather than writing new software, Claude analyzed existing code by translating crash tests into mathematical problems and examining compiled machine code. The model then produced extensive proofs, which were checked by an independent verification program called Rocq. Anthropic describes this workflow as fully autonomous, noting that the process requires minimal human review, although the underlying formal foundations were established by humans at Theorem.

The system proved effective at identifying latent defects. During the verification process, Claude detected hundreds of bugs that conventional testing methods had missed. These errors often only manifested under specific conditions, such as unusually large input files or particular access sequences. In one notable instance, the team found as many security bugs in a widely used PDF reader in a single week as are typically reported over an entire year.

Furthermore, the AI identified a logical error in another application that had remained undetected for more than a decade. Anthropic notes that even when a complete proof fails, the attempt still surfaces genuine bugs, adding value to the testing pipeline.

However, the approach comes with specific limitations. Anthropic clarifies that formal verification does not guarantee a system is entirely error-free. The proofs are valid only under strict conditions, including a specific processor architecture, single-threaded execution, and inputs smaller than 64 MB. In real-world scenarios, these libraries often run on diverse hardware and configurations, meaning the verification covers a defined subset of potential behaviors.

The computational cost remains a significant factor. Individual proofs can consume hundreds of millions of tokens, though Anthropic has not disclosed the total project costs. Efficiency is improving, with newer models like Opus 5.5 solving half the proofs using 40 percent fewer tokens than its predecessor. This trend suggests that the economic barrier to entry for formal verification is gradually lowering.

If AI can both write code and mathematically prove its safety, the software development lifecycle may shift significantly. Developers could place greater trust in AI-generated code, reducing the time spent on manual review. This could lead to cost savings across the entire coding process, not just verification.

Looking ahead, Theorem’s roadmap targets critical infrastructure libraries such as glibc, OpenSSL, and cURL. Anthropic expects that AI-driven formal verification will soon become as routine as standard testing, potentially transforming how critical software is built and maintained.