Why AI Agents Struggle with Software System Maintenance

Can AI agents maintain complex software? No, they're not ready yet. The hype suggests otherwise, but the reality is different.

Key Takeaways

• AI agents excel in additive tasks, not transformative ones.
• Understanding system invariants is crucial for maintenance.
• LLMs cannot autonomously manage large codebases.
• Current tools are not yet production-safe or reliable.

The Core Challenge: Additive vs. Transformative Work

AI agents can generate correct code snippets and automate documentation. But they struggle with maintaining existing software systems. This boils down to a fundamental distinction: additive work versus transformative work.

Additive Work: Where AI Shines

Additive tasks involve creating new elements without altering existing structures. For instance:

• Reading a repository's structure.
• Mapping out project components.
• Planning a set of changes.

These tasks are self-contained and don't disrupt the system's causal structure. LLMs, like those developed by OpenAI, excel here because they can produce statistically plausible continuations of text, whether it's drafting new functions or writing documentation.

Transformative Work: The Stumbling Block

Transformative work means changing a live system's behavior while preserving its invariants—those critical conditions necessary for the system to function correctly. This includes:

• Writing and integrating new code while respecting dependencies.
• Understanding and maintaining system constraints and boundaries.
• Evaluating the impact of changes in complex, interdependent systems.

This level of understanding goes beyond pattern matching; it requires reasoning about consequences—a capability current LLMs lack. As Phroneses puts it, this gap prevents LLMs from autonomously delivering software.