There’s a widespread assumption that AI code assistants will get you to a solution faster. That’s true. But there’s an unstated assumption underneath that: the solution will be high-quality.
That assumption needs testing.
The Training Data Problem
Most large language models trained for code generation were fed primarily by public code—GitHub repositories, Stack Overflow answers, academic source code, open-source projects. This data is vast but representative of average code quality, not exceptional code quality. The crown jewels—the proprietary, battle-tested, performance-optimized implementations that actually run enterprise systems—stay behind firewalls, locked in corporate repositories, not part of any training data.
For IBM i systems specifically, this gap is even wider. Enterprise RPG and COBOL running on IBM i often embeds 20+ years of domain-specific business logic, performance tuning, and integration patterns that never existed in public repositories. When an AI model generates RPG code based on public examples, it’s working from snippets and patterns that don’t reflect the actual constraints of production IBM i workloads, where data volumes are measured in gigabytes, transaction throughput matters enormously, and certain database access patterns are critical for system performance.
What the Data Says
A December 2025 CodeRabbit analysis of GitHub pull requests found:
AI-co-authored changes produced ~1.7× more issues overall than human-only PRs.
More specifically:
- Logic and correctness problems: 75% more common in AI-generated code
- Security issues: Embedded patterns from deprecated or insecure practices
- Edge case handling: Poor treatment of unusual but critical scenarios
The report concluded that public training data frequently embeds vulnerabilities, deprecated patterns, and context-free snippets that are ill-suited for production enterprise work.
That’s not because the AI is stupid. It’s because the AI was trained on data that doesn’t represent production enterprise code.
What This Means For IBM i Modernization
When you’re modernizing legacy RPG or COBOL systems, you’re working with code that:
- Encodes business rules that aren’t documented anywhere else. The original developer who wrote it retired in 2015.
- Handles integrations between multiple internal systems that don’t have formal specifications.
- Implements performance optimization for specific data volumes and transaction patterns.
- Contains workarounds for edge cases discovered over years of production running.
If you feed AI a high-level description of what an RPG program does and ask it to generate a modernized version, the AI will produce code that’s plausibly correct. It will compile. It might pass your basic test cases.
But will it handle the edge case that only occurs twice a year at month-end close? Will it preserve the exact sequence of database operations that matters for transaction consistency? Will it implement the business rule that the original developer called “the December shipment quirk” because nobody wrote it down?
Not necessarily.
What AI Actually Does Well (And Doesn’t)
This is not an argument against using AI for code generation—it’s an argument for appropriate use. AI code assistants excel at accelerating routine refactoring work. When converting SYNON-generated code to RPG Free Form, AI can draft the basic structure and boilerplate, though a developer still needs to validate the business logic. It’s excellent at generating test scaffolding by creating comprehensive test cases based on a function’s signature, though developers need to review and ensure the tests actually cover the business logic. Handling data type conversions—mapping fields from legacy structures to modern schemas—is something AI does well because it’s structured and mechanical. Documentation and comments are AI’s strength, generating meaningful variable names and documenting function intent naturally.
AI struggles when facing complex business logic because if there’s ambiguity about what a program should do, AI will pick one interpretation and commit to it, and that interpretation might be wrong. It struggles with integration points—if an RPG program needs to interface with three other systems in a specific sequence and that sequence matters for data consistency, AI won’t know that without explicit instruction. Performance constraints present another problem: if a certain database operation needs to run in a specific order to avoid locks, AI won’t know that from the function signature alone. Edge cases are the final struggle—the weird scenarios that only happen under specific conditions are things AI only knows about if they’re documented or represented in test cases.
The Right Approach
Use AI code assistants thoughtfully:
-
Scope boundaries clearly. Tell the AI exactly what you want it to do. Not “modernize this RPG program,” but “extract data entry validation logic and convert it to RPG Free Form while preserving this exact sequence of field checks.”
-
Start with structure, not logic. Have AI generate the basic procedure definitions, error handling frameworks, and data structures. You write or carefully review the business logic.
-
Test aggressively. Every AI-generated change needs testing against your actual business rules. Unit tests, integration tests, regression tests. If the AI knew about edge cases, it would have handled them; if it didn’t, your tests will catch them.
-
Pair with experienced developers. Junior developers reviewing AI-generated code will miss problems. Senior developers who understand the business logic and integration points should be the ones approving critical changes.
-
Use AI for breadth, developers for depth. AI is great for handling lots of routine conversion work. Developers are great for understanding why code exists and what it’s supposed to do.
The Economic Reality
Here’s where this gets practical: AI acceleration saves time and reduces risk if you use it correctly.
A typical SYNON-to-RPG conversion without AI:
- 1 developer, 10 weeks
- High confidence in correctness (because the developer hand-reviewed everything)
- Cost: ~$30,000
A SYNON-to-RPG conversion with AI + developer oversight:
- 1 developer + AI, 6 weeks (AI drafts, developer reviews)
- Similar confidence in correctness (because the developer is still doing critical review)
- Cost: ~$18,000 (plus AI subscription)
The time savings are real. The quality is comparable. The key difference is workflow: AI handles boilerplate and routine refactoring; developers handle judgment calls and correctness verification.
What To Watch For
If a vendor tells you “our AI will fully automate your modernization project with zero developer involvement,” you’ve found a vendor who doesn’t understand enterprise code quality requirements.
Real AI-assisted modernization requires developers. The AI makes those developers 2–3x more productive, but it doesn’t eliminate their judgment or responsibility.
If you’re considering AI-assisted modernization for your IBM i systems, the question isn’t “should we use AI?” It’s “how should we use AI to accelerate our work while maintaining the quality standards our business requires?”
Let’s talk if you want to think through how AI tooling fits into your specific modernization strategy.