My AI journey continues to be very strange, even after over 18 months of playing around with it.
When I started, I figured there were really only two ways to approach figuring out what this new tool would do: either give it really simple things to do and hand-verify, or give it extremely hard things to do. This would also require some kind of verification, but I didn’t know if it would be done by hand or not. I started off doing everything manually.
This was very informative, especially when llm coding was new. Sometimes it worked, sometimes it didn’t work. It never coded the way I would, but that’s fine. Interestingly enough, it always told me that it did a good job and also that my ideas were awesome.
What can I say? It obviously knows talent. /s
I immediately switched to some version of TDD in order to do verification. Verification turned out to be more difficult than I imagined, and it continues to be difficult even now, only in more complex ways.
One of my main initial findings was that my AI was never afraid of offering me a simple solution that increased overall system complexity and then system maintenance. It would do thing A, do thing B, kind do thing C, get stuck, then I’m spending hours at the terminal trying to get it to fix itself. This always turns out to be some version of “you made it more complex than it needed to be, and now your maintenance time and cost is completely overwhelming development effort”
This is pretty common for programmers, so I let it go and tried to carry on. At times I would get frustrated and tell it something like “Isn’t there anything you can do to speed up testing?” Testing became a mess because even though it did test the thing I asked for, it did not code the test the way I would have. This then created a double-barrel screw-you: the architecture was a complete mess that did what I wanted to until it stopped, and the testing system was also a total mess that did what I wanted to until it stopped.
Ok, I figured, I don’t care about the architecture. You code it, you run it, you fix it if it’s broken. Cycles are cheap. Go deal with it.
Now everything, every change, had to be fully regression tested before commit. This is also quite common in the development world, that’s why the Good Lord made CI/CD pipelines, so good. So far it’s looking a lot like a real average skilled coder.
But as the code grew, the regressions grew. Within a couple of weeks I’m stuck with full day regressions. They use a lot of tokens, they kill the dev rhythm. Again, can’t you do anything to speed this up?
Sure it could. It started marking tests as SKIP. Then it didn’t run the test and that was much faster. Took me a bit to figure out that trick.
At this point, Jan-Feb 2026, I decided to start top-down. I would make principles, like atomic testing, zero deps between modules, and so forth. In addition, I would use an LLM administrator instead of flailing around with something that seemed like a stupid waste of time.
It worked great. It flailed around instead of me, I was happy, and then we were back to full regression tests taking all day. Either suffer that, the AI seemed to say, or simply deal with the fact that I’m going to say “trust me” and don’t worry about it.
I was becoming a dev customer instead of a dev lead. Nope. That wasn’t going to happen. That would defeat the purpose of trying to understand this rascal. I would create another LLM, a QA. He would handle these testing issues early on, not get involved in the architecture, and all would be well.
This would be hilarious if I weren’t the one going through it. QA is also a very common pattern. QA failed by getting involved in quality decisions that didn’t involve actual code but involve how the code was made and tested. It turns out you can make this a complete mess also.
Every time I asked any of my LLM guys, they always had a good reason for whatever they were doing. Didn’t I know that Requirement 12.5.8a required that this thing had to happen and had to be fixed.
Why yes, yes indeed. Problem is that I never made that requirement document. You guys made it.
At this point it’s good to note that my test software project was simply 6-12 dotnet apps that did things the OS already did, like copy, ftp, and so on. The only complexity I put in there was that it had to be OS independent and these apps had to work like Unix apps do. That seemed to me to be the minimal amount of requirements that would create the most downstream complexity.
My strategy worked great. The LLM execution did not. But it always looked good. And it always told me how awesome I was.
I told it to shut up.
It did, for a while, then I forgot I told it that (context overrun) and went back to praising me. In fact, this was a big part of the problem. If you learn nothing else from this essay, learn this: either create a context so huge that it can hold everything about the problem you’re solving from the beginning of the work until the end, or start enforcing deterministic compilation units that use provable code and don’t need to be looked at again.
You’re alternative is to do some version of “trust me”
I’m not discounting that approach. I’ve been using it for the last month or two. Let’s assume the LLM coder is simply a lazy SOB that will do the minimum required to make me happy so I’ll leave it alone. We’ve all worked with this guy. Who cares as long as he doesn’t become a huge pain? I don’t. I have value I need.
But if I were coding a rocket? A healthcare device? If I were doing some sort of audit for somebody that was very important? Fuck yeah it would matter. That sounds like a complete disaster on the way, a disaster created by people unable to understand what they did to cause it.
Bottom line: AI/LLM coding is the most magical thing I’ve seen in my entire 40+ year career. I can have an idea over dinner and have it completed and online before I go to bed. I can have a voice conversation when I’m traveling and have the LLM work just like a real admin. I can easily see huge and life-changing advances in science. It is an incredible time to be alive.
But I’m also keenly aware of this: the way this is being sold, the way normal humans are interacting with it, the trust we’re using? Like everything else I’ve experienced so far, it’s going to work until it doesn’t. What then?
