Honest Microservices

Recently there was a discussion in the tech community, "Why SOLID is all wrong". We have these discussions every few years or so, and about every imaginable aspect of creating technology.

Welcome To The Endless Cycle Of Tech Bullshit

1. Have you seen this new thing called X? It's like Y only with more coolness.
2. X is pretty good!
3. X is great!
4. All the X!
5. Problems I've had with X
6. You know, X is pretty bad
7. Destroy all X!
8. X becomes Y
9. GOTO 1

We re-brand the same old topics different ways, then come back to the same discussion, only using different terms and slightly-different types of X. (But this X is NOTHING like that old X. This is actually Y.)

I have never seen us go through such a cycle where things got simpler. Each cycle through always involves new books, new consultants, new conferences, new user groups. No matter whether the problem being fixed is big or small, no matter whether the average developer needs the new fix or not, each loop through always gets more complex.

Hyperbolic, ranting essays about loving or hating something get noticed. Reasoned discussions that progress over time do not. Such is the way of the web. As one observer put it, "I'm done with essays titled 'I tried Microservices and it ruined my marriage!'" There's far too much drama, and when you try to pin people down on foundations, there's also far too much hand-wavy "Just write simple code", and "Keep your code clean". These are platitudes, not applied foundational theory in which to build technology systems.

Which brings us to microservices.

If you've been following along with the books and blog essays, you know we've built a theoretical foundation for both analysis and design, both the process of reasoning about what to code, and organizing the code itself. One thing naturally leads to another, and before you know it you have a way of coding that involves neither slogans nor homilies.

In general, it's been Info-Ops I (what to build), followed by Info-Ops 2 (how to build it). They both used the same underlying theory. Info-Ops 2 introduced Code Cognitive Load. This led naturally to Incremental Strong Typing. CCL could be used no matter how you code. IST could be used in most environments but naturally pushed you into a certain style of coding.  One logically follows the other, and we end up at Honest Microservices.

Key point: I did not start off with the intention of inventing or talking about anything new. I simply wanted an underlying self-consistent logical theory that would cover all aspects of technology development. Developing that forced me here. (Much against my will)

Why the word "honest" in the title? Because once the buzzword "microservices" became popular a long time back, the cycle of bullshit began again. By now the term is so polluted that good luck trying to have a conversation about it. Here are things that are not honest microservices:

These Are Not Microservices

1. Your SOA cut into services.
2. Anything involving a large framework. What part of "micro" are folks missing?
3. Development patterns where you deploy things into containers. You want containers,  do containers. Fine with me. Deployment decisions have nothing to with microservices. If you're coupling your code to the architecture that tightly, re-read #2.
4. Things with some cool language. Once again, if you like cool languages, use them. Sounds fun. It's just has nothing to do with microservices.
5. Systems with thousands of little tiny pieces of code scattered all over the place making it impossible to reason about anything. You're loving the idea to death. You've gone too far the other way. Both the "micro" and "service" part are equally important.*

Five is particularly interesting, as many proponents of microservices who actually have some idea of where they're going  end up here. That's why Incremental Strong Typing demands Honest Microservices: it teaches appropriate scoping and explains how the types of things inside your code (types of things, hardy har har. I'm here all week, folks!) directly relate to and drive your scope -- and vice-versa. This isn't "Do microservices because they're cool". This is "You do CCL right, you do IST right, you're forced to actually start writing Honest Microservices" (Although many resist this procedure for some time) There is no sales pitch or coolness. It just is.

Honest Microservices Defined

1. An app that cannot crash. It has a way to fail gracefully but that doesn't involve some downstream coder poking through your call stack.
2. An app that does one and only one useful business function. What's that? See IST.
3. An app that is composable with other apps from the command line.
4. An app that uses dynamic, free-form, late-bound text streams to talk to other apps.
5. An app that can be reasoned about, modified, or completely replaced by a maintenance programmer many years from now with little or no preparation.
6. The app must have tests.
7. The app must do what the tests say it does.
8. The app cannot do anything that the tests do not cover.
9. These tests exist in a compilation unit separate from the app itself and are in no way coupled to it.
10. When in doubt, think of your compiled app as a pure function running inside a program. That program is a plain, vanilla, default operating system.

Following these rules directly forces code into a shape that it easy to understand, manage, refactor, and discuss with business partners. Doing this involves no special hardware or deployment requirements. You can deploy the same set of apps thousands of times around the world or all on your local computer. Deployment has zero impact on code, and vice-versa. Congratulations. You've reached the spot where DevSecOps are three equal concepts, instead of each of them fighting to take control from the other two.

My App Is Too Complex This Is Stupid

That's exactly what I would have if somebody had told me about this ten years ago. (My second objection is that it sounds like a chore and no fun at all. I love having fun coding, and this can be fun, but that's a discussion for another day)

Here's the thing: if you've followed the rules above, you've solved your complexity problem as you coded the system. Tests are decoupled from the apps that fulfill them. When microservice A feeds microservices B and C which then feed a dozen more microservices, those un-typed, dynamic text feeds can be tested as a whole. Outputs have to work with inputs. Error paths can't exist without tests.

CCL leads to IST which leads to Honest Microservices. Honest Microservices lead to supercompilers.

Supercompilers

What do compilers do? They do a lot of things, but basically they organize the logical symbols and their relationship along with the data required such that the computer has the resources it needs to do the things the programmer wants it to do.

There's no reason that this has to result in a single deployment unit. In fact, that's been our entire shtick all along; if we decouple all the things and work only in small units, we can then re-arrange those units in any way we like. We can also maintain those units much easier.

If a normal compiler compiles to an executable, a supercompiler would compile to a cloud deployment. All of those considerations like database engines, scalability, load-balancing, health checks, certificates, pods, edge-computing strategies, CDNs, and so forth would exist as supercompiler options and config files.

We've even got a model of how the innards of the compiler should work, how to assemble functions in a hot-swap, flexible way: the Erlang OTP model. There's no reason that general way of working with live code wouldn't work with other languages now that we've separated things out. A supercompiler even gives us a lot of other things, like a place to bolt in TLA+ so that we don't run into concurrency issues.

We should look at some sample code. We should build a primitive supercompiler. We should talk about where OO and SOLID and all of that other goodness comes into play (It  definitely does). But none of those discussions would have any common ground if we don't begin by recognizing that fundamental principles can drive technology development from idea to deployment, without involving a thousand context switches. I know this because that's how it happened the first time. The second time let's do it so we don't loop back through here again in ten years.

Keep it simple, write clean code, brush your teeth, make your code testable: in our business we're never at a lack of famous smart people giving motivational speeches trying to get us to do the right thing. They mean well. And these things are true as far as they go. They just don't go very far.

My dudes. We gotta break out of this infinite loop. Every cycle through we just continue making things (for the average user) needlessly more and more complex. Please, more making stuff people want and less one bunch of nerds marketing cool stuff to another bunch. Future generations will thank us.**

*You may think that other concepts, such as Domain-Driven Design (DDD) or Ubiquitous Language (UL) also belongs in this list. While great concepts and things I believe in and use daily, the process we're using is working from the bottom-up starting with CCL and IST. We'll get to DDD and UL, but we're going to come to conclusions that change the way you use them. They're not needed here and putting them here would blow up the scope of the essay.

**It's a fair cop to say "Daniel, you're just taking well-established patterns of computing and adding a new label to them. You're just starting the bullshit loop again at #1" Here's the difference: the compiler analogy means that the average beginner programmer only has to work a simple IDE to create solutions in whatever environment they find themselves in.  They type one line of code to make the IDE run, then begin making stuff people want. The rest of it goes away, but like all compilation processes at the same time it can also get extremely complex without changing that person's development experience. This is my acceptance test for what I am describing. Make that work and you can call it anything you'd like. But for the love of the Great Pumpkin and All Things Charlie Brown, pick some label and stick with it.