The Biggest Problem In Real-World Computer Programming

Ask any ten people working in the programming industry what the biggest problem today is in their field and you're likely to get twenty or more answers. My inbox and reading list is full everyday with people telling me about all of the problems they're facing while doing the work of programming, many times accompanied with a white paper or landing--page link describing how they (and others) beat the problem and how for a bit of money I can prevent having these problems also.

Yet no matter how many books, video series, or landing pages are created, somehow all of these problems still remain. Indeed, aside from changing some buzzwords and case study details around, it's possible to run marketing, sales, long-form, and promotional material from twenty years ago today. (I made several examples of this before I realized that every person with the problem I'm going to describe would argue that no, each of these is a separate field with attributes that make it unique, even better, than the area it replaced)

We are unable to limit complexity in our solutions enough to solve problems and then move on to other problems.

Indeed, there is an old joke about consultants (or whizkid new workers) who come into a shop, rewrite everything using the coolness-of-the-week, get it halfway working, then move on to the next shop, leaving dozens or hundreds of workers spending the rest of their careers trying to clean up. It's funny because it's true. The overwhelming majority of work in our field might best be described as cleaning up after somebody else who never knew how to end a project.

There's another well-known trope about the "10x programmer", as if some people were magically-endowed with the ability to do the work of ten normal programmers. There's obviously no intelligence or skills test to give to spot these magical and mythical folks. In fact, some of the best folks in learning, applying, and using programming skills seem the least likely to be able to complete work. It's a trope, yet the industry also has plenty of stories of very small teams outperforming very large ones. In programming it's possible to have a team of ten provide more economic value than a team of ten thousand. I know of no other field where this is true.

Something's going on. Good programming seems like teenage sex: everybody's talking about it, everybody's an expert, yet nobody is actually doing it, and those who are generally aren't talking. If they are talking, there's so much selection bias at work that the skills and knowledge they're passing on to everybody else is hyped so far beyond reality that they're doing much more harm than good to those few poor and lost souls who are sucker enough to receive it.

We Have Siloed Ourselves Too Much

In business, the process of "silo-ing", named after farmers who create giant vertical metal casings to hold product, represents creating specialty areas which are vertically quite deep in their knowledge of one area yet become self-validating echo chambers which provide little or no value to all of the other silos around them. In fact, because folks in a silo are self-validating, any failure of the larger business is viewed as a failure of some worker in some other silo, and more information and skill is required in our own silo in order to be able to help them if one day those other folks ever get their shop in order.

Silos don't fix things. They don't grow the business. Over time, they just get bigger and deeper, eventually splitting into many more baby silos. Whereas the in late 80s you might find a small group in any shop responsible for data storage and retrieval, today there are dozens of fields that do the same thing with slightly different areas if focus. Each of these have seminars, career paths, books, training, certifications and so forth. We still do data storage and retrieval, of course, but each company does it in a highly-nuanced way with deeply-trained experts in that particular set of technologies.

When problems arise, there are standard ways of not-fixing them. We throw more people at the problem, we dig the silos deeper, we split up silos, or we hire outsiders who have their own IT problems and hope that their problems don't become our own. Rewrites never work because we have so many people specialized in so many areas that the only thing rewrites do is wipe the slate clean and re-jigger our collection of silos. Throwing it to some other company seems to be the best solution, although then you end up with vendor capture, somebody else owning all of your data, and so forth. The best solution to being completely broken looks like hiring a lot of companies who don't appear to be broken to slowly bleed you over the years instead of you dying right now. Eventually, a startup will redo what your teams of thousands is working on using a few people. You'll either use government to make you into a monopoly or die. Some companies prolong their death by actively gobbling up new companies as they come along. It works for a while.

This is happening everywhere you look in the world where programmers are used to provide economic value, and that's a lot of places. There's no better indication I know of than this that something is drastically broken. Yes, there's the natural creative destruction of the market at play, but we're not talking about a new product or service radically doing a better job of things than the product or service it replaced. We're talking about functioning organizations with a clear and workable value preposition destroying themselves with or without the presence of a competitor in the market. There are industries where all of this churn happens and nothing new has been created at all. We're just broken and keep becoming broken in new and interesting ways. All of it involves programming.

Magic Beans

I don't have any magic beans to sell you. In fact, the more I looked at this the more I realized that this was a problem with academia itself, not programming. Programming just feels the heat because we're at the sharp end of the stick where smart folks are hired to go do something useful.

I came to believe that there are four philosophies, each of which has over time become highly-specialized and over-siloed themselves, which mix to fix the problem of complexity: Pragmatism, Semiotics, the Philosophy of Language, and Analytical Philosophy. The last one, analytics, formal systems, and so on, is the one programming is taught under. We programmers seem destined to spend our lives here, laboring under new programming languages and platforms, arguing over tabs, spaces, semicolons, monads, and so forth. We dig away at our little silos, strengthening the walls, adding to the depth. If those other folks ever get their house in order, we'll be ready!

Online essays can't cover a critique and effective mix of four philosophical traditions, of course, so I'm limited to only one, and even then only to an introduction and hints at what might be possible.

Tripartite Semiotics

Tripartite Semiotics, the way Charles Sanders Peirce introduced the subject, is the field of inquiry that spun out of the observation Peirce made that smart people, using good logical systems, could endlessly disagree about things. What was wrong with logic that would allow this?

He came to the conclusion that there was nothing wrong with logic at all; we were just missing an important piece when we sought to use it. It wasn't the signs, symbols, and processes we used, it was the fact that it took a third party to assign these symbols to real-world things. Math and logic are perfect; it's only in our use of them that we run into trouble (or gain value).

A hundred years before some computer guy said it, Peirce said it: naming variables is hard! It took us 100 years to get back to the same spot that Peirce began at.

Cut To The Chase Already!

I believe this all ends up with a Types-For-Tests Microservices model of technology delivery. That is, I believe we should construct programs such that the programs themselves can never be in a state of error and instead become tokens in higher-level programming systems. Either we come up with a recursive model of technology solutions creation that scalable or we keep stacking the cow manure deeper and deeper and creating more and more and better silos.

My life's work seems to have ended up describing this and how to make it happen, tut that's a discussion for another day. For today, here's an introduction to Tripartite Semiotics.