We Are Happy Consumers Of Our Own Inefficiencies

This past week I've been reading the excellent book "Citizens" about the French Revolution by Simon Schama. The events around the French Revolution are fascinating. It's a powerful story that's both well-documented and has so many potential analogs with current events.

France was a modern, complex, world-spanning government that had done many, many great things, including being a critical component of the birth of the United States of America (with their support during the Revolutionary War) Schama's thesis in "Citizens" is that France was not only a modern, powerful country, it was a strongly progressive one, wanting to shed itself of centuries of feudalism and the old bad ways of doing things. It wanted to launch itself into the future, just like the British colonies in the New World did. They were huge fans of the ideas and themes of The Enlightenment and America's Founders. Hell, Benjamin Franklin was the close thing the 18th century had to Elvis. Why can't France copy the U.S.?

But they couldn't get there from here. The path was not direct. In that struggle I found an analog to modern businesses. Today we have hundreds if not thousands of old businesses serving various markets, delivering value even though over time they all develop horribly inefficient ways of doing things. People in these organizations know the problems they have. They realize they are doing things in a suboptimal manner. They see younger, newer companies come along with new and improved ways of delivering that same value. Quite frankly, it makes many folks angry. In most all cases, these younger, more innovative organizations end up disrupting and completely taking over the market from the older companies.

Why is that?

In both cases we have large, complex, sometimes world-spanning organizations that have been around for a good while. We have upstarts showing that there might be newer, better ways of doing the same thing. We have people in the older organizations being completely aware of the situation yet unable to change. It's not that the people are bad. It's not that they are old, set in their ways, stupid, or somehow unable to see the advantages of thinking and acting in new ways. Indeed, as Schama points out in his book, the more passion and desire the French brought to their movement, the worse the outcome became. Both his thesis and my observation of the global economy and how disruptions happen in industry are strikingly counter-intuitive. Yet I can easily think of dozens of examples where the parallel holds up. It's uncanny.

It occurred to me that a second book that I read many years ago covered the same ground, only from the side of industry and not world history. It was "The Innovator's Dilemma" by Clayton Christensen. Chirstensen's thesis was, in brief, that companies cannot afford to truly innovate their products because to do so would be to destroy their existing commercial base. You can't eat your own market share on a gamble that some new, super-cool product is going to be a hit. You're already making money. You can't gamble it all every day. But startups can.

Here we have two things: refactoring governments to be better and refactoring organizations (whether commercial or charitable) to be better. Are they similar enough for useful comparison? I resisted this idea; it seems far, far too facile. Specifically, I had always heard that the French Revolution failed because of separate groups with separate agendas and too much debt. It was a clash of groups and economics. But the more I read, the more it bugged me.

Finally, I ran across a quote in "Citizens" that stuck with me to the point that I had to write about it whether I wanted it to or not. Paraphrased, it was a noble, in support of  change, wanting the best for his country and his people, looking at the overall problem and complaining about how difficult it was. Everybody wanted to be like the U.S. He concluded something like "We didn't come out of the deep woods like hillbillies to create our governments completely from scratch. We already have existing organizations. They provide value. We can't destroy everything and expect anything good to come from it. Something has to happen to these existing structures and practices and the people in them"

He was right. As bad as things might be, we can't set off an organizational "bomb" and expect to provide value any more. In fact, quite the opposite would happen: lots of people would thrash around, blaming one another and wasting time and resources on things that seem like they might be good but turn out not to be.

Wow. I hear very similar things from folks in the org change business everyday. Yes, we have cool existing things, like CMMI, DoDAF, Six Sigma and so forth. We can look at other organizations that have newer and seemingly better ways of organizing. They may call these systems Agile or XP, SAFe, holocracy, or whatnot. When you see an upstart competitor eating your lunch with miniscule staff and budget? Something really cool is going on! There are great things happening!  We're not stupid. We want to adopt many of these cool ways of delivering. But we already have an org structure. We already have all these ways of providing value. What are we going to do with those structures and the people in them?

Wow. This is a much different problem from simply picking the process or structure that 's best. We're also not talking about conflict between various groups of people. This is not about people forming clans or economics.

We betray ourselves. We are so used to picking right and wrong. When that fails, we double down, assigning categories and going to a lower level. This type of people are mostly right. This type of people are mostly wrong. It's a much, much more difficult thing to imagine an organization where everybody sees and wants to change. There are no good or bad guys. There's no villains. The problem is that no matter how hard we try, we can't get the existing structures and people to work inside the newer ones. Passion, training, desire for change, and evangelizing is not helping. There's some kind of deeper, personal mismatch going on here inside each of our heads that has nothing to do with ideology or knowledge and only shows itself at scale.

What is it?

Looking At Organizations As Marketplaces

At this point, still trying to resist making the comparison between political revolutions and org change, I thought of a bunch of other essays I've read over the years.

They all seemed to agree with my thesis, as long as my thesis was carefully worded. (It would be quite easy to overstate the case.)

I will synthesize all of these inputs into a simple list that drives a conclusion. Please feel free to take issue with any of these items.

1. Organizations, whether political, commercial, or non-profit, exist to provide value to folks outside that organization. Without some planned value exchange, there is no reason for structures of people to exist doing various things.
2. Small organizations that do only one thing with a small group of people are lucky. They have immediate and direct feedback about how much value they deliver. Being small they can adapt and change quickly.
3. Large organizations cannot do either of these things. In fact, even with a handful of people it can become first hazy and then impossible to know for certain exactly why you have a job and whether or not you're providing value in the way the larger org wants to or not. You look to your peers, your org structure, and your practices to provide you with that information. There's nowhere else to look.
4. It gets worse. Even a small list of values agreed upon by the entire org can be interpreted in vastly different ways by different people in different situations. We set "meta" things up to try to self-correct, things like balanced-scorecard, customer councils, requirements meetings, and so forth. It's not that there is a lack of ways to continue learning and stay on-track. It's just the opposite. When it comes to ways to kick ass and learn, we live in a land of plenty. There's an almost infinite number of ways to try to keep agreement and value production in line.
5. We have choices to make, as individuals, sub-groups, and orgs. These are choices without clear objective criteria as much as we would like otherwise. In the final analysis, each employee or org unit is responsible to participate in lots of indirect things to keep the larger organization alive and create and adapt ways to keep doing that.
6. These things drive every group of people in our org, from pair programmers to executive leadership groups, to create their own languages, slogans, and value criteria. Many times we participate in several of these little cloisters. Inside each little group people honestly and dilligently strive to accomplish the overall mission. But I have yet to see a situation where once you get a step or two outside that group, it doesn't start looking inefficient, mean-spirited, or even counter-productive to the goals of the larger organization. We're trying hard. It's always those other folks that seem to not get it.
7. This is because each of us consume the activities we enjoy, not the overall results of those activities. By the previous statements, we've established that it's impossible to do otherwise. You like making powerpoints? No matter what happens, at the end of the day, you're going to make the gol-darndest powerpoints you can in order to help the org thrive. You like agile, or process X? No matter what you pick up to do, I can guarantee you that you're going to be doing those things as you grow. After all, you like these things because you know that they're the best thing for the org to do! How could any org possibly exist without having meetings where they show graphics? People like doing what they like doing. They will always gravitate towards that. And those things always have some value. They also have some negatives. We tend to dismiss the negatives (which easily go far beyond our own work) and consume and enjoy the experience of our "home" things.

We consume these processes and actictivities we like and enjoy without consideration or understanding of the larger impact because we're not omniscient. It can be no other way. We "buy" these things by reading blogs, books, going to conferences, and so forth. Once "purchased", we bring them back to our daily work and try hard to make them work. Over time we become semi-fans of our chosen brands. They are familiar, comforting, and there's no denying that there's real value in them.

What happens when things get bad? What if we specifically do X and it's not working? The answer is obvious. We either aren't doing it right, we aren't trying hard enough, or we need to throw it all out and go "buy" some other activity or process. Such is life as humans in any sort of large group of humans doing anything. We make local decisions based on our personalities and understanding of the world, we develop brand loyalty, then do the best we can to make a positive difference based on these preferences. We purchase ideas for learning and improvement as products. We use them as products.

The organization is a marketplace where we sell to one another our ideas and ways of thinking about the world. We are both the producers and consumers of products under the general category of "If we only did X, we would do a LOT better".

These products, like all products, face the Innovator's Dilemma. That is, you can have a group of people in various working units that provide real value. We need those folks and all the things they do! New concepts may provide vastly more value in certain areas, but they involve refactoring and rethinking large hunks of everything and it stands to reason that there are lots of pieces that will go unaddressed. It's just like that French dude was saying. It's impossible for any of us to judge if one or two new ways of doing things, products, can replace a bit of 17 other processes and groups, even if we can see other organizations who use these to great effect. So we continue to do the things we can and act in the ways that we know provide some value, even if we desperately wish we could do things the newer and better way.

We can easily determine that process or org structure X is better than process or org structure Y. What we can't do is refactor the entire organization in our heads to make sure that all the other processes and structures will continue working and doing the necessary work that they do while we try to improve. New things always eat into the value of old things, and with complex systems they always do so in unexpected ways. Our improvements and innovations prevent future improvements and innovations.

Our like/dislike and categorization systems are built into each of us and required for us to function as any sort of intelligent creature. There's no getting around that. Yet these same systems, when scaled, always create impedance mismatches. These mismatches end up making our organizations fragile, our ability to evolve weak or lapsed. We are left with simple marketing and purchasing decisions.

It's not a problem of "thinking outside the box". We are not lacking passion. It's not any easy thing we can group and label. We are stuck. The ways we self-organize and work with one another are products that we sell to one another inside the internal idea/process/structure marketplace that we call our organization, and the Innovator's Dilemma is as relevant in this marketplace as it is in others. Passion, knowledge, even demonstrations of new ideas can't fix this. In fact, to Schama's point, the more passion we bring to the effort, the worse we may end up doing. You don't make the round peg fit in the square hole simply by getting a bigger hammer, no matter how much that peg needs to be in there or how good it feels to bang on it.

Where to go from here? I've ideas!  They're for another day. Many of these ideas and themes that are going into book 3 are coming along quite nicely!

It's a fascinating parallel.

tl/dr: Politicians and noobs to organizational architecture take a simple-minded approach to change, focusing on values, core principles, or other easy-to-remember categories and slogans. True experts take a position of profound ignorance, agreeing with all of the good-speak,  but determined to use isolated, controlled experiments to learn what those things actually mean once they're applied in the organization they're part of. This essay explains why they have to do that.

out