Local Maxima and social annealing

Dan's Data has a new post/magazine article up, about local maxima. Frustratingly, there's no comment section, but I've been reading some interesting things that seem relevant, so I'll post them here. (TL;DR of Dan's post: socially we seem to get stuck in local maxima instead of finding the real best way to do things.)



I agree that societies can and often do fall into local maxima. They can also simply fail to make even the locally-best choices, or be so slow to approach the local maximum that it has moved by the time they get there. In fact, I think this latter is a better explanation for the "fighting the last war" phenomenon. The problem is, setting up a military force and finding out whether it is effective is a huge endeavour that takes a long time. The time seems even longer because you can't actually get any feedback on whether it's working unless you're actively fighting a war. (Perhaps this explains US foreign policy?) To see how fast the search for a local optimum happens you have to look at long wars: the First World War stopped being such a meat grinder as the combatants learned how to effectively organize and train their armies for trench warfare (more independence and initiative so that they could follow up a breach, for example). The US occupation of Iraq taught the US army to armour their road vehicles, but it took time.

But what about real local maxima? Strategy A really is the best — unless we do a massive changeover, when Strategy B becomes best. It could be informative to look at how companies handle this: a new paper-making process becomes available (say), and they have to decide whether it's better to keep using the old plant or shut it down and build a new plant. There are all sorts of economic factors at play, and it may well turn out that it's just not worth it to switch. The problem here isn't that we haven't found a better solution (higher maximum), it just hasn't been worth it to switch yet. Usually I think it's a question of timescale: the new paper mill might be more efficient in the long term, but the short-term cost of building a whole new factory turns out not to be worth it. Or the new plant is "better" only in ways that don't matter to the company: pollution output, say, or worker safety.

Are there really local maxima, and do societies really get stuck in them, then? I think the answer is yes there are, but that societies have a certain resistance to getting stuck. And that resistance, I believe, that flexibility and adaptibility, comes from diversity. Cultural diversity.

I'm not a social scientist, so I'll use an analogy from numerical algorithms. There too, finding a local maximum is pretty easy: just go "uphill". Finding a global maximum is much harder. But there are algorithms to do that, and even, in the case of Markov Chain Monte Carlo, to retain the correct statistical transition probabilities along the way.

Simulated annealing is based on a natural process, the annealing of solid objects to reduce internal stresses. But a quick description would be this: start one or several searches for local maxima, but at the beginning make the maximum-searching much weaker, so that they have a tendency to wander away from the local maxima and off into the unfavourable hinterlands. Gradually make the maximum-searching tendency stronger, and with a little luck at least one of the solvers will have explored far enough to find the true global maximum. This works, though it's quite computationally-intensive.

Social diversity, I think, can make societies act like this simulated annealing. I'll explain it in terms of anomalous X-ray pulsars. These are mysterious neutron stars that are pouring out more energy than they're losing by spinning down; so where does it come from? The mainstream answer, the magnetar model, says they're getting it from their magnetic field. Most pulsar folks assume this is right and, if it affects them, do their research on that basis. But the magnetar model is not the only model, and it's possible it's not the real explanation. One model would be that the extra energy is coming from leftover supernova material falling on their surface. Another would be that they are made of stable strange-quark matter, and the energy would come from, um, I'm not sure, maybe a phase transition in the quark matter? Anyway, my point is: there are still other models out there. And, thanks to the diversity of the scientific world, there are people working on each of the above models (or even one that combines two). Personally I think the magnetar model is correct and these other groups are going down a blind alley. But I also think it's important that there are people pursuing these ideas, just in case. And usable, valid results can come out of research into a wrong idea. So, if you like, we have people exploring other local maxima who will (gleefully!) report back to us if one of those is actually the global maximum.

I think that this effect — people with different ideas making sure that we collectively explore all possibilities — happens all over the place. But it needs people with genuinely different ideas, and different ideas come from different educations, different cultures, and different working styles. So I'm glad that, for example, Iran has built an air-shower cosmic-ray telescope, no matter what I think of their government.

There's a problem, though: how do you maintain diversity when all the world is watching Hollywood movies and browsing the same Web sites? I'm not sure. Language barriers help keep some cultural groups separate enough from each other for different ideas to come up, but I think the question is still open: How can cultures maintain their identities, individualities, and differences while still accepting the prosperity, connectivity, and other benefits of modern technology?