Tim Harford is a Financial Times columnist and the presenter of Radio 4’s More or Less, which won the Royal Statistical Society’s 2010 award for statistical excellence in broadcast journalism. He is also the author of several books, including The Undercover Economist. His latest is Adapt: Why Success Always Starts with Failure.

Cory Doctorow: First of all, some context — what’s the thesis of Adapt, and how does it refine, extend or improve upon The Undercover Economist?

Tim Harford: The Undercover Economist was a book about the economic principles behind everyday life, from the way Starbucks prices drinks to the rise of China. Adapt isn’t primarily an economics book at all — it’s a book about how complex problems are solved. (If ideas from economics help, great. But sometimes they don’t.)

That said, the two books start from a very similar place: describing the amazing complexity of the economy that produces the everyday objects which surround us. In Undercover it was a cappuccino, and in Adapt I describe a memorable project in which a student called Thomas Thwaites attempts to build a simple toaster from scratch. But in Adapt this complexity isn’t just a cause for a “wow, cool” moment — it’s a headache, because it’s a measure of the obstacles facing anyone who wants to solve problems in this very intricate, interconnected world.

Ultimately Adapt argues that the only way forward is experimentation, which can either be formal or ad hoc. Whether we’re talking about poverty in Nigeria or innovation in Boston, solutions tend to evolve rather than be designed in some burst of awesome genius. And then the question is — what do we need to encourage those experiments?

Cory: Is adaptability a function of scale? Are smaller firms really nimbler?
Or is a one-person shop prone to becoming a folie-a-un?

Tim: There are pros and cons to scale. Smaller firms probably do spot problems faster, because big companies have all sorts of communication problems between the boardroom and the shop floor; but at the same time, it’s hard to a lot of experimenting if you’re small. You have to try one or two things at a time and hope the failures don’t wipe you out.

The most poignant example for me is probably Johannes Gutenburg, who invented the moveable type printing press but then went bankrupt, partly because he was vulnerable after running up debt to pay for the famous 42-line Bible. His immediate imitators eventually figured out that the real money was in printing Papal indulgences, not Bibles. But it was too late for the man with the original idea.

In big organisations, one of the things which came up was the importance of front-line knowledge. I looked at some of the US army colonels posted to Iraq, who all but staged a rebellion against Donald Rumsfeld’s appalling strategy. One general pointed out to me that, of course, it’s always the guys on the front line who adapt quickly, because they have to. I also looked at the banking crisis and big industrial accidents such as Deepwater Horizon, and found that there were almost always people who could have blown the whistle — and sometimes did — but the message didn’t get through. So those communication lines need to be opened up and kept open.

Cory: What are the characteristics of an adaptable plan or organization?

Tim: Three things: you need to have lots of different experiments going on; they need to be at the right scale, so that a failure doesn’t finish you off; and you need to be able work out what is working and what isn’t, which is not always easy. (If it was easy, we wouldn’t need double-blind randomised controlled trials in medicine.)
It sounds sensible enough in principle; the book explores how that actually might work in practice in politics, banking, development aid, science funding, and so on. And of course, in our own lives.

Cory: What about in your own work?

Tim: Writing the book made me realise that it’s very easy to spout clichés about learning from your mistakes, but the mistakes still hurt. You’ll be well aware of a painful recent example, which is my treatment of Hayek and Allende in the book. As part of a discussion on centralising or decentralising knowledge, I talk about Salvador Allende’s Cybersyn project in Chile in the early 1970s, in which Allende hoped to use a supercomputer to coordinate Chile’s economy. (My father actually used to install and work on the same kind of computer Allende was using.) And I explain, citing a famous essay Hayek wrote in 1945, that economies don’t work like that because there’s a lot of vital knowledge that simply can’t be centralised.

But that juxtaposition is crass, right? Because whatever “1945 Hayek” may have thought, in abstract, about the decentralisation of knowledge, the “1978 Hayek”, at the age of 79, was writing a letter to the Times of London praising General Pinochet, the dictator, murderer and torturer who deposed the democratically-elected Allende in a coup. Hayek also visited Chile during Pinochet’s rule. This may not have strictly been relevant to an argument about the decentralisation of knowledge, but I sure as hell should have mentioned it. Some things have to be said.

So — why didn’t I say it? The simple answer is that I didn’t know. Hayek was never one of Milton Friedman’s colleagues at the University of Chicago economics department, which actually turned him down in 1950. At the time, he was in retirement in Salzburg. While writing the book, I consulted a detailed profile of Hayek written by the excellent John Cassidy in the New Yorker; Cassidy never mentions Pinochet. (Wikipedia has the details, though — of course.)
If I’d known, I would have said. And of course I’ll fix it in future editions. But right now I feel sick about it.

In a weird way the whole episode has driven home to me some of the points I make in the book. For one, I point out that it’s very easy to get together in a circle with people who reinforce your views, whereas diversity is very important for feedback. I’ve spoken to several economists, one of whom has a long-standing interest in Chile, and all of them were surprised when I told them what Hayek had done. But step outside of that world and talk to anyone who’s read, say, Greg Grandin’s book “Empire’s Workshop” and the information is right there.

And the second point is, we don’t really want honest criticism and people often don’t want to give it to us. So, Cory — you had a draft copy of this book, you read the stuff about Hayek and Allende, I guess you quite rightly thought, “Harford’s being a dick”, and then you put the manuscript down and got on with something more important. You didn’t email me to tell me to fix it, because you didn’t want to offend me and because it’s not your job anyway. By the time I emailed you to ask what you thought about it, the book itself was on sale. That was too late — so much for taking my own advice about prototyping.
Most of us don’t really ask for feedback — not real, honest feedback — because it hurts too much. And when we do ask, we often don’t get it, for much the same reason.

Cory: The Lehman regulatory example is great — and it suggests that you feel
that the problem isn’t regulation per se, but rather regulation that
doesn’t anticipate how it might be gamed. Is that fair? Do you have any
other examples?

Tim: Yes, a lot of the regulations had counterproductive effects. For example, there was a weird interaction between a collection of apparently-sensible rules about how banks had to hold more capital if they had riskier portfolios, and how portfolios were regarded as safer if they were insured with credit default swaps. All reasonable-seeming stuff, but the consequence was that when trouble struck a single big bond insurer, everyone was legally obliged to sell the same stuff, simultaneously. From a safety regulation to a global price crash in one easy step.

So we need to be smarter about the way we regulate finance in future. What really interested me — and I stumbled across this literature completely by accident — is the idea of applying principles of safety engineering to the banking system. For instance Charles Perrow, a sociologist, James Reason, a psychologist, and Trevor Kletz, an engineer, have been thinking really hard about how complex, tightly-coupled systems go wrong and how to make them safer. I think there’s some real insight there. (A few financial experts such as Andrew Lo, Rick Bookstaber and John Kay have picked up on the safety literature. But not many.)

So for example, safety experts understand that safety systems can often be counterproductive. They can encourage people to be careless — as with the case of credit default swaps, which regulators approved of as a way to load up on extra risk. The banking system didn’t get safer but it certainly got harder to understand, because the risk ended up in weird places.

Safety devices also make the whole system more complex and unpredictable. Three Mile Island was triggered by a false alarm system; the Piper Alpha disaster, a truly tragic drilling-rig fire, might have been prevented if the rig’s huge sea-water pumps had been operational, but they were disabled by default for safety reasons because they were dangerous to divers. And repackaged subprime debt, apparently safer thanks to the repackaging, was in fact simply more incomprehensible and more fragile in the face of small mathematical errors.

The bottom line is that it’s worth looking for ways to make the banking system simpler and more modular and less interconnected. And at the moment regulators are focusing more attention instead on bigger, better safety systems. I doubt that alone will fix much.

EXCERPT: From Adapt: Why Success Always Starts With Failure by Tim Harford

One Saturday evening in September 2008, while Tim Geithner and a slew of top investment bankers in New York were busily spending forty-eight hours on the wrong thing, Tony Lomas was enjoying a meal at a Chinese restaurant with his family when his phone rang. At the other end of the line was the senior lawyer for the British operations of Lehman Brothers.

The lawyer asked Lomas to come along the next day to the firm’s offices at Canary Wharf in London with a small team of insolvency experts. Lomas already knew that Lehman Brothers was in trouble. The shares had lost more than three quarters of their value in the past week. Some kind of rescue deal was being brokered in New York, but Lehman’s European directors wanted a Plan B — wisely, as Lehman Brothers fell apart shortly after the New York deal evaporated, leaving each national subsidiary to fend for itself. Plan B meant sending for the boss of the biggest insolvency practice in the UK. And that man was Tony Lomas.

The speed of Lehman’s collapse took even Lomas and his seasoned colleagues at PwC by surprise. Insolvency is typically a less sudden process — potential administrators tend to be lined up, just in case, weeks before a company declares that it is bankrupt. Yet suddenness is in the nature of a financial-services bankruptcy. Nobody wants to do business with a bank that seems like a credit risk, so there is no such thing as an investment bank that slowly slides towards bankruptcy. It happens fast, or it does not happen at all. The effect of such a sudden end to Lehman’s was chaos, most immediately for the personal lives of the accountants. One PwC partner said goodbye to his family at Sunday lunchtime and didn’t leave Canary Wharf for a week. His car ticked up an enormous bill in the short-stay car park — just one modest contribution to the cost of the administration process. PwC earned £120 million in the first year of working on the European arm of the Lehman bankruptcy, while the first year’s fees paid to administrators in the US and Europe totalled about half a billion dollars.

Lomas quickly took over the 31st floor of the Lehman offices in Canary Wharf, previously the executive dining suite; ostentatiously expensive works of art found themselves sharing wall space with hand-scrawled signs of guidance for the mushrooming team of PwC number-crunchers. The situation was an instant crisis. (I am indebted to my Financial Times colleague, Jennifer Hughes, for her vivid reporting of the situation as Lehman went under.)

On Sunday afternoon, the administrators learned that the New York office had swept up all of the cash in Lehman’s European accounts on Friday evening — standard practice every day, but on this occasion there was little chance that the money would come back. That would make it impossible, and illegal, to trade on Monday morning. And Lehman had countless unresolved transactions open with many thousands of companies. On Monday morning — after a 5 am board meeting — a judge signed over control of Lehman Europe to the PwC team, making the bankruptcy official. This happened at 7.56 am; the ink wasn’t even dry by the time the London markets opened four minutes later.

The PwC team scrambled to figure out how Lehman’s operations worked. They were shown a baffling diagram of the bank’s Byzantine but tax-avoiding legal structure, with hundreds of subsidiary legal entities, only to be told that what looked like the Gordian knot was in fact just the simplified summary.

It wasn’t that the team lacked experience: they’d overseen the restructuring of the European arm of Enron, the disgraced energy trading company famous for its financial wizardry. But Enron’s contracts were nowhere near as complex. Lomas was forced to assign staff to ‘mark’ senior Lehman officials, following them around all day in a desperate attempt to figure out what they actually did.

The scale of the chaos was mind-boggling. As a broker, Lehman Europe held over $40 billion in cash, shares and other assets on behalf of its clients. That money was frozen, so some clients found they were, as a result, at risk of bankruptcy themselves. Lehman was responsible for fully one in eight trades on the London Stock Exchange, but the last three days’ worth of trades had not been fully settled. Remarkably, this was typical. These unsettled trades were swinging in the winds of an unprecedentedly volatile market. Lehman had also hedged many of the risks it faced, using derivatives deals to protect it from volatility. But as the cancellation emails started to arrive on Monday, it became apparent that the bankruptcy made some of these deals void. When Lehman Brothers failed, it had one million derivatives contracts open.

It was only Lehman’s traders who understood how to untangle these deals, so only if some of them could be persuaded to stay on temporarily could the open positions be closed without the loss of still vaster sums of money. Infuriatingly for Lehman’s creditors — the cleaners, the cooks, the providers of telephone service and electricity — Lomas had to conjure up a $100 million loan to hand the traders generous bonuses.

Even then, they couldn’t do it alone: any trader from another firm who became aware that Lehman was on the other end of the phone trying to offload an asset would be able to exploit the knowledge that the sale was forced. So Tony Lomas recruited teams at other banks, operating under hush-hush conditions, to do the job instead. To make matters worse, as it was itself a rather large bank Lehman didn’t have its own bank account. It couldn’t open one with another bank because they were all Lehman creditors and so would be legally able to grab any money Lehman deposited. Lomas had to enlist the help of the Bank of England, opening dozens of accounts in different currencies directly with the Old Lady of Threadneedle Street.

And that was just the immediate firefighting. Tidying up the charred remains would take a long, long time. It was over a year after Lehman Brothers collapsed before a British court started to hear testimony from Lehman’s clients, the financial regulator and PwC about what might be the correct way to treat a particular multi-billion dollar pool of money that Lehman held on behalf of clients. Who should get paid, how much and when? As PwC’s lawyer explained to the court, there were no fewer than four schools of thought as to the correct legal approach. The court case took weeks. Another series of court rulings governed whether Tony Lomas was able to execute a plan to speed up the bankruptcy process by dividing Lehman creditors into three broad classes and treating them accordingly, rather than as individuals. The courts refused.

It slowly emerged that the bank had systematically hidden the extent of its financial distress using a legal accounting trick called Repo 105, which made both Lehman’s tower of debt and its pile of risky assets look smaller and thus safer than they really were. Whether Repo 105 was legitimate in this context is the subject of legal action: in December 2010, New York State prosecutors sued Lehman’s auditors, Ernst & Young, accusing them of helping Lehman in a “massive accounting fraud”. But if that case remains unproven, it is quite possible that Lehman’s financial indicators were technically accurate despite being highly misleading, like the indicator light at Three Mile Island which showed only that the valve had been told to close, and not that it actually had.

Interviewed by the Financial Times on the first anniversary of Lehman Brother’s collapse, Tony Lomas was hopeful of having resolved the big issues sometime in 2011, about three years after the bankruptcy process began.

Lomas explained what would have made a difference: ‘If we had walked in here on that Sunday, and there had been a manual there that said, ‘Contingency plan: If this company ever needs to seek protection in court, this is what will happen’ — wouldn’t that have been easier? At Enron, we had two weeks to write that plan. That wasn’t long enough, but it did give us an opportunity to hit the ground running. Here, we had no time to do that.’

Tony Lomas found an operation of bewildering complexity, and he was dealing only with the European office of Lehman Brothers — just a subsidiary of the entire bank, itself just a component of the global financial machine. But as we have seen, complexity is a problem only in tightly-coupled systems. The reason we should care about how long it took to untangle Lehman Brothers is not because bankers and bank shareholders deserve any special protection — it is that tens of billions of dollars of other companies’ money were entombed with the dead bank for all that time. If that problem could be solved, the next Lehman Brothers could be allowed to fail — safely. That means turning a tightly coupled system into one where the interconnections are looser and more flexible.

The rather quirky sport of domino toppling is perhaps the ultimate example of a tightly-coupled system. You’ve seen domino stunts as the last item on the evening news: record attempts in which someone has painstakingly stacked up thousands upon thousands of dominoes, ready to topple them all with a single gentle push. Dominoes, unlike banks, are supposed to fall over — but not too soon. One of the first domino toppling record attempts — 8000 dominoes — was ruined when a pen dropped out of the pocket of the television cameraman who had come to film the happy occasion. Other record attempts have been disrupted by moths and grasshoppers.

It might be possible to topple dominoes in a strictly controlled environment, free of insects and television crews. This would reduce the complexity of the domino system, meaning that being tightly coupled isn’t so much of a problem. But it is clearly far more practical to loosen the coupling of the system instead. Professional domino topplers now use safety gates, removed at the last moment, to ensure that when accidents happen they are contained. In 2005, a hundred volunteers had spent two months setting up 4,155,476 stones in a Dutch exhibition hall when a sparrow flew in and knocked one over. Because of safety gates, only 23,000 dominoes fell. It could have been much worse. (Though not for the hapless sparrow, which a domino enthusiast shot with an air rifle — incurring the wrath of animal rights protesters, who tried to break into the exhibition centre and finish the job the poor bird had started.)

The financial system will never eliminate its sparrows (perhaps black swans would be a more appropriate bird), so it needs the equivalent of those safety gates. If the system’s coupling could be loosened — so that one bank could run into distress without dragging down others — then the financial system could be made safer even if errors were as common as ever.

Banks can act like dominoes — toppling many other firms when they fall over — in two ways. Most obviously, they can go infectiously bankrupt, meaning that they can collapse while holding their customers’ money. The nightmare scenario is that depositors from ordinary consumers to large companies find their cheques bouncing, not because they have run out of money but because the bank has.

Then there are zombie banks. They avoid going bankrupt, but only by stumbling around in a corporate half-life, terrorising other businesses. Here’s what happens. All banks have assets (a mortgage is an asset because the homeowner owes money to the bank) and liabilities (a savings account is a liability because the bank has to give the saver her money back if she asks for it). If the assets are smaller than the liabilities, the bank is legally bankrupt. Banks have a buffer against bankruptcy, called ‘capital’. This is money that the bank holds on behalf of its shareholders, who are at the back of any queue for repayment if the bank gets into trouble.

If the assets are barely larger than the liabilities, the bank is on the brink of bankruptcy — and to avoid that fate, it is likely to resort to the undeath of zombiehood. We’d ideally want the bank to avoid bankruptcy by seeking fresh capital from shareholders, inflating the capital cushion and letting the bank continue doing business with confidence. Yet most shareholders would be unwilling to inject capital, because much of the benefit would be enjoyed by the bank’s creditors instead. Remember: the creditors get paid first, then the shareholders. If the bank is near bankruptcy, the capital injection’s biggest effect is to ensure that creditors are paid in full; shareholders benefit only if there’s money left over.

So zombie banks do something else. Instead of inflating their capital cushion, they try to shrink in size so that a smaller cushion is big enough. They call in loans and use the proceeds to pay off their own creditors, and become reluctant to lend cash to any new businesses or homebuyers. This process sucks cash out of the economy.

Both zombie banks and infectiously bankrupt banks can topple many dominoes. No wonder governments responded to the financial crisis by guaranteeing bank debts and forcibly injecting big chunks of capital into banks. This prevented the crash from having more serious effects on the economy, but it had a cost — not only the vast expenditure (and even bigger risks) that taxpayers were forced to take, but also the dangerously reassuring message to bank creditors: ‘Lend as much as you like to whomever you like, because the taxpayer will always make sure you get paid.’ Instead of a capital cushion, it was the taxpayer who was pushed into the middle of the crash to soften the impact on the financial system. Decoupling the financial system means setting up the financial equivalent of those safety gates, so that when a bank such as Lehman Brothers gets into distress in future, it can be allowed to topple.

Adapt: Why Success Always Starts with Failure