I remember watching in fascinated horror in October 1987 as the stock market crashed.

 On Monday October 19 the Dow Jones Industrial Average dropped by 22.6%. $500 billion evaporated.

As a young (well, relatively) business editor I got pressed into service calling up the smartest people we knew on Wall Street (at least they’d been smart the week before) to ask them what had happened. On a day when Procter & Gamble (PG), one of the bluest and most conservatively managed of blue chips, fell $2 then $3, then $4 and finally $5 a share before it caught a bid, you can imagine what those conversations were like. Wall Street money managers were in shock. “This can’t be happening,” more than one told me. “Prices don’t behave like this.”

“This can’t be happening. Prices don’t behave like this.” That phrase connects the financial disasters of the last ten years.

And, I can promise you now, it will be the theme song for the next financial market disaster too. Because efficient market theory, the set of assumptions that connects the invention and the collapse of portfolio insurance in 1987 with the collapse of mortgage-backed derivatives in 2007, just won’t die. It may be as much intellectual Swiss cheese as Ptolemy’s great system of astronomy that had the sun revolving around the earth, but it’s far too profitable for Wall Street to let it die.

“Those who cannot learn from history are doomed to repeat it,” George Santayana wrote. Fortunately, we don’t have to. Justin Fox has just published an extraordinarily interesting and readable history of efficient market theory titled “The Myth of the Rational Market: A History of Risk, Reward, and Delusion on Wall Street.”

Read it and you’ll understand how we got here. Why Wall Street will keep recreating these disasters. And how you can tip-toe around the worst of the damage.

What Fox is best at, in my opinion, is showing the reader the assumptions behind efficient market theory from its development in the 1960s (to pick one possible birth decade) to its triumphant takeover of business schools, Wall Street, and the corporate board room in the 1980s. (I had the key formulas of efficient market theory, models like CAPM, drilled into me in the year I spent business school in the early part of that decade.)

What are some of those key assumptions? The human beings are rational agents who are out to maximize their self-advantage. The agents rationally decide what their self-advantage is. That information flow in financial markets is free and instantaneous. That prices always accurately reflect all the available information. That markets always clear to equilibrium because enough buyers and sellers will always emerge.

Fox is to my mind extraordinarily fair to the great names of economics and finance who put this structure in place: Franco Modigliani, Eugene Fama, Merton Miller, Fischer Black, Harry Markowitz, Milton Friedman, Myron Scholes, and others. They never come across as anything else than what they are—brilliant thinkers who knew that they were making radically simplified assumptions about reality so that they’re models would work.

Fox’s most interesting chapters, to me anyway, are his discussions of such amazingly honest thinkers as Fisher Black who never forgot that their theories were built on simplified versions of reality. I can’t imagine being tough enough to constantly question the validity of your life’s work, but some of these folks did exactly that.

Events, of course, helped them along. Because reality struck back hard not too long after efficient market theory became the ruling orthodoxy. The 1987 stock market crash, facilitated if not created by a financial product called portfolio insurance built out of the pricing models created by efficient market theory, was only the first.

Then there was the collapse of the hedge fund Long-Term Capital collapsed in 1998. Some of the best minds on Wall Street, including board members Myron Scholes and Robert Merton who shared the 1997 Nobel Memorial Prize in Economic Sciences, had devised immensely profitable strategies that exploited tiny unjustified differences in the prices of financial assets such as 30-year Treasury bonds and Treasury bonds with 29.75 years until maturity. In the years after its founding in 1993 Long-Term Capital averaged returns of 40% annually.

And then came 1998. By that time Long-Term Capital had borrowed billions–$124.5 billion to be precise—to get more bang from the tiny differences in price its computers identified. In four months that year, though, Long-Term Capital lost $4.6 billion. Prices for its portfolio assets collapsed—shouldn’t happen in an efficient market—and liquidity dried up—not part of the theory. The New York Fed finally had to engineer an orderly winding up of the fund in order to prevent its problems from rippling out through the global financial system.

There are more recent examples too. Federal Reserve chairman Alan Greenspan’s decision not to prick the technology bubble in 1998 or 1999 that then led to completely irrationally high stock prices, which then collapsed to irrational lows. Where was the efficiency to a market that valued Cisco Systems (CSCO) at $80.06 in March 27, 2000 and at $8.60 on October 8, 2002?

Surely one, or perhaps both, of these prices was better explained by comparing investors to a herd of sheep jumping into ocean than to the rational decision maker of efficient market theory.

And now in the current crisis investors have received a painful refresher course in how panic—neither efficient nor rational—can so dry up market liquidity that there are no prices, efficient or otherwise, for some assets at all. It’s hard to get to price equilibrium when no one is bidding.

Fox’s book also makes it depressingly clear why, despite its failures and its role in global financial disasters efficient market theory isn’t about to go away. The theory provides a set of mathematical formulas that let people on Wall Street calculate the “true” price and quantify the risk for things like options and their increasingly complex descendents in the derivatives world. And the bottom line on Wall Street is that if you can price a product and give it a risk rating, you can sell it.

After watching the flood of profits created by new products that priced the risk of mortgage-backed assets, do you doubt that for a moment?

And as we know so clearly, the rewards for creating profitable instruments based on this flawed theory far out way the punishment for being wrong. Sure, a Lehman Bros. goes under, but life and bonuses go on at Goldman Sachs (GS) and even at Merrill Lynch.

Fox wasn’t out to write a what do you do about it book for investors. But I think a very valuable how-to book logically falls out of his history.

Efficient market theory isn’t worthless, Fox points out. Its assumptions are reasonable and its conclusions accurate much of the time. Say 80% of the time.

Call these normal times. During these times you ought to follow the strategies of the mythical efficient investor by looking for the kind of information arbitrage profits that efficient market theory predicts. So you can make money during these times by studying global stocks and domestic small caps that few investors follow; by taking a long-term view when everyone else is focused on the short-term, by digging out technology trends or balance sheet facts that no one else knows yet. Efficient market theory says that you’ll get paid for putting that information in play for the larger market so it can bid inefficiencies out of existence.

And then, there’s the other 20% of the time when investors aren’t rational and the herd rushes about as if from one extreme to the other. At times like these you can’t trust the wisdom of the market, because the herd has lost its bearings. You protect your portfolio from losses by going against the flow, by ignoring, selectively, the common wisdom, and by having the courage to follow your own lonely way.

An example of this? Fox includes a great one. Efficient market theory has spent decades trying to explain away data that shows value investing strategies outperform growth strategies in the stock market. Turns out, researchers in the new field of behavioral finance have shown to my satisfaction that the out performance is a result of how hard it is to go against the herd. Value investors have to stand up to the ridicule of the consensus and that makes them work just a bit harder to justify the courage of their convictions.

So that’s the strategy that I come away with after reading Fox is actually very simple. Follow the efficient investing strategies during normal times and invest like a contrarian during the 20% of times that aren’t normal

Now if someone will just tell me an infallible way to tell the 80% from the 20%.