Crude Oil ( ) Brent Crude ( ) Natural Gas ( ) S&P 500 ( ) PHLX Oil ( )

Story by Bloomberg

The world’s biggest oil companies faced ruin in the summer of 1931. Crude prices had plummeted. Wildcatters were selling oil from the bonanza East Texas field for a nickel a barrel, cheaper than a bowl of chili. On Aug. 17, Governor Ross Sterling declared a state of insurrection in four counties and sent 1,100 National Guard troops to shut down the fields and bring order to the market. A month later the Railroad Commission of Texas handed out strict production quotas.

That heavy-handed intervention in the free market was remarkable enough. Even more remarkable was who pulled it off. The person in charge of shutting down the wildcatters, National Guard Brigadier General Jacob Wolters, was the general counsel of Texas Co., an ancestor of Chevron (CVX). And the Texas governor who ordered Wolters in was a past president of Humble Oil and Refining, a forerunner of ExxonMobil (XOM). Big Oil played hardball in those days.

History is repeating itself, with a twist. The stressed-out giants of today are Saudi Arabia and its fellows in the Organization of the Petroleum Exporting Countries. The descendants of the 1930s wildcatters are today’s producers of oil from shale, who are driving down the world price of crude by flooding the market with millions of barrels of new oil each day. At $64 a barrel, Brent crude is down 44 percent since June. The twist is that today’s upstarts aren’t draining oil from neighbors’ plots, as happened in the 1930s. And OPEC can’t call in the National Guard against them. All it can do is gape at the falling price of crude and contemplate the destruction of their cartel at the hands of the Americans, whom they thought they had supplanted for good 40 years ago. Energy economist Philip Verleger says shale is to OPEC what the Apple II (AAPL) was to the IBM (IBM) mainframe.

Theories as to why OPEC didn’t reduce quotas at its meeting in Vienna on Nov. 27 are as cheap and abundant as crude in North Dakota. One holds that the Sunnis of Saudi Arabia want to hurt the Shiites of Iran, who need high-priced oil to finance their government. Another, expressed by Russian President Vladimir Putin, is that the whole thing is a conspiracy to undermine Russia, the world’s biggest oil producer. Yet another is that the Saudis hope to drive oil prices below where it makes sense for American shale producers to invest in new production. But shale producers have lowered their costs so much that in key fields they can make profits at $50 to $70 a barrel. That’s above core OPEC members’ exploration and production costs but below what many need to cover their government spending. “If my calculations are correct, this will go down as one of the worst commodity trading decisions ever,” Wilbur Ross, billionaire investor and chairman of WL Ross (IVZ), wrote in an e-mail.

In fact, prices are being forced down not by any action (or inaction) of the Saudis but by the American shale producers, who are simply producing all the oil they can to maximize their profits. “Collectively, they’re not the most sophisticated folks, especially when it comes to world markets,” says Charles Ebinger, a senior fellow in the Energy Security Initiative at the Brookings Institution.

With apologies to Ebinger, the shale producers don’t need to be sophisticates. Each operator is so small, it can increase production without pushing down the market price. That makes them price “takers,” not price setters. And because shale wells are short-lived, producers don’t have to plan far ahead, says Karr Ingham, a petroleum economist in Amarillo, Texas. Singly the shale busters are nothing. Collectively, their breakneck production is breaking OPEC’s neck. This is the remorseless, leaderless free market at work.

OPEC used to be something to reckon with. For a brief period in the 1970s its influence was so strong, it could set prices to the penny for scores of crudes, says Bhushan Bahree, senior director for OPEC Middle East research at market researcher IHS (IHS). Its power has waned considerably, but until this year Saudi Arabia could still be counted on to cut output for the good of the cartel when gluts emerged. The Saudis’ refusal last month to take one for the team is historic, says Michael Wittner, head of oil research at Société Générale (GLE:FP) in New York. “That is such a tremendous, dramatic change,” he says. “It’s hard to think of a way to exaggerate how fundamental it is.”

And rare. For nearly all of oil’s history it’s been under some kind of price control. John D. Rockefeller created the Standard Oil trust in the 19th century to dominate the market. Later came the “prorationing” rules of the Railroad Commission of Texas, which set the world price for decades. The Western oil companies known as the Seven Sisters carved up world oil supplies among themselves. Then came OPEC, which was at first confrontational toward oil-consuming nations but soon achieved a modus vivendi with the International Energy Agency, the group representing them. “They do good lunches. Vienna is a very good city to visit,” says Christopher Segar, an IEA energy analyst who’s been responsible for the agency’s relations with OPEC.

Given oil’s chronic volatility, it would be a mistake to count OPEC out for good. The glut that’s undermining the cartel today will set the stage for future shortages that could restore its influence. Cheap oil from shale already shows signs of shaking out investment in costlier technologies. Rystad Energy, a Norwegian consulting firm, estimates that oil averaging $60 a barrel next year would lead to the delay or cancellation of one-third of all oil and gas projects slated for go-aheads in 2015, mainly higher-cost investments in the Alberta oil sands, the Arctic, Brazil, West Africa, and the North Sea. As night follows day, scuppering projects will lead to shortages and a price spike. Untamed oil is “in a continuous boom-bust cycle,” says Michael Webber, deputy director of the Energy Institute at the University of Texas at Austin.

For now the greater risk for consumers is a move to prop up prices and save marginal drillers. Mike Cantrell, a veteran Oklahoma oilman, recalls lobbying Washington in the 1980s for a tax on imported oil. “When you’re looking at going out of business, there are a lot of things you’ll try. I am not proud of those things, but I did them to try and survive.”

There are better ways than tariffs or cartels to tame oil. One is a crude reserve that an oil-consuming country can tap if supplies are disrupted, says Jason Bordoff, who was President Obama’s energy adviser from 2009 through 2013 and now directs Columbia University’s Center on Global Energy Policy. China is already adding to its strategic reserve at today’s low prices, putting a floor under the market—“wisely,” says John Studzinski, a senior managing director at Blackstone Group (BX). The other calming force is the futures market, which guides investment by reflecting investors’ collective judgments about the trajectory of prices. Alas, there is less trading in contracts out past two years, when the guidance would be most useful. Meanwhile, cheap oil isn’t all good. To fight global warming, the world needs to tax carbon fuels to promote the transition to renewable energy.

The U.S. is in a good place at the moment. In international affairs it often plays the galumphing giant bedeviled by small, nimble foes. With shale, it’s the American upstarts who are small and nimble. Says the IEA in its latest monthly report: “It is increasingly clear that we have begun a new chapter in the history of the oil markets.”