Two hundred years of experience stood around a frozen heat exchanger. None of it was on the subject that mattered.
Longford, Victoria, Australia - The morning of 25 September 1998: A pump tripped in Gas Plant 1. Without the warm oil it circulated, the process ran far colder than anyone had planned for. One heat exchanger dropped to about minus 48 degrees and ice formed on the outside of it.
The operators and supervisors gathered round. Between them they had more than two hundred years at Longford. They could see the ice. They could see a leak. Everyone there had spotted the problem and they reached for the fix their training pointed to. It was the best anyone could have done with the resources they had using what they had been taught. Nobody in the group knew what that kind of cold does to steel. That was typically the engineering team's domain - but they weren't there.
So the team on site did the sensible thing. They put the warm oil back in.
The steel shattered. About ten tonnes of hydrocarbon escaped and ignited. Two men died, eight were injured and the state of Victoria lost its gas supply for nearly three weeks.
Where we are in this series
Last time I proposed a picture: three elements, three overlaps and one apex at the centre. (read it here) This article is about the first element, Total Awareness: every direction, every perspective.
Every direction, every perspective
Total Awareness means seeing the whole system you are about to change, not just the part in front of you.
I find it helps to look in four directions: the plant - every layer of it; the process - how a decision actually gets made and recorded; the history - what has happened here before, including the parts nobody wants to hear; and then the people - who will be there when the plant does something unusual? Will they know the right thing to do… in time?
It's so easy to miss any one of these, or a detail within one. In this case the fourth was missed, possibly because people don't appear on a drawing(?)
Total Awareness is not knowing everything. It's being able to show you looked in every direction, taking advice where appropriate to a depth that matches the risk. Longford is what happens when three directions are fine and the fourth is empty.
Where the engineers went
Go back six years.
Until 1991 the plant's engineers were based at Sale, a few kilometres up the road. They came to the plant every day. They walked the control room, read the charts and talked to the operators. The Royal Commission later called that "an ideal position to monitor the plant operating conditions and operator practices".
In 1992 Esso restructured. Every plant engineer was moved to head office in Melbourne, about 215 kilometres away. Esso did not simply cut the link. It installed a data link so the engineers could see live process data from Melbourne. It kept a company aeroplane. From the office, it looked as if nothing had been lost.
But looks can be deceiving. The engineers in Melbourne were put to work on projects, so the daily watching of the plant fell to operators and supervisors, whose attention was on today's production. The Commission found the live data "was rarely, if ever, looked at". Four weeks before the explosion, a similar kind of upset had happened in the same plant. Nobody investigated it. The people whose job had been to spot a recurring problem were two hundred kilometres away.
Esso defended the move. Its defence deserves a hearing. Its operations manager said plenty of gas plants ran safely without engineers next door. The plant manager on site was himself an engineer. In general, he was not wrong.
The Commission answered in a few sentences. The relocation "qualified as a permanent change to operating practices requiring risk assessment and evaluation before implementation". None was ever done. On the day, "there were no experienced engineers on site". The reduction in supervision, including the move, was "probably a contributing factor". The ultimate cause, in its words, was "the failure of Esso to equip its employees with appropriate knowledge to deal with the events which occurred".
Look at that again. It wasn't the pump and it wasn't the operators. It was a change to where the people sat, treated as an office move.
What if it wasn't 215 kilometres?
Now flip the story.
Make it 2,000 kilometres. In July 2010 a pipeline ruptured near Marshall, Michigan. The control room was in Edmonton, Alberta, roughly 2,000 kilometres away. Both operators on shift had decades of experience and were working through the alarms the way their procedures and their software told them to. They watched those alarms for seventeen hours and restarted the pumps twice. Even so, about 840,000 US gallons - over three million litres - of crude went into a river, uncontained. The investigators blamed training and procedures, not distance…(?)
Make it a contractor. In 2006 an RAF Nimrod caught fire over Afghanistan and fourteen people died. Those responsible for its safety case were carrying cuts, a reorganisation and two wars at the same time. The Ministry of Defence had contracted the aircraft's safety case out to industry and relied on a decision from those without full context. The review that followed called the result "a lamentable job from start to finish". Its sharpest line was aimed at the in-house team that should have checked the work - it had "outsourced its thinking". My hunch - maybe this was to simply try and ease the pressure they were already under from existing MOD cuts in the same department? Is it even possible to track the cause and effect of this complicated sequence of events? I couldn't find any data to link the two directly, but how humans respond to imposed change is potentially an entirely new article series for another day!
What if we make it another company in another country? Over a bank holiday weekend in May 2017, British Airways' IT systems failed and about 75,000 passengers were stranded. The airline blamed a power supply fault and an engineer. The GMB union said the data-centre staff had been among hundreds of UK IT jobs cut the year before, with the work handed to an outsourcing company overseas. BA denied it. No investigation was ever published, so I can't tell you who was right - which, frankly, is rather the point.
Somebody did ask the question
The question nobody asked at Longford was asked a year later, in Britain. British Energy had been privatised in 1996 and was cutting staff under a programme called Vision 2000. The nuclear inspectorate of the Health and Safety Executive went in and audited it. It found the staff had gone but the work had not. It told the company to stop cutting until it could show the change would not harm safety. Then it wrote a new condition into every nuclear site licence in the country. Any change to an organisation or its resources that might affect safety must be controlled, like a change to the plant. The guidance under it names "relocation of offices" as something that "may result in the loss of key staff and expertise". Nothing blew up, so there is nothing more to tell.
I can't help feeling at this point that the lesson seems like a bit of a damp squib(!)
It's already in the rulebook
None of this is new thinking. The International Association of Oil and Gas Producers (IOGP) publishes an operating management framework. It says changes "that affect the organisation" must go through a formal, risk-based management of change process. The same as a change to the kit. Try moving a pump without the paperwork and see how far you get. Trevor Kletz, a British chemical engineer pivotal to establishing the discipline of process safety, put it plainly in 2003: "As with changes to plants and processes, changes to organisation should be subjected to control by a system." His example was Longford. He also left a line worth keeping: "Empowerment can become a euphemism for withdrawal of support."
Over to you
Suppose an organisation is confident enough to take hundreds of years of experience out of one office, in one city, in one country. It moves the support somewhere else. Ten kilometres away, or 200, or 2,000. It is placing a bet that the people at the other end can effectively and safely manage a very bad day on their own with just the training and support they've been given.
Sometimes the bet pays. Esso's manager was right that plenty of plants run that way. My hunch is that the difference lies in two things. Did anyone assess the move as if it were a change to the plant? How long did they give it? My hunch is that it wasn't a weekend.
Next time: Integrated Signal Intelligence, the whole chain from the sensor on the plant to the person deciding - and a radar that saw them coming. Longford had the sensors. Nobody was at the other end of the chain.
So until next time, I'd like your perspective on two things.
How confident are you that the people at the far end of your last reorganisation could safely and effectively handle a bad day on their own?
Think of the last transition you were part of - how long did that transition actually get and was it considered from all perspectives?
In the spirit of continuous improvement, I'm always happy to be challenged or proven wrong if I've missed anything. That's the only way to target our operational apex right?
This article first appeared on LinkedIn, where the conversation carries on: join it there.