It Happens! A Leadership Guide To Getting The Most From Your Problems.

As a leader, making your organization better each day should be a priority. Part of improving is investing the time and resources to get to the root cause of problems when they happen. Companies that ignore problems or poor results by not investigating the root cause and developing an action plan for correction, will continue to struggle with mediocrity. But when do you do a root cause investigation? What level of problem should result in an investigation? Which problems should be “let go” because they are too small to be worth the effort? If you can’t find an obvious root cause, when do you stop looking and move on? Of course, the answer is, it depends. Like Forrest Gump, sometimes you have to just say “It Happens!” But a little thought should go into that decision.

Determining when and how deep to investigate a problem is one of the more difficult things you will learn to master as a leader. Make mountains out of molehills and you will destroy morale and waste resources that has little return for your efforts. Your team will begin to stop telling you about small problems because they don’t want to turn it into a crisis. You will miss some of the “triggers” (small problems that are signs of bigger culture issues). But, if you set the bar too high and don’t correct minor issues when they are minor, you potentially turn into a big problem that costs you time, money and could even cause injury or death.

As always, I have a couple of sea stories from my days in submarines and a couple of things to consider when faced with this type of decision. So, let’s get to it.

In 1999, I was a fairly young and inexperienced chief engineer on USS CHICAGO (SSN 721) in Pearl Harbor. I had just taken over as engineer in charge of the 55 or so officers and sailors responsible for the nuclear propulsion plant. The Pearl Harbor Naval Shipyard was conducting an overhaul and the submarine was up on blocks in a dry dock – not where any submarine sailors wants to be. To add to the pain, we were behind by about 8 months on our overhaul schedule. Like any project that is behind, we were under great pressure to get back on schedule or at least not lose any more time.

One day we were conducting a valve line up on one of the nuclear plant systems. This is a very rigorous two person process where we ensure the valves in a mechanical piping system are in the right position before we start some operation on the overall nuclear plant. Obviously it is important, especially on a nuclear reactor, to have the valves in the right position. You don’t want to accidentally spill potentially radioactive water into the environment, into the ship, or on people. So we are doing this valve line up and a valve is found “out of position.” The valve was open and it was supposed to be shut. Nothing was leaking out because we hadn’t started the operation and there are usually two valves that must be out of position to cause a spill.

So what do you think the response should be? A valve was found not in the position we expected, but no one was hurt and nothing was spilled. No harm, no foul, right? Just shut the valve. Get on with life. Not so fast. The risk of a major incident on a nuclear power plant due to a valve out of position is too high to just ignore. Did we have problems in other systems? Did we have procedural compliance problems? Did we have administrative tracking problems? (the position of every valve in the nuclear propulsion plant was tracked on a status board) This seemingly small problem that did not result in any harm could lead us to discover a larger systematic problem that could cause a major incident.

Once this out of position valve was reported up to supervisors, we immediately stopped all work on the nuclear propulsion plant. A timeline and facts were gathered immediately. We conducted a critique to identify the root causes and corrective actions. (For a more detailed explanation of a “critique” in the submarine force, see my previous post here: You want the Truth? or see my recently released book titled Extreme Operational Excellence for a really in depth discussion – shameless plug completed.)

This investigation effort cost a lot of time and money. Some business managers may be asking, “What was your return on investment (ROI) for all that time and effort?” I don’t know a direct and specific answer to that question because it is impossible to measure. If we prevented something major like a spill of radioactive water or injury or death to some operator, then the ROI was enormous reaching infinity in the case of preventing a death. But, I can’t say for sure what we prevented or if something would have happened if we hadn’t done this critique. We found some issues with our training and formality but there wasn’t a very clear “smoking gun” found during the critique. But the critique was the right thing to do especially because it reinforced our Navy Nuclear culture of excellence and integrity. That alone was worth the effort.

Fast forward to 2004, I was the Executive Officer on board USS ALBUQUERQUE (SSN 706). We were conducting some training exercises in the Atlantic. We had just finished a Battle Stations training scenario where we simulate enemy submarines in our systems and go to battle against them. The training is pretty good because the signals are injected into our system and look real to everyone operating the equipment. As XO, my job during Battle Stations is Fire Control Coordinator. It’s a pretty intense role taking in all the sonar information on the enemy submarine and working with the Fire Control team to figure out the enemy’s range (distance from our sub) and bearing (direction from us) so we could load that into the torpedo and launch it. All this time, the Captain is pushing me for a “firing solution” so he can order the torpedo launch before the enemy submarine shoots us. Of course, this was just an exercise, but we train pretty hard to make sure the simulation is as close to the real thing as possible. We fight like we train in combat, so we have to train hard.

So when this training was completed and we “secured from Battle Stations” I went to my stateroom to take care of some “business.” To explain this “business” I must explain that on a Los Angeles class submarine there is a restroom (called a “Head” in Navy jargon) with a toilet and shower between the Captain’s and XO’s stateroom. It’s all stainless steel for ease of cleaning and preservation which will come in handy on this day. There are several of these Heads around the ship that drain to the same sewage tank. A Los Angeles Class submarine has a large sewage pump to get rid of all the stuff done during “business” in the Head. The stuff from all 140 people basically sits in a large tank until it is pumped overboard. There is also an air pressure piping system connected to the tank in case the pump is broken. This way the submarine can blow the stuff overboard. The pressure is high enough that this stuff can be blown overboard at pretty deep depths.

To blow our stuff overboard, we follow a procedure to hang some signs letting people know not to use the toilets because the tank will be pressurized. We do a valve line up similar to the one I talked about above but a little less intense than the line up for a nuclear reactor. Then when all the conditions are set, we open a valve on the hull and out goes our stuff. For any environmentalists concerned, we only do this out at sea beyond several miles and it’s only bio-degradable waste. Whales and fish poop a lot. We just add a little to that.

Well, there were no signs hanging and we had just finished a major training exercise so I was pretty confident I could use the Head and do my business without any risk. After a little “reading” time, I stood up to open the valve and flush my stuff down to the tank. As you might have guessed by now, the second I opened the valve to let my stuff flush down…crap! In my face, in my hair, up my nose, and in my ears. I was saturated by a foggy mist of 140 shipmates “stuff.” I don’t know how much pressure was behind it, but the blast of stuff came at me like a poop tornado. I was covered head to foot. The Captain heard the noise and opened the outer door asking if I was okay. I cried out “look away, I am hideous!” Just kidding. I told him “Captain, you better keep the door shut. We have a lot to clean up. Please send the Officer and Chief in charge of Auxiliary Division.” I may have even said a few choice words I won’t repeat here. I think the Captain had slight smile on his face as he closed the door…he was probably thinking “better him than me.” And I would agreed. If the Captain is covered in crap, it’s a bad day for everyone. So, it might as well be me.

Now, compare this issue with the last one regarding the nuclear power plant valve out of position. This time a valve out of position clearly caused a spill of hazardous material (at least I think it was hazardous) and some injury (although mostly to my pride.) It also caused a noise which might have been detected by an enemy submarine. That’s no minor issue. Transient noises like that can be life or death in a submarine battle.

So, do we hold a critique? Do we conduct a root cause investigation? This time, I decided not to pursue it. There were many reasons but mostly I didn’t want the crew to think it was some sort of revenge witch hunt I was on. This was a good crew and they were working hard. They had just performed well on a surge deployment. Morale was on the upswing and so was our performance. I decided to let it go for the greater good of morale and the Captain agreed with me. Sometimes… it happens.

So what lessons can we learn and apply? When should you do a critique and root cause analysis? Here are six things to consider when you are considering how to address a problem in your organization:

1. Identify Critical Systems or Processes: What is the risk in the system or operations that are involved? This seems obvious but it’s harder than it looks. If you are operating a complex system that involves high energy equipment like high voltage, high pressure gases or liquids, or hazardous chemicals then any incident that risks the breach of integrity of your systems must be investigated and documented. But what about less clear examples in industries where there is not a direct operation of equipment? I recently spent several years working for design engineering firms in nuclear and non-nuclear power plant design. Although these design engineers were far removed from the actual operations of the plant, certain elements of their design process were obviously critical to preventing re-work in the field at a minimum or worst case an operational incident. The key is to identify in writing through Quality Control or other programs what are the critical systems or processes that require an incident investigation. In the Nuclear Navy, that list is maintained at a high level by Naval Reactors (the Technical Authority originally established by Admiral Rickover.) As Commanding Officer, I didn’t have a choice when we had a problem on “the list.” That was a good thing because quite honestly sometimes in the heat of the battle, my tendency was usually to look for a way out of doing an investigation. (Please don’t leave me all alone here…) At times, I needed that policy to force me to investigate. It also helped us be consistent across the fleet on what constituted an incident. Does your corporate office have similar policies?

2. Culture Change: Are you trying to change the culture of your organization? For example, if you are trying to get your team to be more formal about following procedures, then any incident that involves procedures is a golden opportunity to impact the culture. Like the saying goes: “Never let a crisis go to waste.” For an example, study the Texas City Refinery explosion of 2005. Texas City had had several other more minor incidents previously that if they had investigated and corrected the root cause, the 2005 incident, which resulted in the deaths of 15 workers, may have been avoided. (For the best succinct review of this incident see Failure to Learn by Andrew Hopkins.) Take advantage of opportunities to guide your team when the cost is low.

3. Team Performance: Another consideration should be your team’s level of performance. High performing teams do not need as much guidance. In the case of my first example above, our submarine had been in the shipyard for over a year and half. The crew’s level of training and experience in operating the propulsion plant had degraded. It’s a natural, although undesired, side-effect of the way our submarines are worked on in the shipyards. Because of this level of their performance, we needed to dig in deep on this incident to make sure we learned from it and exposed any other shortcomings. If this same event had happened two years later when we were a highly experienced crew, we would have still conducted an investigation but not nearly to the depth of this one.

4. Culture Pendulum: Another important factor is the direction or momentum your team is trending toward. What I mean is, are you performing well and trending in a good direction related to the number of recent incidents or are you trending in the wrong direction? We talk about this a lot in our book in Chapter Nine on Pendulum Leadership. Recognizing that your team’s culture is headed in the wrong direction is good if you take action to stop the direction. Sometimes going “a little overboard” on an incident to get everyone’s attention is warranted and useful. The key is knowing when your culture is degrading. Many of the signs are lagging indicators such as near misses and minor incidents or quality control problems. Measuring the culture gap and actively working on it will help you stay ahead of these problems. The critique is one tool to help you put your team back on track.

5. Leadership Tenure: How long have you been in your role? Whether you are the General Manager, Plant Manager, Project Manager, Front Line Supervisor, Quality Manager, or Operational Excellence Director, you should establish early in your tenure your expectations for when you are informed of problems, when they need to be critiqued and root causes identified, and how that process is going to work. The longer you are in your role, the harder it is to effect change. I was told when I was headed to command of USS KEY WEST (SSN 722) that I had six months to effect real change. After that, I was just part of the problem. Take the opportunity early in your tenure to establish your desired policy or processes. Before too long, it will become “just the way we do it here.”

6. Share the Lessons: If you think the problem you just encountered may be something other business units in your company could benefit from, then share it. Do a thorough investigation and root cause analysis and send it out so others can learn from it. I know this type of airing of dirty laundry is somewhat rare in the business world, but it can really benefit your company to take this step. In the Nuclear Navy we embraced learning from others’ mistakes and sharing our mistakes with others. We all got better for it. I have referenced this before, but check out Chapter 5 of The High Velocity Edge by Dr. Steven Spear for his study of how this one attribute of the Nuclear Navy accelerates its learning.

There may be other things to consider and a lot more could be said about how to scale your investigation effort to match the size and risk of the problem. We can make these things as complex as we want. But, perhaps this is enough for one article. If you are still reading at this point, I am impressed! Thank you for your stamina.

Well, that’s all for now. I need a bathroom break. My house is in the country and we have a septic tank system, but I think I am pretty safe today from high pressure air building up in that tank. For any of my ex-shipmates, don’t get any funny ideas…this time I will investigate the root cause and take corrective action!

Bob Koonce served for over 20 years in the U.S. Submarine Force and retired from active duty in 2011 after commanding USS KEY WEST (SSN 722), a nuclear submarine based in Pearl Harbor, Hawaii. Bob frequently speaks and writes on Operational Excellence and High Reliability Organizations based on the leadership and culture of the U.S. Nuclear Navy. He is co-author of Extreme Operational Excellence: Applying the Culture of the Nuclear Submarine Force to your Organization available now on Amazon. You can learn more by visiting www.highrelgroup.com.