The MSH Team to the Rescue

They finally hauled the UNIX teams into a conference room. Patience had run out.

We were all there:
  • The bald guy who claimed he had hair when he started this job
  • The guy who took 3 years off when the Internet Bubble burst to get a law degree (“I don’t know, it sounded interesting at the time”)
  • The one who got a math degree because it was the easiest subject to do while watching TV
  • The guy from Hong Kong who explained to me that “The reason a lot of people around here can’t drive is because they are from Asia.”
  • The one who read in IBM’s documentation that RTFM stood for Read The Friendly Manual.
  • The guy who giggles a lot... and has been playing a cat-and-mouse game with HR since before everyone else joined the company...

The MSH team was put in charge. We knew things were serious because the CIO calls on the MSH team when he needs someone to make things happen.

Anyone associated with UNIX Systems Administrators on a personal basis sometimes wonder how we stay employed. The concern originates from the mistaken idea that we don’t believe in following instructions. Of course we believe in following instructions… we wrote most of them. But typically we tend to follow them only as a last resort.

The real reason we are indispensable is that people think that we know where things are held together with duct tape and where they are only held together with spit. Or at least so they hope. In any event, when something breaks, we’re needed on the conference call. Even when we tell them we don’t know what’s going on. Especially when we say we don’t know what’s going on.

We, on the other hand, believe that the real problem is that the guys who draw up the time lines think that there’s no difference between “in theory” and “in practice.”

We all suspected this day was coming.  Under terms of our acquisition, the CIO needed for us to start using some “knowledge transfer software” before he could take his money and leave. But we had a lot going on.

Not to say they did not give us adequate warning. And at one point it even seemed like we had enough time. But then, disaster: the junior guy who usually dealt with the crap our new parent company was trying to shove down our throats had left for more money. This inspired a senior guy, called Eric, to post his resume and a few days later he quit and now we had about 5 open positions we could not fill.

We all know what happens when someone has trouble keeping up: an annoying task comes along that happens to be high priority. In this case, we had to start converting our servers to an operating system that has the word “Unbreakable” in the name. The guy who had given me the most challenging technical interview of my life was told to find a few things that broke the “Unbreakable.” This task was particularly challenging because he was asked to report something that sounded particularly frightening. It took him about an hour. He went right for the jugular and found an incompatibility between one of the storage formats we use and this OS. Annoying project successfully shot down. Then he spent another hour finding some lesser evils^H^H^H^H^H bugs. Management mischief managed.

It had also been an eventful day for the guys who manage the mail servers. The really colorful guy on the team was spouting a monologue. This is a regular occurrence because he can talk as an expert on any subject for about 3 minutes (but is genuinely too ADHD to hold most types of jobs). He said “You know, this is all Eric’s fault. We did warn him.” For a moment I thought this was an underhanded bit of blame-storming against the guy who had just left. Then the lawyer reminisced about how at Berkeley they had warned Eric about spam but unfortunately for the human race Eric was convinced that bad people would never use email. Turned out they were talking about Eric Allman.

By now, my long-time readers are anticipating my digressions. Here is one, with a tangent.

People have been trying to replace Eric Allman’s software since before the word “internet” was coined. In the end, despite its complexity, few organizations bothered replacing it. You cannot send an email without it being handled and forwarded by his software somewhere along the way. And by the way, you pretty much cannot use anything else on the internet without it touching software written by his husband. I found this quote from Allman on Wikipedia : “There is some sort of perverse pleasure in knowing that it's basically impossible to send a piece of hate mail through the Internet without its being touched by a gay program.” So all you homophobes be sure to wash your hands with something that kills the AIDS virus every time you read email.

We all gathered in the conference room with our laptops. Despite some early success that day, we had plenty of fallout to keep under control. We all multitasked.

The representative of the MSH team explained that it was a BFD (Big Friendly Deal) that we hadn’t started using this admittedly poorly designed tool. But, we needed it for sharing information. Thanks to our nemesis (poetic justice) we were supposed to have started this on April 1. We were 6 weeks behind. Truth be told, we were hopelessly behind on March 31.

To our annoyance, we couldn’t just edit some text-based configuration files. We had to click this and then click that.

For a moment, the MSH team representative thought she had some people in the room who knew what was going on: our managers knew the buzzwords. But to their surprise it turned out that, at least in this case, click this and click that was harder than it looked. They were as lost as the rest of us.

The representative of the MSH team walked us through how to sign up for things. Then the cat-and-mouse-with-HR game flared up. The rest of us tried to ignore the giggling and suggestions for passwords that MIGHT be interpreted as off-color. Someone’s wife called. Flat tire. We paused our conversations while Triple A info was passed over the phone. When the call ended, we heard giggling and “I prefer Triple X.”

The representative of the MSH team got us back on track. She told us where to go, what to click, what to install. She even came and looked over everyone’s shoulders after each step to make sure we were doing what we were supposed to be doing. Just like I used to do when I taught beginner computer classes.

Then things came together. Our team has a mathematician. And a woman. In the same person. Who works from home full time so she could help her daughter conquer her math phobia. Certainly the teacher wasn’t able to. (Soon after this meeting, the girl crushed the AP Calculus test). Our mathematician had come into the office that day for this meeting.

With feminine ability to listen, a mathematician's attention to detail and a mom's well-practiced skill with handling distractions, she was clicking all the right things. As if it were all so simple. As if it were as easy as showing your daughter how to take a derivative of a trigonometric function while reconfiguring a server.

And then, with alarm and frustration in her voice, she asked: “Wait a minute! Was I supposed to do step 2 AFTER step 1?”

Here’s the thing. We all knew she wasn’t trying to be funny. After all, we had all skipped step one.

And that is when the S in MSH hit the fan.

Peter V. Tamas has been a UNIX Systems Administrator for so long that he remembers when keyboard settings would regularly go haywire, causing the delete key to produce ^H instead of doing its job.
Thoroughbred Racing History
 Peter V. Tamas is author of With Seabiscuit and War Admiral at The Race of the Century , published as an Apple iBook. The second edition will be released on December 15 and will be available via or as an Apple iBook.