Network died, hard, during company Christmas party, leaving lone techie to fix it

Who, me? Welcome, gentle reader, to another instalment of Who, Me? in which we cushion your entry to the working week with tales of Reg readers having worse days than you. So kick off your shoes and socks, make fists with your toes, and read on.

This week meet a reader we'll Regomize as "Roy" who was contracted to a very large corporation. The company in question was upgrading the core switches on its network from Cisco 4500s to "the shiny new Cisco Nexus 7000 series" – that should give some idea of when this was happening.

As our story begins, the shiny new 7000s were in place and operating in parallel with the existing 4500s. All that remained was to "move all the root bridges for the vlans from the 4500 to the 7000."

How hard can that be? Sounds like a doddle. Roy does mention that there were some 80 of these switches distributed about the building, so maybe a little bit tricky.

Also, the plan was to switch the Nexus switches to use VTP, or VLAN trunking protocol. That should, Roy believed, involve sending a single command to the main switch – again, not a big deal for a resourceful guy like Roy – and the change would ripple out across the network.

He timed the changeover for the night of the company Christmas party, which he did not plan to attend. The idea was that with everyone celebrating the season, network downtime (Roy predicted about half an hour) would not cause a loss of productivity.

All appeared to go well at first. In fact Roy was on his way out of the building when the security guard in the foyer stopped him.

There was a problem. All of the security camera feeds had gone blank.

Roy tried to log into the network to investigate, and found he was locked out. He used his console to access the switches directly, and found that authentication was down. In fact, the entire VLAN was gone.

This could only mean one thing.

• Turning a computer off, then on again, never goes wrong. Right?
• Hacking a Foosball table scored an own goal for naughty engineers
• Security? Working servers? Who needs those when you can have a shiny floor?
• Data cleanser did its job, but – oopsie! – also doubled customers' bills

No, not terrorists, nor even exceptional thieves. It meant Roy had to reconfigure each and every individual switch manually. All 80 of them. And it had to be done before work in the morning, or his bosses at Nakatomi – er, actually that very large corporation – would not be well pleased.

Welcome to the party, pal.

It transpired that for VTP to work, the command had to be sent to each of the switches – not just one of them. Roy had not done that.

So he made his way around the building (he does not specify using the air conditioning ducts, but we like to imagine), reversing the changes he'd made to the Nexus switches and manually reconfiguring each one with the appropriate VTP command to get the VLAN working again.

What had been scheduled for half an hour ended up taking until 7:30 the following morning.

Ultimately it was a happy ending – thankfully without any inferior sequels.

Have you ever encountered a technical snafu that could potentially be adapted into a popular seasonal action flick? Tell us all about it in an email to Who, Me? and we'll share it with the world.