The Panacea-Tool Incident

A few years ago — 2014 maybe? — we were in the early days of distributed teams and were spread across three timezones. Timing was awkward. So, many of us would start our day from home to join calls and meetings. This was, for us the beginnings of regular telecommuting. To help ease the communications challenges, we also embraced the concepts of video conferencing, screen shares, and multimedia to communicate.

One morning, there was expert brought in to demonstrate and train the lot of us on the new Panacea that the company had invested in: an app that would help manage all of our systems. It was a unified, do-everything tool that would provide visibility of specific known-states and anomalies on any number of systems across our several geographic locations and datacenters. It would pin down the precise, exact origin of a problem, and eliminate the need to log into a server (via SSH, of course) ever again… in order to resolve the issue.

Anyway, while doing the demo, there was this one error that would occur, which would prevent moving any further with a demo or training.

It was something about a missing object, or log file, or permissions to it.

If only there was a tool that had the power and capacity to identify the problem and resolve it… we could use that. It would be a perfect opportunity!

Their sales engineer was stumped.

After he fought with it for half an hour or so, I suggested, that we take a quick look at the actual logs on the system. Odds are pretty good that they’d indicate where the problem was. There was no harm in checking.

“No!” he’d assert. “That’s the wrong way!” And we endured continuous rants of frustrations and borderline vulgarities from him. “This guy!” he had jokingly exclaimed, “What you want to do is impossible!

Oh, I’m sorry… I thought you had used the word, “IMPOSSIBLE.” just there.

Challenge accepted.

I quickly shared my screen and jumped over and skimmed the actual logs from the app on the server itself. Let’s see… at the end of the log file, it had logged that it had crashed. Why? Scroll up a few lines and… permission denied trying to write to one of its own files.

“Oh! I’ll just ‘chmod’ that file so its owner can write to it…”

He boisterously interrupted, “If that’s it, I’ll buy you a steak dinner!”

**tap,tap,tap** **Enter** “Okay, all set… let’s give it another try really quick…”

The problem went away. He was clearly offended that somebody could’ve done it “the wrong way” to find the problem and fix it so quickly.

Took about 20 seconds.

And the really amusing part is that all of this was perfect scenarios to demonstrate the power and capability of the app itself.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.