Life as a Telecommuter

Living in a tiny space has its benefits and its drawbacks. One drawback is that when the weather outside is somewhat cool or rainy or windy, spreading the living/working space to the outdoors becomes more challenging and we find that we’re cooped up in the house.

Sometimes, for days and days (or even weeks) at a time.

Today, it’s finally the perfect temperature outside with no wind and not a chance of rain for days, so I grab my chair and portable desk and set up my telecommuting workstation outdoors…

Then a bloke starts up a leaf blower for an hour…

And another 75 feet away starts up a power washer.

**sigh**

Eventual Consistency…

Eventual consistency is eventually consistent eventually.

Working with Terraform to create/destroy a non-trivial deployment into AWS can be an exercise in patience. The primary reason isn’t necessarily the quirks of Terraform – and it has a few manageable ones – but instead, it’s those things that we have absolutely no actual control over like Amazon’s necessary Eventual Consistency.

Eventual consistency, of course, is necessary because AWS is a distributed infrastructure. It takes time for configurations to be passed around to the assorted systems involved and implemented.

It works a bit like this:

Requestor: “Hey, Amazon, could you create me a VPC?”

AWS: “Done. Your new VPC ID is vpc-1234567890”

Requestor: “Great, thanks – now in vpc-1234567890, create me a loadbalancer named ‘webservers’ and one called ‘databases’”

AWS: “Whoa, woah, woah, woah! You need a VPC to do that.”

Requestor: “To do what?”

AWS: “To create a loadbalancer. You need a VPC.”

Requestor: “Yes, vpc-1234567890 – on that one, which you just created, put those loadbalancers in there.”

AWS: “I have no record of a working VPC with that name.”

Requestor: “But you just created it! You said, ‘Done: Your new VPC ID is vpc-1234567890’!”

AWS: “I said I created it. I didn’t say it was ready for use.”

Requestor: gives up… just before vpc-1234567890 is ready for use.

Just because the result code said it was “complete”, it doesn’t mean “functioning”.

However, because of Eventual Consistency, Amazon could take, per their docs, up to five minutes to complete some tasks, as engineers, we need to be aware and compensate for it. In practice, I’ve observed it can be just a few seconds, or even much longer than five minutes for some tasks.

I know what you’re thinking. You’re thinking, “I’ll just add a 30-second wait here or there.”

It’s a trap!

Instead, get creative and design your functions so they do a few things:

  • First: submit your Create request to AWS and wait for the receipt
  • Next: poll AWS every few seconds with a Describe
  • Next: Compare the Describe to the initial Create request
  • Finally: If they do match, poll once more to confirm again that they match.
  • Yes, really.

Another Successful Demo

Part of our fortnightly Sprints is an end of Sprint demo, where we showcase our creations to the organization.

I usually need a few days to prepare for a good demo – a minimum of a full workday to come up with an outline, a script, slides, ensure configurations are in place, run through it two or three times and iron out any glitches.

But it seems the less prepared I am and the more improvised things are, the more the team likes them.

Next Sprint, I’m not preparing a thing.

It’s A Trap!

Some time ago, we had to coordinate with a security firm for a scan of our environments: infrastructure, servers, applications, services, accounts, physical access – the lot. The conversation went a bit like this:

Security: “Could you get us your root password so we can test your systems for vulnerabilities?”

Me: “No.”

Security: “We need it to check for specific kernel-level threats.”

Me: “No.”

Security: “We were told we’d have your complete cooperation.”

Me: “Yep.”

Security: “But you aren’t cooperating. Are you now saying you aren’t going to help us?”

Me: “I don’t think we’re all on the same page here.”

Security: “What do you mean?”

Me: “You’re the security consultant. You’re experts. The premise is that you’re testing our systems to identify actual threats. By all means, do so. But you’ll not be getting a leg up from us by giving you instant access to anything. Additionally, while management has asked that we provide full cooperation, it’s actually a violation of company policy to give anyone any passwords to any systems – even for actual security tests. Further, even if it weren’t a violation of company policy and general good administrative practice, there simply aren’t any root passwords to give you.”

Panic in the eyes of management and a smug look on the security consultant’s face when they think that no password means anyone can log in.

Me: “I know what you’re thinking – but we use operating systems that, by default, don’t have root passwords because root accounts are gaping security threats. We can’t give you a root password because nobody can log in to the root account. We actually require each person to have their own credentials and approppriate levels of access so we can manage and audit. Nobody can log in to root. Ever.”

Security: “But if you could just go ahead and enable the root account for us so we can scan these systems for vulnerabilities…”

Me: “Sorry, but no. What you’re saying is that you want us to violate company policy by divulging passwords that don’t exist and to create a security hole so you can tell us that our systems now have a security hole that wasn’t there until you insisted that we create it. And so you can test what it is that root can do on a system? It’s root – it can do anything it wants, which is precisely why we have it disabled. I’m not going to make things less secure so we can make them more secure.”

Fast forward a few months and see a security report from another agency testing another company and find that they remarked that the primary security threat consisted of easily obtaining the root passwords to key systems by simply making a few phone calls, asking around, then using some good, old-fashioned social engineering to trick people into divulging them.

So, can you have our root passwords for your company-sanctioned security test?

Not a chance.

Breaking HipChat

Troubleshooting a scenario with a product with our team over HipChat but couldn’t remember exactly what was said. So I do a quick search for the words “nginx default” thinking I’d find that message where I’m sure somebody had described where our particular config for our “nginx default site” lived.

And I apparently broke HipChat.