So I’m hacking through some stuff for work. And, as the story goes, one road leads to another, leads to another and before I know it, I’m deep in the low-level weeds and playing around with two different AWS accounts.

Yes, AWS has a simple solution for this, but I haven’t really needed it until now. I’ve been perfectly content to just stick my API keys in two different credentials files then use a little script to quickly switch between them.

First, my `aws-profile` script:

#!/bin/bash
if [ $# -lt 1 ]; then
    echo "You must specify one of the following:"
    ls ~/.aws/*.* | tr '.' ' ' | awk '{print $3}' | sort -u
    exit 4
fi
cd ~/.aws/
profile=${1}
for f in config credentials
{
    rm -f ${f}
    ln -s ${f}.${profile} ${f}
    ls -alh ${f} | awk '{print $9,$10,$11}'
}

Pretty basic stuff there. A basic validation expecting at least one input value followed by some output showing what’s available if nothing is provided. Then it nukes a symlink from my `~/.aws/` directory and creates a new one pointing to one of my more-static credentials files.

What’s in those credentials files? Pretty simple as well – `credentials.prod` is formatted thus:

[default]
aws_access_key_id = MYAWSPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

Then my `credentials.nonprod` would have, as you might guess, my non-production AWS account’s credentials:

[default]
aws_access_key_id = MYNONPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

See? Basic.

Want to use the prod credentails before running AWS CLI commands?

$ aws-profile prod
config -> config.prod
credentials -> credentials.prod

Switch back?

$ aws-profile nonprod
config -> config.nonprod
credentials -> credentials.nonprod

Why I’ve been doing things this way isn’t hugely important at the moment, but understand it’s kind of necessary for now. Short story is that we have some automation that pulls credentials out of a user’s local .aws/credential file so they don’t have to key it in or, worse, accidentally commit it into a git repository.

Today I decide I’ll just tweak some stuff so our workflow will tolerate pulling a user’s credentials right from their single, AWS-sactioned, profile-compatible .aws/credentials file so I don’t have to switch profiles the dirty elegant way.

First, collect your AWS keys together for your accounts. Next, decide what you want to name each profile – I do ‘default’ and ‘zomgdontdoit’ to mark my production profile. And then create the new profile using the aws configure command:

$ aws configure
AWS Access Key ID [None]: ****************HERE 
AWS Secret Access Key [None]: ****************blah 
Default region name [None]: us-east-1
Default output format [None]: json 

Okay, now add a profile to it:

$ aws configure --profile zomgdontdoit
AWS Access Key ID [None]: ****************HERE 
AWS Secret Access Key [None]: ****************blah 
Default region name [None]: us-west-2
Default output format [None]: json 

The resulting credentials file will look something like this:

[default]
aws_access_key_id = MYNONPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

[zomgdontdoit]
aws_access_key_id = MYAWSPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

Probably old news here. But the important part for those who may not know is that we now have two headers in the file named default and zomgdontdoit. Also we have variables identically-named, which would of course preclude us sourcing the file with something like `. ~/.aws/credentials` in hopes that aws_access_key_id and aws_secret_access_key will become envvars we can consume in, say, bash.

I know! I’ll just do a quick sed command to differentiate by ignoring everything up to the header I want and then deleting everything after where the next header might occur:

$ sed -n '1,/zomgdontdoit/d;/[/,$d;p' credentials
aws_access_key_id = MYAWSPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

So far, so good… how about _default_?

$ sed -n '1,/default/d;/[/,$d;p' credentials

What the…

Did I misspell *default*? No?

Much time passes along with much second-guessing reality and my life/career choices until I finally stumble across this little gem over on StackOverflow:

because, the `0,/pattern/` works only on GNU sed, (e.g. doesn’t works on OS X), here is an _tampered_ solution. ;)

(echo "dummy-line-to-the-start" ; cat - ) < list.txt | sed "1,/$PATTERN/d"

Seriously? OSX is promoted as the most advanced form of Unix in the world and a fundamental command lacks the ability to work like the rest of the civilized world? Okay, fine, we’ll try it:

$ ( echo "hacky-osx-workaround"; cat - ) < credentials | sed -n '1,/default/d;/[/,$d;/^$/d;p'
aws_access_key_id = MYNONPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

Well, I suppose that works. Let’s try the other one to make sure we didn’t do something else unexpected:

$ ( echo "hacky-osx-workaround"; cat - ) < credentials | sed -n '1,/zomgdontdoit/d;/[/,$d;/^$/d;p'
aws_access_key_id = MYAWSPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

**sigh**

Well, now it’s just a matter of sticking that into the workflow so it can capture ’–profile’ options and switch them cleanly without doing destructive filesystem operations.

Eventual consistency is eventually consistent eventually.

Working with Terraform to create/destroy a non-trivial deployment into AWS can be an exercise in patience. The primary reason isn’t necessarily the quirks of Terraform – and it has a few manageable ones – but instead, it’s those things that we have absolutely no actual control over like Amazon’s necessary Eventual Consistency.

Eventual consistency, of course, is necessary because AWS is a distributed infrastructure. It takes time for configurations to be passed around to the assorted systems involved and implemented.

It works a bit like this:

Requestor: “Hey, Amazon, could you create me a VPC?”

AWS: “Done. Your new VPC ID is vpc-1234567890”

Requestor: “Great, thanks – now in vpc-1234567890, create me a loadbalancer named ‘webservers’ and one called ‘databases’”

AWS: “Whoa, woah, woah, woah! You need a VPC to do that.”

Requestor: “To do what?”

AWS: “To create a loadbalancer. You need a VPC.”

Requestor: “Yes, vpc-1234567890 – on that one, which you just created, put those loadbalancers in there.”

AWS: “I have no record of a working VPC with that name.”

Requestor: “But you just created it! You said, ‘Done: Your new VPC ID is vpc-1234567890’!”

AWS: “I said I created it. I didn’t say it was ready for use.”

Requestor: gives up… just before vpc-1234567890 is ready for use.

Just because the result code said it was “complete”, it doesn’t mean “functioning”.

However, because of Eventual Consistency, Amazon could take, per their docs, up to five minutes to complete some tasks, as engineers, we need to be aware and compensate for it. In practice, I’ve observed it can be just a few seconds, or even much longer than five minutes for some tasks.

I know what you’re thinking. You’re thinking, “I’ll just add a 30-second wait here or there.”

It’s a trap!

Instead, get creative and design your functions so they do a few things:

• First: submit your Create request to AWS and wait for the receipt
• Next: poll AWS every few seconds with a Describe
• Next: Compare the Describe to the initial Create request
• Finally: If they do match, poll once more to confirm again that they match.
• Yes, really.