AWS CLI + OSX + sed = Not quite as advanced as I expected

So I’m hacking through some stuff for work. And, as the story goes, one road leads to another, leads to another and before I know it, I’m deep in the low-level weeds and playing around with two different AWS accounts.

Yes, AWS has a simple solution for this, but I haven’t really needed it until now. I’ve been perfectly content to just stick my API keys in two different credentials files then use a little script to quickly switch between them.

First, my `aws-profile` script:

if [ $# -lt 1 ]; then
    echo "You must specify one of the following:"
    ls ~/.aws/*.* | tr '.' ' ' | awk '{print $3}' | sort -u
    exit 4
cd ~/.aws/
for f in config credentials
    rm -f ${f}
    ln -s ${f}.${profile} ${f}
    ls -alh ${f} | awk '{print $9,$10,$11}'

Pretty basic stuff there. A basic validation expecting at least one input value followed by some output showing what’s available if nothing is provided. Then it nukes a symlink from my `~/.aws/` directory and creates a new one pointing to one of my more-static credentials files.

What’s in those credentials files? Pretty simple as well – `` is formatted thus:

aws_access_key_id = MYAWSPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

Then my `credentials.nonprod` would have, as you might guess, my non-production AWS account’s credentials:

aws_access_key_id = MYNONPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

See? Basic.

Want to use the prod credentails before running AWS CLI commands?

$ aws-profile prod
config ->
credentials ->

Switch back?

$ aws-profile nonprod
config -> config.nonprod
credentials -> credentials.nonprod

Why I’ve been doing things this way isn’t hugely important at the moment, but understand it’s kind of necessary for now. Short story is that we have some automation that pulls credentials out of a user’s local .aws/credential file so they don’t have to key it in or, worse, accidentally commit it into a git repository.

Today I decide I’ll just tweak some stuff so our workflow will tolerate pulling a user’s credentials right from their single, AWS-sactioned, profile-compatible .aws/credentials file so I don’t have to switch profiles the dirty elegant way.

First, collect your AWS keys together for your accounts. Next, decide what you want to name each profile – I do ‘default’ and ‘zomgdontdoit’ to mark my production profile. And then create the new profile using the aws configure command:

$ aws configure
AWS Access Key ID [None]: ****************HERE 
AWS Secret Access Key [None]: ****************blah 
Default region name [None]: us-east-1
Default output format [None]: json 

Okay, now add a profile to it:

$ aws configure --profile zomgdontdoit
AWS Access Key ID [None]: ****************HERE 
AWS Secret Access Key [None]: ****************blah 
Default region name [None]: us-west-2
Default output format [None]: json 

The resulting credentials file will look something like this:

aws_access_key_id = MYNONPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

aws_access_key_id = MYAWSPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

Probably old news here. But the important part for those who may not know is that we now have two headers in the file named default and zomgdontdoit. Also we have variables identically-named, which would of course preclude us sourcing the file with something like `. ~/.aws/credentials` in hopes that aws_access_key_id and aws_secret_access_key will become envvars we can consume in, say, bash.

I know! I’ll just do a quick sed command to differentiate by ignoring everything up to the header I want and then deleting everything after where the next header might occur:

$ sed -n '1,/zomgdontdoit/d;/[/,$d;p' credentials
aws_access_key_id = MYAWSPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

So far, so good… how about _default_?

$ sed -n '1,/default/d;/[/,$d;p' credentials

What the…

Did I misspell *default*? No?

Much time passes along with much second-guessing reality and my life/career choices until I finally stumble across this little gem over on StackOverflow:

because, the `0,/pattern/` works only on GNU sed, (e.g. doesn’t works on OS X), here is an _tampered_ solution. ;)

(echo "dummy-line-to-the-start" ; cat - ) < list.txt | sed "1,/$PATTERN/d"

Seriously? OSX is promoted as the most advanced form of Unix in the world and a fundamental command lacks the ability to work like the rest of the civilized world? Okay, fine, we’ll try it:

$ ( echo "hacky-osx-workaround"; cat - ) < credentials | sed -n '1,/default/d;/[/,$d;/^$/d;p'
aws_access_key_id = MYNONPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah

Well, I suppose that works. Let’s try the other one to make sure we didn’t do something else unexpected:

$ ( echo "hacky-osx-workaround"; cat - ) < credentials | sed -n '1,/zomgdontdoit/d;/[/,$d;/^$/d;p'
aws_access_key_id = MYAWSPRODKEYGOESHERE
aws_secret_access_key = blahblahblahblahblahblahblahblahblah


Well, now it’s just a matter of sticking that into the workflow so it can capture ’–profile’ options and switch them cleanly without doing destructive filesystem operations.