Eventual consistency is eventually consistent eventually.

Working with Terraform to create/destroy a non-trivial deployment into AWS can be an exercise in patience. The primary reason isn’t necessarily the quirks of Terraform – and it has a few manageable ones – but instead, it’s those things that we have absolutely no actual control over like Amazon’s necessary Eventual Consistency.

Eventual consistency, of course, is necessary because AWS is a distributed infrastructure. It takes time for configurations to be passed around to the assorted systems involved and implemented.

It works a bit like this:

Requestor: “Hey, Amazon, could you create me a VPC?”

AWS: “Done. Your new VPC ID is vpc-1234567890”

Requestor: “Great, thanks – now in vpc-1234567890, create me a loadbalancer named ‘webservers’ and one called ‘databases’”

AWS: “Whoa, woah, woah, woah! You need a VPC to do that.”

Requestor: “To do what?”

AWS: “To create a loadbalancer. You need a VPC.”

Requestor: “Yes, vpc-1234567890 – on that one, which you just created, put those loadbalancers in there.”

AWS: “I have no record of a working VPC with that name.”

Requestor: “But you just created it! You said, ‘Done: Your new VPC ID is vpc-1234567890’!”

AWS: “I said I created it. I didn’t say it was ready for use.”

Requestor: gives up… just before vpc-1234567890 is ready for use.

Just because the result code said it was “complete”, it doesn’t mean “functioning”.

However, because of Eventual Consistency, Amazon could take, per their docs, up to five minutes to complete some tasks, as engineers, we need to be aware and compensate for it. In practice, I’ve observed it can be just a few seconds, or even much longer than five minutes for some tasks.

I know what you’re thinking. You’re thinking, “I’ll just add a 30-second wait here or there.”

It’s a trap!

Instead, get creative and design your functions so they do a few things:

• First: submit your Create request to AWS and wait for the receipt
• Next: poll AWS every few seconds with a Describe
• Next: Compare the Describe to the initial Create request
• Finally: If they do match, poll once more to confirm again that they match.
• Yes, really.