AWS Cost Optimization: Fixing a $40,000/Month AWS Bill

The day I worked on a project that was basically giving AWS a BMW every month, just because the engineering team had no idea how cookie-based sessions worked.

Yes, not knowing how cookie-based sessions work can cost you $40K a month. Crazy, right? A cookie is basically a tiny piece of data stored by the browser, sometimes containing just a small amount of data to identify the user or their session and load their unique data from the DB, yet misunderstanding how sessions work can turn that into a $40K monthly AWS bill. Three certified AWS experts came up with the following solution: use ALB sticky sessions, so AWS would keep each visitor pinned to a specific server. And to handle 50 users, their implementation ended up running 250 servers just to make sure each visitor could get their own unique server and avoid overwriting each other’s data and causing confusion.

Then people have the audacity to say AWS is expensive.

And this is the problem with the industry: the more time passes, the less humanity remembers. In the case of technology, that loss of knowledge is incredibly expensive and causes massive waste in spending, which then gets blamed on the third party, in this case AWS. But even if you had your own servers in the office, you would have bought $100K worth of hardware to solve the exact same problem. The difference is that if you figure out the problem and implement the fix in AWS, you reduce the costs immediately. If you did the same with office hardware, you’d be stuck with unused servers. Sure, you could sell them, but what are the chances there would be someone in the office with the time to manage eBay listings?

The majority of high cloud bills are just a skill issue, laziness, and ego. I worked at companies where they had big RDS instances not because they were needed, but because of very poorly designed queries and the unwillingness to take responsibility and spend the time to improve them, redesign the schema in a way that was more efficient, even bother with indexes, or leverage features of the DB engine that would improve performance. Not to mention the times the DB had to be big, not because of load, but because the data we had was serious.

So yes, blame cloud providers all you want. Regardless of where inefficient code is running, you will still pay a bigger bill than necessary if your team lacks the knowledge or willingness to fix the underlying problems, and if leadership is not technical enough to recognize when to push for a better solution.

So it is a knowledge problem, not a cost problem. When I hear that AWS is expensive, I see short-sightedness and a lack of understanding of what AWS, or the cloud in general, has to offer. Let’s consider RDS, which you might think of as just a regular database, and ask: why pay extra for what AWS offers when I can run it in my own colocation center, or spin up an EC2 instance, install something like PostgreSQL, and be done with it? What you lose is all the operational work AWS handles for you:

• Minor updates that, when automatic minor version upgrades are enabled, happen during the maintenance window without you having to install them manually. You do not have to worry about them, manage them, or deal with the work itself; it is handled automatically for you.
• Multi-AZ DB instances, where you have a primary and a synchronous standby, not only for redundancy but, more importantly, to make database maintenance and failover much smoother. For underlying operating system maintenance, AWS can update the standby first and then switch traffic over before maintaining the old primary, with the failover typically lasting less than a minute. You do not have to manage that process at all; it just happens.
• Major version upgrades that are almost as simple. You tell RDS to upgrade from version X to a supported version X1, and RDS handles the rest: running the engine upgrade workflow, performing upgrade checks, and, for RDS PostgreSQL when backup retention is enabled, taking snapshots before and after the upgrade.

It costs more, but what you get for that price is that you simply do not have to think about the plumbing. Otherwise, you need someone who, on a daily basis, checks, maintains, updates, and takes responsibility for your database, and that can become an incredible amount of work. Of course, you can say, “Pff, I will never upgrade the database. I’ll just leave it on the version I started with and be done with it.” Sure. Then expose an old, unpatched database to the public and increase the risk that a vulnerability results in the data leaking, getting deleted, or, worse, getting modified. You do you, but what AWS provides is incredibly cost-effective, transparent to maintain, and it just works. In my career, I have had to upgrade so many different types of databases that I would never have enough time to learn all the intricacies of every database engine and how to perform every update and upgrade manually, let alone reproduce the smooth transition AWS provides. On AWS, when the target version is supported, I can simply say, “Upgrade from version X to X1,” and the job is done.

Do we also want to talk about power consumption? With AWS, you get no separate power bill. In a colocation center, you have to pay for the power you use, and if you need more, you have to ask and provision for more. With AWS, who knows, who cares? I want the biggest, meanest server available to crunch data for one hour. No problem. It might consume a ridiculous amount of power, but all you see is the AWS bill for the resources you used.

And that brings me back to the project burning $40K a month to serve roughly 50 users. AWS was not the expensive part. The misunderstanding was. AWS simply made it incredibly easy to turn a software problem into 250 servers.

But it also made it incredibly easy to undo the mistake.

Once you understand the problem, you fix the application, terminate the unnecessary servers, and the compute cost disappears immediately. If you had solved the same problem by buying physical servers, now you have a pile of hardware you need to sell. If those servers were sitting in a colocation center, you might also have committed power capacity that you are still paying for even though you no longer need it, until your provider lets you reduce that commitment or the contract comes up for renewal.

That is the part people miss. The cloud makes it extremely easy to waste money, but it also makes it extremely easy to stop wasting it.