Setting up RDS on AWS Elastic Beanstalk

And why you shouldn’t trust the automatic RDS instance.

I have been using Elastic Beanstalk for a bit more than 6 months now and I have to admit that until last night I didn’t fully understand how it integrated with AWS Relational Database Service.

The point that escaped me was how when you access your environment’s configuration settings and choose or create a DB from there, a set of environment variables are automatically added to your environment, without them being explicitly shown anywhere.

So if you create a DB from here:

Elastic Beanstalk will create these variables and add them to your environment:

The mistake that I had been previously making was to go in the RDS settings to collect the relevant information about the DB, and then enter this information among my other environment variables in the EB environment.

For example like this:

What I didn’t realize was that the environment variables automatically created by Elastic Beanstalk had a higher priority than the ones manually entered by me.

I had cautiosly entered these variables and it so happened that they matched the database that I had embedded in the environment, but actually when from inside my app i referenced these variables, the ones being read were the ones that I could not see.

Why is this potentially a problem?

My environmnent instances CPU was maxed out and I needed to make an upgrade. I tried simply upgrading the environment from t2micro to t3small, but to no avail. The process would reach the last stage and then inevitably fail.

The AWS support told me that in this case all I can do is to create a new environment directly at t3 and then redirect the traffic to it. “Well, ok..” I said as I created the new environment and connected it to the DB from the first environment.

It’s important to note here how when I created the new environment, I didn’t attach a DB to it from inside the Elastic Beanstalk settings. So the infamous invisible environment variables weren’t there to be used. I entered the environment variables I knew and i set up the permissions so that the new environment could access the old DB.

Perfect. Done. Now I just had to terminate the old environment to avoid incurring in useless charges.

Somehow I knew that clicking on “Terminate Environment” on the old one was gonna cause me some issue but I really did not foresee what was about to happen.

When you terminate the environment, the environment will proceed to take down (delete!) every service attached to it. Yes that includes the database. Elastic Beanstalk f*****g deleted my production database!

How to avoid Elastic Beanstalk taking down entire DBs?

It’s simple, don’t use their automatic RDS instance embedded in the EB settings. Make your own instance from the RDS manager and then connect it to your environment using your own environment variables.

As an example I’ll show how I temporarily put a plug on the problem.

Environment variables:

And code:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': os.environ.get('RDS_ALT_DB_NAME'),
        'USER': os.environ.get('RDS_ALT_USERNAME'),
        'PASSWORD': os.environ.get('RDS_ALT_PASSWORD'),
        'HOST': os.environ.get('RDS_ALT_HOSTNAME'),
        'PORT': os.environ.get('RDS_ALT_PORT'),
    }
}

Happy hacking.