Boto3 and Flask Demystified
I was recently tasked with building and deploying a micro-service that required interacting with S3. Everything was well until the moment it was deployed and running in EC2 (Amazon Linux 2 AMI). At this moment I realized that there is some information missing about how to properly configure your EC2 instance to work with AWS.
First, let’s go over the official Boto3 installation and configuration instructions was recently tasked with building and deploying a micro-service that required interacting with S3. Everything was well until the moment it was deployed and running in EC2 (Amazon Linux 2 AMI). At this moment I realized that there is some information missing about how to properly configure your EC2 instance to work with AWS.
Install the latest Boto3 via pip
$ pip install boto3
Set up your AWS authentication credentials using the AWS CLI
$ aws configure
The result from running the previous command should leave you with a similar credentials file and config file
# ~/.aws/credentials [default] aws_access_key_id **=** YOUR_ACCESS_KEY aws_secret_access_key **=** YOUR_SECRET_KEY # ~/.aws/config [default] region = us-east-1
Please also note that alternatively, you can create these files yourself if the AWS CLI is not installed in your machine.
Ok so far so good :)
Once Boto3 credentials are configured we can import it into our application
import boto3 # Let's use Amazon S3 s3 = boto3.resource('s3')
At this point, if the application is running in EC2 the following error might be the reason you are reading this post.
Everything was running just fine locally in my Macbook’s local environment. What could have gone wrong?
AWS will tell you over and over the solution is to configure your credentials properly with their AWS CLI. Well, let me tell you this is not the solution at all and you shouldn’t waste any of your time trying to fix it this way.
My preferred deployment platform for Python projects is WSGI which stands for “Web Server Gateway Interface”. It is used to forward requests from a web server (such as Apache or NGINX) to a backend Python web application or framework. In other words, because webservers don't natively speak Python we need WSGI to make communication happen between the two.
And this is where the problem begins, I scoured the internet and found my answer in some random blog hidden in the deepest depths of a google search.
Here is the blog for reference
So basically the reason Boto3 can’t find my credentials is because WSGI is not ready at the time the application initializes.
We have a few options to fix that:
For the first option, we could create a .env file in our current working directory and save our AWS credentials there.
. ├── .env ├── run.py └── config.py # .env AWS_ACCESS_KEY_ID=YOURACCESSKEY AWS_SECRET_ACCESS_KEY=YOURSECRECTKEY
To make this work python-dotenv has to be installed first
$ pip install -U python-dotenv
And then we can load the env file to our *config.py *file like this.
# config.py from dotenv import load_dotenv load_dotenv() # OR, the same with increased verbosity: load_dotenv(verbose=True) # OR, explicitly providing path to '.env' from pathlib import Path # python3 only env_path = Path('.') / '.env' load_dotenv(dotenv_path=env_path)
Remember not to commit your .env file to version control to keep sensitive information safe, in this instance, it would our AWS credentials.
A second option is to add the credentials to our wsgi file like so:
# myapp.wsgi from run import app as application import sys import os sys.path.insert(0, '/var/www/html/app_name') os.environ['AWS_ACCESS_KEY_ID'] = 'YOUR_ACCESS_KEY' os.environ['AWS_SECRET_ACCESS_KEY'] = 'YOUR_SECRECT_KEY'
Project structure example:
. ├── run.py ├── myapp.wsgi └── config.py
Also, remember this file should not be committed to version control to protect sensitive information.
Ok so Boto3 is now able to find our AWS credentials, but there is still one issue. What if your project strictly requires you to keep this sensitive information out of the directory that is being served to the internet, specifically
/var/www/html/my_app. We are up to the challenge, let’s see how we can fix that.
The best way I was able to figure this one out was to pass my AWS credentials as Apache environment variables, this way they are set by the server environment. To do that we have to add them to the VirtualHost configuration.
In Ubuntu you would find this file in /etc/apache2/sites-enabled/000-default.conf
<VirtualHost *:80> ServerAdmin email@example.com ServerName example.com ServerAlias example.com SetEnv AWS_ACCESS_KEY_ID your_access_key SetEnv AWS_SECRET_ACCESS_KEY your_secret_key DocumentRoot /var/www/html/ WSGIScriptAlias / /var/www/html/my_app.wsgi ErrorLog /path/to/logs/error.log CustomLog /path/to/logs/access.log combined </VirtualHost>
If you restart apache
$ sudo service apache2 restart at this moment you will get the same no credentials found error. This is because wsgi does not pass the environment variables by default so we have to set them manually in our application’s wsgi file.
# myapp.wsgi import sys import os sys.path.insert(0, '/var/www/html/app_name') def application(environ, start_response): for key in ['AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY']: os.environ[key] = environ.get(key, '') from run import app as _application return _application(environ, start_response)
And now Boto3 can find the AWS credentials again and they are as secure as ever. Also make sure to restart apache to apply the changes.
$ sudo service apache2 restart
Feel free to hit me up if there are any typos, errors of a way to improve this article, questions, also if you would like to learn how to deploy your Flask application to AWS EC2 ubuntu head to my tutorial Deploy a Flask app on AWS EC2.