Creating Python AWS Lambda Layers with Docker

By default your AWS Lambda functions include only core Python functionality. Lambda layers are a fantastic feature allowing you to add external dependencies to your Lambda functions. In this post I'll walk you through how you can create your own library of Lambda layers using Docker.

What are Lambda Layers?

If you're unfamiliar with Lambda layers, a layer is additional code you can include at runtime with your Lambda functions to incorporate external libraries into your Lambda function. By default Lambda functions only include core Python libraries. Layers are a way for you to include other external libraries into your Lambda functions. A useful feature of layers is that they exist outside of the Lambda function itself. Once you create a layer you can use that layer across all your Lambda functions. You can read more about layers from the AWS documentation.

Creating a Lambda Layer with Docker

For this post we're going to create a layer using Flask as an example. Creating a Lambda Layer with Docket has four main steps:

Setup the a local directory for the layer
Use Docker to install the layer packages
Create a zip file of your directory
Upload to AWS using the AWS CLI

The next few sections will describe each of these steps in more detail. The last section will show how these steps can be combined into a single bash script to make deployment even easier.

Setting Up Your Layer Directory

The first step is to build the local directory structure that will contain all the artifacts for our layer. In the terminal enter the commands below. A new directory called flask-layer is created along with a subdirectory python/lib.python3.8/site-packages where we will install our Python packages to.

mkdir flask-layer
cd flask-layer
mkdir -pv python/lib/python3.8/site-packages

If you want to create your Layer for multiple Python versions you can repeat the last command and change "python3.8" to whatever other version of Python you want to include.

Within the flask-layer directory create a requirements.txt file that includes the following.

flask==1.1.1

The requirements.txt file defines the packages that will be included in your layer. If you have additional dependencies you want to include in your layer you can add those to this requirements.txt file.

One thing to be aware of with layers is that when they are loaded into your Lambda function they cannot exceed 250MB in size. A Lambda function can include up to 5 different layers. Given these limitations, having smaller layers you can mix-and-match across your functions is preferable rather than building large, single-purpose, layers.

Installing the Requirements with Docker

Docker will install the dependencies you specify in requirements.txt. Docker has images to replicate the AWS Lambda environment. Using those images Docker will install the libraries you specified in requirements.txt that are compatible with the Lambda runtime. In the terminal enter the following code. Make sure you have Docker running before entering this command!

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"

After running the command you should see the site-packages directory you created in the previous step populated with all your dependencies.

If you have additional versions you want to create your Layer for make sure you substitute the occurrences of "python3.8" with the correct version you want to use.

Uploading Your Layer to AWS

AWS requires all the layer code to be in a zip archive, so we need to zip everything in the python directory.

zip -r flask-layer.zip python

All the contents of the python directory are included in the zip archive called flask-layer.zip.

Now the layer can be uploaded to AWS using the AWS CLI. You need to provide a few parameters in this step:

layer-name is the name your want to give your layer
description to briefly summarize the layer
zip-file is the path to the zip archive you created in the previous step
compatible-runtimes details the Python versions your layer is compatible with

Enter the following into the terminal to publish our layer to AWS:

aws lambda publish-layer-version \
    --layer-name "flask-layer" \
    --description "Lambda Layer for Flask 1.1.1" \
    --zip-file "fileb://flask-layer.zip" \
    --compatible-runtimes "python3.8"

And with that your Flask layer is published to AWS! Now let's simplify this process a little more by packaging some of these steps into a single bash script.

Creating Your Own Lambda Layer Directory

To manage my layers I created a single bash script packaging most of the steps above together to make deploying layers easier. I have a directory where I manage all my layers. Below is a sample of those layers. Each layer has its own directory and the create-lambda-layer.sh script handles installing the layer dependencies and uploading them to AWS.

Screenshot of Lambda layer directory

Here is what create-lambda-layer.sh looks like:

#!/usr/bin/env bash


while [[ "$#" -gt 0 ]]; do case $1 in
  -v|--version) pythonEnvs+=("$2"); shift;;
  -n|--layer-name) layerName="$2"; shift;;
  -d|--desc) layerDescription="$2"; shift;;
  *) echo "Unknown parameter passed: $1"; exit 1;;
esac; shift; done


# Create and install requirements to directory.
for penv in ${pythonEnvs[@]}; do
    mkdir -pv python/lib/${penv}/site-packages
    docker run -v "$PWD":/var/task "lambci/lambda:build-${penv}" /bin/sh -c "pip install -r requirements.txt -t python/lib/${penv}/site-packages/; exit"
done


# Create zip file of environments.
zip -r ${layerName}.zip python


# Publish Layer to Lambda.
aws lambda publish-layer-version \
    --layer-name "${layerName}" \
    --description "${layerDescription}" \
    --zip-file "fileb://${layerName}.zip" \
    --compatible-runtimes ${pythonEnvs[@]}

The script accepts parameters for Python versions (you can provide multiple!), a name and description for your layer. From those parameters the script will generate the file structure, run the docker command for each Python version, create the zip archive, and then update the layer to AWS.

We can test this by installing the Twilio helper library. In the Terminal, create a directory where you're going to store your Lambda layers.

mkdir my-lambda-layers
cd my-lambda-layers

Create a file inside that directory, create-lambda-layer.sh, and paste the code above into the file.

Each layer needs to have its own directory, run the following code in the Terminal to create a directory twilio-layer.

mkdir twilio-layer
cd twilio-layer

The last bit of setup is to create the requirements.txt file, which should include:

twilio==6.40.0

Now we can call our bash script to finish creating and deploying our layer (again, make sure you have Docker running before executing this code!)

bash ../create-lambda-layer.sh -v "python3.8" -n "twilio-layer" -d "Twilio 6.40.0 Layer"

And your layer is deployed to AWS!

If you want to create your layer for multiple different Python versions all you have to do is specify additional -v parameters with those versions. The example below creates a layer for the Python 3.6, 3.7 and 3.8 runtimes.

bash ../create-lambda-layer.sh -v "python3.8" -v "python3.7" -v "python3.6" -n "twilio-layer" -d "Twilio 6.40.0 Layer"

Easy Peasy!

Wrapping Up

In this post I showed you how to deploy a Lambda Layer through the commandline using Docker. The bash script provided isn't fully featured, but in its current form provides a convenient way deploy your Lambda layers. Layers are a neat tool that can help you add functionality to your Lambdas. Docker simplifies creating layers so you can manage your own library of layers that you can share with your friends and coworkers!

Happy Building!