Creating An AWS Lambda With Dependencies Using Python

AWS Lambda is an event-driven, serverless platform allowing you to produce anything from small, simple routines to larger, complex systems that run in an environment with their resources automatically managed by AWS.

You can write lambdas in most languages. The code is run on a Linux platform and can be configured to have access to any of your other AWS resources. Recently I’ve been writing some lambdas in Python, and they have some dependencies on external packages. It’s not immediately obvious how to make this work, so that’s the source of the problems covered by this article.

As I’ve mentioned in other articles, Amazon’s AWS documentation is generally good, but it often stops short of giving you the information you actually NEED to do a job. The lambda I was working on needed to made some HTTP calls, so I wanted to use the Python requests package. My first version of the Python script used:

import requests

so that I could use:

requests.get(...)

Running the script resulted in the following error though:

Unable to import module 'lambda_function': No module named 'requests'

The requests package isn’t installed by default in AWS lambdas, which is understandable given the number of Python packages that are available. There seems to be a way forward though: several sites and posts recommended using the version of requests made available by Amazon in the “vendored” package:

from botocore.vendored import requests

instead of the more usual:

import requests

This worked, but every time requests.get() was called the following dire warning was displayed in the output:

./botocore/vendored/requests/api.py:67: DeprecationWarning: You are using the get() function from 'botocore.vendored.requests'. This is not a public API in botocore and will be removed in the future. Additionally, this version of requests is out of date. We recommend you install the requests package, 'import requests' directly, and use the requests.get() function instead.

Just to ram the point home, attempting to use the requests.post() method with give you the following:

DeprecationWarning: You are using the post() function from 'botocore.vendored.requests'. This is not a public api in botocore and will be removed in the future. Additionally, this version of requests is out of date. We recommend you install the requests package, 'import requests' directly, and use the requests.post() function instead.

So, just when you think you’ve solved the original problem of there being no requests package available by using a different version, it turns out you really need the original anyway. Even more “helpfully”, most of the articles I found go on to explain how you should use pip to install requests on your environment. That’s fine if you’re not using lambdas, which don’t have a command line, but I am, so it’s not much use at all.

When you create a lambda and choose Python as a language, AWS initially creates a single file in the root directory:

lambda_function.py

This contains the lambda_handler() method, which the AWS lambda system looks for to handle events. You can edit this directly or, if this isn’t what you want, you can upload a deployment package in a ZIP file. This lets you create your lambdas locally, build them into deployment packages (ideally using a build server of course), and then upload an entire installation to AWS. It also lets you install dependencies locally, which is what I needed to do.

The full details of how to create a lambda deployment package are described at:

https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

The basic process to create one is as follows:

Create a project directory for the deployment package

Create requests.txt and lambda_function.py files in the project directory. List the dependencies as usual in requirements.txt, and put the event handler in lambda_function.py.

Download the Python packages you need for your project, and put them into the packages directory

Develop your lambda_handler method in lambda_function.py.

Create a ZIP file containing the packages, the lambda_function.py file and requirements.txt.

You can do this with or without a Python virtual environment – below is an example script to build a virtual environment called “virtualenv”, download the packages you specify and build the deployment package. It assumes you’re using Python 3.7 – AWS Lambda currently lets you use Python 2.7, 3.6 or 3.7.

#!/bin/bash
# Create a ZIP file allowing a lambda with dependencies to be uploaded to AWS.
# Jason Ross, https://www.softwarepragmatism.com
#
# Put values we use it repeatedly into variables.
packagename="deploymentpackage.zip"
venvname="deploymentvirtualenv"
echo "Creating Virtual Environment"
python3 -m venv --clear ${venvname}
echo "Virtual Environment created"
echo "Downloading dependencies"
source "./${venvname}/bin/activate"
pip install -r requirements.txt
deactivate
echo "Dependencies downloaded"
echo "Creating zip file"
cd "${venvname}/lib/python3.7/site-packages"
rm -f "${OLDPWD}/${packagename"
zip -r9 "${OLDPWD}/${packagename}" . 
cd ${OLDPWD}
zip -g $packagename requirements.txt
zip -g $packagename lambda_function.py
rm -Rf "./$venvname"
echo "Finished"

The advantage of using a virtual environment is that it simplifies the development process – you can develop the lambda and test it in the virtual environment (just don’t deploy any test code or modules to production!). The virtual environment keeps all of your development projects separate and independent, and if anything goes wrong you can delete the virtual environment and recreate a new version.

Once you’ve produced the ZIP file, you can use the upload option on the lambda’s edit page, or you can use the Amazon AWS CLI update-function-code command to upload it to the lambda. With a few changes, the script above can be combined with update-code-command in your build server, and you can automate the build and deployment.

If you’re developing lambdas using Python, and you’re having problems because your code depends on other packages, this should get you started.

Update

It's likely that if you try to use a large, compiled Python package such as Pillow or NumPy, you'll have problems importing the dependency or you'll get "Undefined Symbol" errors. In that case, take a look at my newer article Creating An AWS Lambda With Dependencies Using Python, Part 2 - Fixing Import and Undefined Symbol Errors.

External Links