Automatic Lambda Layer Deployment via AWS CodePipeline
Lambda layers allow us to package libraries and other dependencies for use with Lambda functions. In this article, I aimed to make the process of creating and publishing the layer easier. I used multiple services to achieve automation namely CodeCommit, CodeBuild, Lambda, S3, and CodePipeline. II is an automation product that automates the software deployment process; it manages all services I mentioned.
The pipeline consists of four stages: Source, Build, Deploy, and LayerDeploy. In the source step, I will cover both S3 and CodeCommit as Source, as they can replace each other according to the project needs. In the Build stage, I will cover CodeBuild; in the Deploy stage we will use S3; and lastly, in the LayerDeploy stage, it will be Lambda. We will start with creating resources individually. After creating the resources, we will continue with CodePipeline.
Source
CodeCommit is a managed, git-based version control service. It can be integrated with other AWS services seamlessly. Create a repository in CodeCommit, and give a name, other fields can be default.
After creation, pull the repository to your local and push the requirements.txt or just simply create a file. As an example, I will use the following requirements.txt.
or
Build
In the build stage, we will use CodeBuild. CodeBuild will take the requirements.txt file and then download pip modules to the folder that we specified. As the last step, it will create the zip file.
Create a CodeBuild named BuildLayer. As Source you can select either AWS CodeCommit or Amazon S3. You can set the fields from the ScreenShots below. For Environment Select Amazon Linux 2. Select Standard as Runtime and select the latest option for image and image version; the environment type will be Linux. Leave Service Role as default.
BuildSpec is a YAML-formatted file where we specify the steps to build an artifact. We can include it as a file from the source or insert it directly.
The BuildSpec does not check whether the package is pre-installed to the lambda or whether it meets the layer’s size requirements.
Check the reference: https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html
We don’t need to select artifacts. It would be beneficial to enable CloudWatch logs to troubleshoot in case of any errors.
Deploy
In order to store artifacts of CodePipeline, we need an S3 bucket. Create another S3 bucket, but you need to be careful because all resources should be in the same region. Give a name, and select the region as same as other resources. Leave other settings as default.
LayerDeploy
Create a lambda named layer_creator, select Python 3.9 as the environment and leave other fields as default. Paste the following content to the lambda function. In the following piece of code, contents are hard-coded, which is not ideal; for the production environments, you should use Environment Variables. Also, you can parameterize the lambda to achieve that you can follow: https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-variables.html
The function also needs some permissions to give the following policy to the Lambda role. Do not change the policy created by AWS; you may attach Customer Managed Policy or an Inline Policy.
Also, the completion time will take longer than three seconds, so we need to increase it to a higher value. The completion time depends on the packages; to be on the safe side, we can give it three minutes. If the layer does not exist, the code will create.
Crete another Lambda named lambda_layer_function with Python 3.9 runtime, other settings can be default. This Lambda will be the Lambda which CodePipeline will attach the layer.
Now it’s time to move to CodePipeline.
Create a Pipeline named LayerCreator you can change advanced settings if you want.
Select Source Provider, whichever you created. It can be S3 or CodeCommit.
Select CodeBuild as BuildProvider and select the Project name that you created.
In the deploy stage, select the lambda-python-layer-artifacts S3 bucket. Make sure you have selected “Extract file before deploy” because CodePipeline compresses the layer.zip again.
After creation edit the Pipeline and add another stage After Deploy named LayerDeploy then add action group select AWS Lambda as Action provider you can select any action name. Then save the pipeline.
After running Pipeline you will see
Check the function named lambda_layer_function for the automatically created layer :)
