Run Scrapy command from AWS Lambda function using Serverless Framework
This repository provides a guide on running Scrapy commands from an AWS Lambda function using the Serverless Framework.
Pre-requisite mandatory steps to use this repository
(Skip if you already have AWS CLI, Nodejs, npm, Serverless and Docker installed)
1. Create Access and Secret Access Keys
For new IAM user:
Already have IAM user:
2. Installing AWS CLI on your system
Install AWS CLI on your system.
After installation go to the command prompt to review your installation type “aws help”
Type
aws configure and use the Access and Secret Access Keys from above step.3. Setup NodeJS
To install NodeJs go to nodeJs website and download/install.
To review your installation use
node --version from CLI to check the version.4. Installing and configuring AWS Serverless framework
Serverless is provided as npm package. To install serverless open CMD prompt and type
npm install -g serverlessTo review your installation use
serverless -v from CLI to check the version5. Install Docker
Install docker from Docker website
Create account on docker hub if you do not already have.
Run the docker on local machine and login to Docker hub using above credentials.
Steps to deploy new application on AWS Lambda using Docker and Serverless Framework:
1. Create new folder in your local machine
Name the folder
MyLambdaFunction (or any other name that you feel suitable for this project).2. Move into the created folder
cd into above created folder from CLI of your local machine.
3. Copy all the application data
Clone this Repo and then copy all the contents into
MyLambdaFunction folder.4. Create ECR Repository on AWS:
Use AWS Console or AWS CLI
Name the repo “my-repo-custom” (or any other name that you feel suitable for this project).
Copy the URI that was generated for this ECR repo, we will need it later.
5. NOTE
Please follow the pre-requisite section (above) that has 5 steps at the start of this documentation.
Proceed only if pre-requisite is fulfilled.
7. Edit serverless.yml
Locate the file
serverless.yml in MyLambdaFunction folderEdit the file and replace
region and image.The value for
image was copied from point 4 above. Make sure to add “:latest” at the end of image URI.For
region, use the region where you would like to deploy the Lambda function.8. Move into the created folder
cd into
MyLambdaFunction folder from CLI (if you are not already in it).9. From the CLI
Kindly make sure you have AWS CLI logged in using
aws configure command and use below command.Go to AWS Console for ECR and select the repository and click “View push commands”.
The commands here contains ->
docker build, docker tag and docker push.Select the OS that you are using on your local machine.
Start from step 2 (building the image) under the commands as step 1 was performed in previous step.
Please build image (Step 2) using command shared below in case of MacOs with M1 chip.
In case of MACOS, build the image using below command
docker buildx build --platform linux/amd64 -f ./Dockerfile -t <repo-Name> .
NOTE: Make sure not to change the name from any of these commands unless you know what you are doing.
10. Move into the created folder
cd into
MyLambdaFunction folder from CLI (if you are not already in it) and run below command serverless deploy --region <region>Replace the above where you want to deploy Lambda function.
13. Patience!
Wait for the stack to deploy and navigate to AWS Lambda console and test the same.
Steps to Update Lambda function.
Update the files as per your requirement in your local machine (Dockerfile/Serverless.yml/Code etc.) under
MyLambdaFunction directory.Follow steps 8 till 13 from above.
0 likes · 7 views · Jan 29, 2024
Run Scrapy command from AWS Lambda function using Serverless Framework
