Event-Driven Image Resizer Using AWS S3, SQS, and Lambda.
Muhammad Tahir A
Like this project
Posted Dec 2, 2024
Developed a scalable, event-driven architecture for resizing images using AWS services, deployed via an Azure DevOps CI/CD, and built with AWS Cloudformation.
Developed a scalable, event-driven architecture for dynamically resizing images using AWS services, deployed via an Azure DevOps CI/CD pipeline, and built with AWS CloudFormation templates.
Technologies Used:
AWS Services: S3, SQS, Lambda, CloudFormation, IAM, CloudWatch
Azure DevOps: Pipelines for CI/CD
Programming: Python (with Pillow library)
Infrastructure as Code (IaC): AWS CloudFormation templates
Architecture and Workflow:
Source Bucket (AWS S3):
Images are uploaded to the S3 source bucket.
Each upload triggers an S3 event notification.
AWS SQS (Queueing Service):
The S3 event notification is configured to send a message to an SQS queue, ensuring decoupling and reliability in processing.
AWS Lambda (Processing Service):
A Lambda function is triggered by SQS to process the image.
The Lambda function resizes the image into predefined dimensions using the Pillow library.
Destination Bucket (AWS S3):
The resized images are saved in the destination bucket with organized folder structures based on dimensions (e.g.,
/thumbnails, /medium, /large).AWS CloudFormation (Infrastructure as Code):
Automated the provisioning of all AWS resources, including S3 buckets, SQS queues, Lambda functions, and permissions, using CloudFormation.
Azure DevOps CI/CD Pipeline:
Set up a pipeline to build, package, and deploy the Lambda function and CloudFormation templates.
Integrated unit tests and artifact storage for versioning Lambda function code.
Key Features:
Event-Driven Processing: Automatically processes images as they are uploaded, ensuring real-time resizing.
Serverless Architecture: Reduced operational overhead and ensured scalability with AWS Lambda.
Decoupled Components: Improved reliability and fault tolerance using SQS.
Reusable Infrastructure: CloudFormation templates provide modular and replicable infrastructure.
CI/CD Integration: Streamlined deployment processes with Azure DevOps pipelines.
Key Challenges and Solutions:
Scaling Processing for High-Volume Uploads:
Used SQS to manage event bursts and prevent Lambda throttling.
Handling Large Image Files:
Optimized the Lambda function memory and timeout settings to handle processing of high-resolution images.
Version Control and Deployment Management:
Integrated Azure DevOps for CI/CD, ensuring seamless deployments and artifact management.
Security:
Implemented fine-grained IAM roles and bucket policies for least-privilege access control.
