AWS Scraping Fleet with Terraform, Packer and Docker

A scraping fleet on AWS that processes hundreds of thousands of records in parallel and scales by changing two numbers: the input chunks and the instance count.

Packer builds a custom AMI, Terraform provisions the AWS resources, and an orchestrator launches EC2 instances that pull a prebuilt Docker image from ECR. Each instance fetches its chunk of inputs from S3, runs the scraper, writes structured results to DynamoDB and streams its logs to CloudWatch, with no shared state between instances.