The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
Orphan cloud infrastructure: How deleted virtual machines leave behind thousands of dollars in unattached persistent disks, idle IPv4 addresses, and orphan load balancers.
When software developers and DevOps engineers terminate a virtual machine, test environment, or Kubernetes cluster, they assume the billing stops. They click "Terminate Instance" or delete an environment branch in CI/CD, and consider the task complete.
In reality, cloud hyperscalers (AWS, GCP, Azure) are designed so that companion resources persist independently after instance deletion—silently accumulating charges month after month:
1. Unattached Persistent SSDs: When an EC2 or Compute Engine instance is deleted without the ``DeleteOnTermination`` flag enabled, its root and data block volumes remain in the cloud account. A single unattached 1TB gp3 volume costs ~$80/month. Across a development organization where hundreds of test VMs are spun up and deleted annually, forgotten unattached disks quietly accumulate $1,000s in waste. 2. Idle Public IPv4 Surcharges: Due to global IPv4 exhaustion, hyperscalers now bill ~$0.005/hour for *every* unattached Elastic IP or idle external address. Organizations that reserve static IP pools for legacy deployments pay hundreds of dollars per year for addresses that route zero bytes of traffic. 3. Abandoned Application Load Balancers: An engineer provisions an ALB or NLB for a pilot customer or staging test. The backend instances are shut down months later, but the load balancer remains active at ~$25–$35/month plus hourly LCU metrics charges. 4. Abandoned Snapshot Trees: Automated backup scripts create daily volume snapshots without lifecycle retention policies, resulting in snapshot chains dating back 3 years that cost more than active production compute.
Stopping the zombie resource bleed requires automated, declarative FinOps hygiene: • Audit Unattached Block Storage: Execute automated CLI and IaC scans to detect volumes in the ``available`` state (unattached to any running instance) and tag them for scheduled deletion after a 7-day grace period. • Automate Elastic IP Release: Identify and release any public IP address not bound to an active network interface, or migrate workloads to private VPC subnets with consolidated NAT. • Delete Classic Load Balancers with Zero Targets: Implement Prometheus and CloudWatch alerts that flag load balancers reporting zero active backend targets for more than 48 hours. • Declarative Infrastructure as Code (IaC) Lifecycle: Manage 100% of cloud resources via OpenTofu or Terraform with strict tagging schemas (``Environment``, ``Owner``, ``ExpiresAt``) so that deleting a workspace cleanly purges all associated disks, IPs, and balancers.
Reclaim trapped capital from zombie infrastructure and direct your budget toward core business development.
Slash wasteful cloud expenditures by 30%–50% with our 2-week Cloud FinOps & Infrastructure Cost Optimization Sprint on Contra: https://contra.com/s/rLYsZ2vY-cloud-fin-ops-and-infrastructure-cost-optimization-sprint
#FinOps #Cloud #AWS #GCP #Infrastructure #DevOps #SRE
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Related posts
Mounting docker.sock in CI/CD: Why providing build jobs access to the host Docker daemon gives any pull request total root control over your cloud account.
To build container images inside continuous integration pipelines (GitLab CI, GitHub Actions, Forgejo/Gitea Act Runners), developers need a container runtime. The most common shortcut recommended across internet tutorials is simple: mount the host Docker UNIX socket into the runner container: ``volumes: ["/var/run/docker.sock:/var/run/docker.sock"]``.
In any environment where developers, contractors, or public contributors can submit pull requests, **mounting the host Docker socket is equivalent to granting unauthenticated root access to the entire host**:
1. Trivial Container Breakout: Access to ``docker.sock`` allows any process inside the container to communicate with the host Docker daemon. An attacker submitting a malicious pull request or tainted dependency simply executes: ``docker run -v /:/host -it alpine chroot /host`` In under 2 seconds, the attacker escapes the container, obtaining full root access on the host operating system. 2. Cloud IAM Instance Metadata Theft: Once on the host, the attacker queries the cloud instance metadata service (IMDS at ``169.254.169.254``). They extract temporary IAM session tokens associated with the runner's cloud instance role. If the runner instance has permissions to manage S3, push to ECR, or touch Kubernetes clusters, your entire cloud organization is compromised. 3. Persistent Pipeline Backdoors: Attackers can silently install rootkits, modify adjacent container volumes, or poison production build artifacts before they are deployed.
Hardening production CI/CD runner platforms requires eliminating privileged daemon sockets: • Adopt Rootless Podman & Quadlets: Deploy sovereign CI/CD runners (like Forgejo Act Runner) using rootless Podman supervised by systemd Quadlets. Containers run entirely inside user namespaces without root privileges on the underlying host. • Use Daemonless Image Builders: Replace Docker build commands with daemonless build tools like ``kaniko``, ``buildah``, or ``podman build``. These tools assemble OCI images completely in user space without requiring access to a host daemon socket. • Enforce IMDSv2 and Hop Limits: Configure your cloud provider to mandate IMDSv2 (requiring token-based HTTP PUT handshakes) and enforce an HTTP hop limit of 1 (``http-put-response-hop-limit=1``). This prevents containers running inside the VM from reaching the metadata endpoint entirely. • Ephemeral Disposable VMs: For untrusted public pipelines, run jobs inside single-use, ephemeral KVM virtual machines that are destroyed immediately upon job completion.
Stop insecure pipeline breakouts and protect your production cloud credentials.
Deploy a secure, air-gapped developer platform with our 1-to-2 week Sovereign Git Forge & CI/CD Runner Platform Sprint on Contra: https://contra.com/s/7vW66W2v-sovereign-git-forge-and-cicd-runner-platform-deployment
#Security #DevOps #Docker #CyberSecurity #Cloud #Infrastructure #SRE
Upstream dependency outages: Why relying on live internet package registries during CI/CD builds paralyzes releases during critical emergencies.
When a severe production zero-day vulnerability hits your application, engineering teams sprint to patch the code, commit the hotfix, and trigger the deployment pipeline.
Then, the build abruptly fails: ``npm ERR! 503 Service Unavailable``, ``pip._vendor.urllib3.exceptions.MaxRetryError``, or ``Failed to download metadata for repo 'baseos': Cannot download repomd.xml``.
A third-party package registry (npmjs.com, PyPI, Docker Hub, or an upstream Linux distribution mirror) is experiencing an outage or has throttled your IP address with rate limits. Because your pipeline reaches out to the public internet for every build, your entire release process is dead in the water.
1. The Upstream Single Point of Failure: Modern software builds download hundreds of megabytes of third-party libraries: npm modules, Python wheels, Go modules, Ruby gems, and RPM/DEB OS packages. Depending on external servers for every test and build run ties your deployment uptime to the uptime of dozens of external commercial platforms. 2. Silent Upstream Package Tampering: If a popular upstream package is yanked, hijacked by an attacker (account takeover), or republished with a breaking micro-patch, unpinned or non-cached pipelines immediately absorb the breakage or security payload. 3. Wasteful Bandwidth and Slow Pipelines: Downloading the same 2GB of base dependencies across 200 daily pipeline runs wastes gigabytes of cloud bandwidth and adds 5–10 minutes of needless waiting to every release.
Engineering true deployment resilience requires sovereign, local caching mirrors: • Deploy Dedicated Caching Proxies: Implement local, pull-through caching mirrors inside your infrastructure for primary package ecosystems (e.g., Verdaccio for npm, Devpi for PyPI, and Athens for Go modules). Builds fetch packages at LAN wire speeds (10Gbps+), while the proxy caches artifacts locally forever. • Maintain Sovereign RPM/DNF Mirrors: For operating system dependencies and container base layers, run automated local mirrors using ``reposync`` and ``createrepo_c``. Sign internal repositories with enterprise GPG keys and host them on internal hardened Nginx/Caddy distribution endpoints. • Enforce Deterministic Lockfile Pinning: Mandate cryptographic SHA-256 integrity verification across all application package managers (``package-lock.json``, ``poetry.lock``, ``go.sum``). Builds must reject any package whose hash differs from the audited lockfile. • Air-Gapped Build Execution: Configure production CI/CD runner fleets to operate in an air-gapped network mode where jobs resolve dependencies *only* from verified internal artifact registries, immunizing your releases against external internet blips.
Guarantee that your engineering team can build and deploy critical patches anytime, anywhere—regardless of external internet status.
Deploy a secure, resilient developer platform with our 1-to-2 week Sovereign Git Forge & CI/CD Runner Platform Sprint on Contra: https://contra.com/s/7vW66W2v-sovereign-git-forge-and-cicd-runner-platform-deployment
#DevOps #Packaging #Linux #Infrastructure #Architecture #CI-CD #SRE
Designed and deployed a containerized cloud environment to support reliable, always-on business applications. The setup included service deployment, restart and recovery controls, health monitoring, resource management, and operational safeguards to improve stability, maintainability, and production readiness.
