Multi-Tenant Infrastructure Design and Implementation
Muhammad Indrawan
Multi-tenant infrastructure generation, and the three design decisions that turned out to matter most.
Press enter or click to view image in full size
customer cloud account
Most internal developer platforms deploy into infrastructure the platform team owns. You run the Kubernetes cluster, you hold the credentials, you set the quotas. The platform and the workload live in the same blast radius.
The platform I work on does not. A customer designs their backend on a visual canvas, and we generate Terraform and apply it into their own cloud project — an account we do not own, using credentials they granted us, against quotas we do not control.
That single constraint reshaped almost every design decision. This is what it changed.
The pipeline
A user drags services onto a canvas: HTTP endpoints, Pub/Sub topics and subscriptions, Firestore databases, scheduled jobs. The canvas is a directed graph — nodes and edges, saved as JSON.
From there:
A second path generates the application code for each service, pushes it to a per-project repository, and a CI runner builds the container and deploys it.
Nothing here is hand-written Terraform. The largest tenant generated
main.tf is 10,000+ lines and 613 resources — 131 Cloud Run services, 77 Pub/Sub topics, 108 subscriptions, 216 IAM bindings, 60 schemas. Hold that number; it comes back.
Decision 1: split state by lifecycle, not by service
The obvious question for generated infrastructure is how many Terraform states a project should have. One per service isolates failures. One per project is simpler to reason about.
We split by neither. Each project gets exactly two states, under prefixes that differ only in the last segment —
{tenant}/{cloud}/{project}/onboard and {tenant}/{cloud}/{project}/deploy.
Onboard holds the foundations: API enablement, the service accounts, the Firestore database, base project IAM. Deploy holds the workloads: Cloud Run services, topics, subscriptions, schedulers. Every workload in a project — all 613 of them for the largest tenant — lives in that single deploy state, generated as one root module.
The split is along churn, not ownership. The deploy graph is regenerated wholesale every time someone touches the canvas. The onboarding graph should change almost never. Keep them in one state and every routine deploy re-evaluates the tenant’s foundations — including, in the worst case, proposing to replace the very service account the deploy is authenticating with.
Blast radius follows from that. Losing or corrupting the deploy state is survivable: the workloads are described by the canvas, so they can be regenerated and re-imported. Losing the onboarding state means losing the record of how the tenant was bootstrapped — which APIs, which identities, which base IAM. That is the part you cannot rebuild from the design.
The costs are real and worth naming. One state per project means one lock, so two deploys against the same project serialise instead of running side by side. It also means one very large apply — which is exactly how Terraform’s concurrency setting turned out to be the single biggest lever on our deploy times, and a story of its own.
Both of those states live in our storage, not the customer’s. That split — between where the state lives and where the resources live — forces the next decision.Decision 2: two identities in every apply
Decision 2: two identities in every apply
This one is not obvious until you hit it.
Terraform needs credentials for two different things, and in a multi-tenant setup they are not the same principal:
The resource identity provisions Cloud Run, Pub/Sub and Firestore inside the customer project. This is a service account the customer granted us, fetched at deploy time from a secrets store, scoped to their project alone.
The backend identity reads and writes the Terraform state file, which lives in our bucket, not theirs.
If you use one identity for both, you have made a bad choice in one direction or the other: either the customer service account needs write access to a bucket holding every tenant state, or your platform identity needs standing permissions inside every customer project.
So the apply runs with the resource identity, and the backend is configured separately with the platform own credentials. The secret path is namespaced per tenant, so the path itself is the tenant boundary — you cannot accidentally read tenant B credentials while deploying tenant A, because the mount name is the tenant.
Decision 3: assume the resources already exist
Hand-written Terraform assumes state is authoritative: if it is not in state, it does not exist, so create it.
Generated infrastructure cannot assume that. State gets lost. A tenant is onboarded with resources already running. Someone creates something in the console. In every one of those cases, “not in state” does not mean “not in the cloud” — and the create path then fails with
409 Resource already exists, or worse, succeeds by clobbering something.So the generator runs an existence check before emitting HCL, and for anything already present it emits an
import block instead of a create. Terraform adopts the resource into state rather than recreating it.
The test that matters: delete the state file entirely, redeploy, and confirm the plan reads
0 to change, 0 to destroy. Everything that exists gets imported; nothing gets recreated.That check has a subtlety worth knowing about. The fast way to answer “does this exist?” is a bulk inventory API — one call listing everything in the project. But bulk inventory APIs have blind spots; some resource types simply never appear in the results. For those, absence from the inventory is indistinguishable from absence from the cloud, and you are straight back to a 409. The fix is a second signal: a direct
GET per candidate for the types the inventory will not report. Slower, but correct.What this architecture costs you
Three honest trade-offs, because a platform post without them is not worth much.
Generated graphs are wide and shallow. 131 Cloud Run services with no dependencies on each other. That is great for parallelism and terrible for anything that assumes a meaningful dependency tree — including Terraform default concurrency, which is tuned for the graphs humans write. Ours spent months bounded by a setting nobody had thought about.
There is a gap between declared and running. Terraform has to create a Cloud Run service before its container image exists, so it uses a placeholder image and CI replaces it later. Two independent events — and if the second one never happens, you have a service that is deployed, healthy, and serving a placeholder page. Your IaC says green. Reality disagrees. Nothing in the Terraform layer can detect this, because from Terraform point of view everything converged.
Press enter or click to view image in full size
Shared, project-scoped resources are the contention point. Per-service state solved isolation for everything a service owns. It solved nothing for project IAM, which is a single policy object that every service config wants to append to. That is a read-modify-write race, and it is the first thing that breaks when you turn concurrency up.
What I would tell someone starting this
Design for adoption, not just creation. The interesting state of a multi-tenant platform is not the empty project — it is the one that already has resources in it. If your generator can only create, you will discover that the first time you onboard a customer who is not starting from zero.
Separate the identity that owns state from the identity that owns resources. It costs one extra config block and it is the difference between a tenant boundary and a hope.
Know which of your operations are idempotent at the API layer, not just in Terraform. Enabling an already-enabled API is a no-op. Adding an existing IAM member is additive and harmless. Creating a named singleton — a schema, a scheduled job — is a hard failure. That distinction determines what is safe to leave on the create path and what genuinely has to be imported.
Measure the tail. The averages on a platform like this are meaningless: tenants differ by two orders of magnitude in size. Every real problem we found lived in the p90.
Next in this series: making Terraform state loss survivable, and what happened when a bulk inventory API quietly refused to admit two resource types existed.
Like this project
Posted Aug 6, 2026
Implemented multi-tenant infrastructure using Terraform with critical design decisions.
Likes
0
Views
0
