Field Notes · DevSecOps · HashiCorp Vault
Segregating Vault by Environment and Business Division
A reference architecture for running HashiCorp Vault across Dev, UAT, and PRD while keeping multiple business divisions cleanly isolated inside it — topology, namespace hierarchy, policy design, resilience, and the promotion workflow that ties it together.
01The two axes of isolation
Every Vault design ends up answering two separate questions, and conflating them is where most implementations go wrong:
- Environment isolation — Dev, UAT, and PRD have different change velocity, different blast-radius tolerance, and different approval gates. A secret that leaks in Dev is an inconvenience; the same leak in PRD is an incident.
- Tenant isolation — inside a given environment, separate business divisions (e.g. Cash Equities, Structured Products, Derivatives) need their own administrative boundary: their own policies, their own auth methods, their own secrets engines, with no ability to see or touch a sibling division's namespace.
Environment isolation is solved with separate Vault clusters. Tenant isolation is solved within each environment using Vault Enterprise namespaces — Vault's native multi-tenancy primitive. The two are orthogonal, and the diagrams below treat them that way.
02Environment topology
Three independently sealed clusters — never one cluster with an environment-as-namespace shortcut. A compromised or misconfigured UAT namespace should never be one policy typo away from reaching a PRD secret; a separate cluster makes that structurally impossible rather than policy-dependent. HashiCorp's sealing guidance is built on the same premise: production deployments should follow the published reference-architecture patterns for multi-cluster deployments, with independent unseal mechanisms per cluster so no single lost seal device cascades across environments.[1]
Why not one cluster with a "dev / uat / prd" namespace layer?
- Blast radius of the control plane itself. A Vault outage, a bad auto-unseal rotation, or an operator error against the shared cluster takes down all three environments at once.
- Different change cadence for Vault itself. You want to upgrade Vault's version, tune performance, or trial a new auth method in Dev before it touches PRD — impossible if they share a binary and a Raft ring.
- Compliance segmentation. Auditors expect PRD credential issuance traceable to a hardened, change-controlled system distinct from Dev.
03Namespace hierarchy inside each environment
This is where business-division segregation happens. Inside every environment cluster, each division gets a top-level namespace, and subdivides by team or application beneath it. Namespaces are hierarchical and fully isolated: policies, auth methods, secrets engines, and identity entities defined in one are invisible to another unless explicitly delegated. This is the pattern HashiCorp documents as Secure Multi-Tenancy (SMT) — each namespace behaves as an isolated, self-contained Vault instance, with delegated administration so a division admin runs their own tenant without platform-team tickets for routine changes.[2]
What each division namespace actually contains
| Component | Scoped to the namespace |
|---|---|
| Auth methods | Own AppRole (CI/CD), own OIDC or LDAP mount (human access), own Kubernetes auth mount (workload identity) — role bindings only that division controls. |
| Secrets engines | Own KV v2 mounts per application, own database secrets engine connections, own PKI mount / intermediate CA if the division issues its own certs. |
| Policies | ACL policies versioned entirely within the namespace — a Cash Equities policy cannot reference a Derivatives path even by accident, because the boundary is enforced at the API layer, not by convention. |
| Identity entities | Entities and aliases for that division's humans and service identities, so audit logs attribute every action without leaking identity data across the tenant boundary. |
04Policy design
Two policy layers do the real work: a thin platform policy every namespace inherits (deny-by-default, audit requirements, mandatory lease TTLs) and a division policy scoping access to that division's own paths only. HashiCorp's policy guidance leads with the same principle — restrict the root policy to emergency use, write policies as explicit and narrow as the task requires, and favor short-lived credentials so a leaked token has a small window of usefulness.[3][5]
# deny everything not explicitly granted
path "*" {
capabilities = ["deny"]
}
# every token must carry a bounded TTL
path "auth/token/create" {
capabilities = ["create", "update"]
max_wrapping_ttl = "15m"
}
# self-service lease renewal only
path "sys/leases/renew" {
capabilities = ["update"]
}
path "cash-equities/trading-apps/data/*" {
capabilities = ["read"]
}
path "cash-equities/trading-apps/database/creds/reader" {
capabilities = ["read"]
}
# explicit deny — even same-division prod
# secrets are unreachable from UAT AppRole
path "cash-equities/trading-apps/data/prod-only/*" {
capabilities = ["deny"]
}
Policies are authored as HCL, stored alongside the application's infrastructure code, and applied through Terraform's Vault provider — never clicked into the UI. HashiCorp's programmatic-best-practices guidance reinforces this: manage Vault resources through Terraform, avoid long-lived secrets in Terraform state, and use Sentinel to keep changes inside the pipeline.[4] The hardening guide also cautions against heavy use of wildcards and globs in path patterns — explicit, narrow paths are easier to reason about and audit.[3]
05Identity, auth, and the request path
Humans and machines authenticate differently, but both resolve to a Vault entity scoped to exactly one division's namespace, and every credential Vault hands back is short-lived by default.
06Business continuity, disaster recovery, and fault tolerance
Vault sits in the critical path of every application that needs a credential — if Vault is down, deployments stall and dynamic credentials stop being issued. Resilience is therefore designed in three layers, each answering a different failure question.
Layer 1 — Fault tolerance within a cluster (node fails)
Each cluster runs Raft integrated storage across an odd number of nodes — five for PRD — spread across availability zones. Raft tolerates the loss of a minority of nodes: a 5-node cluster survives two node failures with no data loss and automatic leader election. PRD adds performance standby nodes so read traffic scales horizontally and a leader failover doesn't drop read capacity.
Layer 2 — Disaster recovery (cluster or region fails)
PRD replicates to a DR secondary cluster in a separate region or data center. DR replication mirrors everything — secrets, configuration, policies, and tokens/leases — so on promotion, applications keep working with the tokens they already hold. The secondary remains sealed and passive until deliberately promoted, with its own independent unseal device so the loss of the primary's KMS can never strand both clusters.[1]
Layer 3 — Business continuity (process, not just infrastructure)
| Control | Practice |
|---|---|
| RTO / RPO targets | Define per environment. PRD: minutes-level RTO via DR promotion, near-zero RPO via continuous replication. Dev/UAT: snapshot-restore is usually acceptable — cheaper than a full DR pair. |
| Promotion runbook | DR promotion is a documented, rehearsed operator procedure — who decides, who executes, how DNS/load-balancer cutover happens, and how clients re-resolve. Rehearsed on a schedule, not discovered during an outage. |
| Snapshot discipline | Automated Raft snapshots on a schedule, encrypted, stored in a separate fault domain, and restore-tested regularly in an isolated environment to verify recovery objectives are actually met.[6] |
| Break-glass access | Sealed-envelope recovery-key ceremony (quorum of named officers) for the scenario where auto-unseal infrastructure itself is the casualty. Recovery keys are held by different people than the KMS administrators. |
| Client-side resilience | Vault Agent caching on application hosts, so a brief Vault outage doesn't immediately fail every credential lookup — apps ride through short blips on cached, still-valid leases. |
| Namespace-aware DR drills | Each division validates its own auth paths and secret access after every DR drill — a promotion that works for the platform team but breaks one division's OIDC mount is a failed drill. |
07The Dev → UAT → PRD promotion workflow
Vault configuration itself — policies, auth roles, secrets-engine mounts, namespace structure — is promoted the same way application code is: written once, applied to Dev automatically, and gated into UAT and PRD by review and a change record.
08Putting it together: one request, start to finish
The diagrams above show each layer in isolation. Here's the same system as a single lifecycle — a new application onboarding into cash-equities/trading-apps and earning its way from a Dev secret to a PRD credential.
cash-equities/trading-apps/<app>.cash-equities/trading-apps/<app>) exists independently in all three clusters — the same relative structure, three separate physical boundaries. Nothing is copied between environments except the reviewed Terraform; secrets themselves never cross a cluster boundary.09Governance controls worth setting on day one
These line up with HashiCorp's published production-hardening checklist — least-privilege policies, short TTLs, restricted root-token use, and tested disaster recovery are baseline recommendations for any production deployment, not extras.[3]
| Control | Purpose |
|---|---|
| Namespace resource quotas | Cap lease count and rate limits per division so one team's runaway job can't exhaust cluster capacity for the others. |
| Dedicated audit device per environment | Ship PRD audit logs to a separate, write-restricted sink so a PRD investigation isn't diluted by lower-environment noise. |
| Root-token lifecycle | Generate PRD root tokens only for a scoped, multi-operator, time-boxed ceremony — then revoke. Day-to-day admin runs through named, audited policies.[3] |
| Sentinel / OPA guardrails | Enforce structural rules platform-wide — e.g. "no policy may grant capabilities on sys/*" — so a division admin can self-serve without violating a firm-wide control.[4] |
| Namespace-scoped DR drills | Test failover per environment on a schedule — a Dev outage that takes a week to recover still stalls every division's delivery. |
—Takeaways
| Question | Answer |
|---|---|
| How do we separate Dev / UAT / PRD? | Separate Vault clusters, each independently sealed, each with its own audit sink. Never an environment-as-namespace shortcut inside one shared cluster. |
| How do we separate business divisions? | Vault Enterprise namespaces, one per division inside each environment cluster, with delegated administration so divisions self-serve inside their own boundary. |
| How does config move between environments? | Terraform-as-code against each cluster's backend, gated by pull-request review in Dev and a change record with named approval for UAT and PRD. |
| How does it survive failure? | Raft quorum across AZs for node loss, a sealed DR secondary with an independent unseal device for region loss, and rehearsed promotion runbooks + restore-tested snapshots for everything else. |
| What ties it all together? | Short-lived, dynamically issued credentials and per-request audit logging — the same identity-broker pattern that governs agentic-AI credentials applies directly to human and CI/CD access here. |
Vault Enterprise NamespacesACL PoliciesAppRoleTerraformSentinel / OPARaft StorageDR ReplicationPerformance Standbys
—References
This design pattern follows publicly documented HashiCorp guidance rather than a house-invented approach. Primary sources:
-
Sealing best practices — Vault | HashiCorp Developer Reference-architecture guidance for independent unseal mechanisms and multi-cluster production topology.
-
Namespace and secure multi-tenancy (SMT) support — Vault | HashiCorp Developer Namespaces as isolated tenant environments with delegated administration; see also the companion Secure multi-tenancy with namespaces tutorial.
-
Production hardening — Vault | HashiCorp Developer Baseline hardening checklist: least privilege, explicit path policies over broad globs, short-lived credentials, restricted root-token use, immutable upgrades.
-
Programmatic best practices — Vault | HashiCorp Developer Managing Vault resources through Terraform/IaC, protecting state, and using Sentinel to keep configuration changes inside the pipeline.
-
Access controls with Vault policies — Vault | HashiCorp Developer RBAC model, least-privilege policy authorship, and restricting root-policy use to emergency scenarios.
-
Extend Vault Enterprise for hybrid and multi-cloud deployments — HashiCorp Validated Patterns Deployment-plan workflows including baseline replication performance testing and IaC-driven multi-cluster management.
-
How to use Vault namespaces — HashiCorp Solutions Engineering Blog Practitioner guidance on when namespaces do (and don't) fit a multi-tenant deployment.