← Back to Portfolio Home Portfolio Vault Environment & Namespace Segregation

Field Notes · DevSecOps · HashiCorp Vault

Segregating Vault by Environment and Business Division

A reference architecture for running HashiCorp Vault across Dev, UAT, and PRD while keeping multiple business divisions cleanly isolated inside it — topology, namespace hierarchy, policy design, resilience, and the promotion workflow that ties it together.

01The two axes of isolation

Every Vault design ends up answering two separate questions, and conflating them is where most implementations go wrong:

  • Environment isolation — Dev, UAT, and PRD have different change velocity, different blast-radius tolerance, and different approval gates. A secret that leaks in Dev is an inconvenience; the same leak in PRD is an incident.
  • Tenant isolation — inside a given environment, separate business divisions (e.g. Cash Equities, Structured Products, Derivatives) need their own administrative boundary: their own policies, their own auth methods, their own secrets engines, with no ability to see or touch a sibling division's namespace.

Environment isolation is solved with separate Vault clusters. Tenant isolation is solved within each environment using Vault Enterprise namespaces — Vault's native multi-tenancy primitive. The two are orthogonal, and the diagrams below treat them that way.

02Environment topology

Three independently sealed clusters — never one cluster with an environment-as-namespace shortcut. A compromised or misconfigured UAT namespace should never be one policy typo away from reaching a PRD secret; a separate cluster makes that structurally impossible rather than policy-dependent. HashiCorp's sealing guidance is built on the same premise: production deployments should follow the published reference-architecture patterns for multi-cluster deployments, with independent unseal mechanisms per cluster so no single lost seal device cascades across environments.[1]

Fig. 1 — Cluster-level environment isolation
CI/CD PLATFORM
separate AppRole per environment · promotion gated by change record
authenticates per environment — no shared credentials
Dev Cluster
Raft integrated storage
Auto-unseal — cloud KMS
Own root token · own audit sink
UAT Cluster
Raft integrated storage
Auto-unseal — cloud KMS
Own root token · own audit sink
PRD Cluster
Raft — 5 nodes, multi-AZ
Auto-unseal — HSM-backed KMS
Performance standby nodes
PRD only — async replication to a separate fault domain
DR CLUSTER
passive replica · stays sealed until promoted · independent unseal device
Each environment is its own Vault cluster with its own root token, unseal mechanism, and audit sink. Nothing crosses cluster boundaries except the CI/CD platform's own gated pipeline; secrets never replicate between environments.

Why not one cluster with a "dev / uat / prd" namespace layer?

  • Blast radius of the control plane itself. A Vault outage, a bad auto-unseal rotation, or an operator error against the shared cluster takes down all three environments at once.
  • Different change cadence for Vault itself. You want to upgrade Vault's version, tune performance, or trial a new auth method in Dev before it touches PRD — impossible if they share a binary and a Raft ring.
  • Compliance segmentation. Auditors expect PRD credential issuance traceable to a hardened, change-controlled system distinct from Dev.

03Namespace hierarchy inside each environment

This is where business-division segregation happens. Inside every environment cluster, each division gets a top-level namespace, and subdivides by team or application beneath it. Namespaces are hierarchical and fully isolated: policies, auth methods, secrets engines, and identity entities defined in one are invisible to another unless explicitly delegated. This is the pattern HashiCorp documents as Secure Multi-Tenancy (SMT) — each namespace behaves as an isolated, self-contained Vault instance, with delegated administration so a division admin runs their own tenant without platform-team tickets for routine changes.[2]

Fig. 2 — Namespace tree (shown for PRD; identical shape in Dev/UAT)
root namespace
admin/ — platform-team owned · delegates one namespace per division
cash-equities/
division admin delegated
cash-equities/trading-apps/
cash-equities/market-data/
structured-products/
division admin delegated
structured-products/pricing/
structured-products/risk/
derivatives/
division admin delegated
derivatives/clearing/
derivatives/settlement/
The platform team owns the root namespace and delegates a namespace-admin policy to a lead per division. Each division then owns its own sub-namespace tree — creating child namespaces, auth methods, and secrets engines for its own teams. HashiCorp's namespace documentation describes this delegate-admin model directly, including subordinate delegates creating further child namespaces.[2]

What each division namespace actually contains

ComponentScoped to the namespace
Auth methodsOwn AppRole (CI/CD), own OIDC or LDAP mount (human access), own Kubernetes auth mount (workload identity) — role bindings only that division controls.
Secrets enginesOwn KV v2 mounts per application, own database secrets engine connections, own PKI mount / intermediate CA if the division issues its own certs.
PoliciesACL policies versioned entirely within the namespace — a Cash Equities policy cannot reference a Derivatives path even by accident, because the boundary is enforced at the API layer, not by convention.
Identity entitiesEntities and aliases for that division's humans and service identities, so audit logs attribute every action without leaking identity data across the tenant boundary.

04Policy design

Two policy layers do the real work: a thin platform policy every namespace inherits (deny-by-default, audit requirements, mandatory lease TTLs) and a division policy scoping access to that division's own paths only. HashiCorp's policy guidance leads with the same principle — restrict the root policy to emergency use, write policies as explicit and narrow as the task requires, and favor short-lived credentials so a leaked token has a small window of usefulness.[3][5]

Platform baseline — applies namespace-wide
# deny everything not explicitly granted
path "*" {
  capabilities = ["deny"]
}

# every token must carry a bounded TTL
path "auth/token/create" {
  capabilities = ["create", "update"]
  max_wrapping_ttl = "15m"
}

# self-service lease renewal only
path "sys/leases/renew" {
  capabilities = ["update"]
}
Division policy — cash-equities/trading-apps CI role
path "cash-equities/trading-apps/data/*" {
  capabilities = ["read"]
}

path "cash-equities/trading-apps/database/creds/reader" {
  capabilities = ["read"]
}

# explicit deny — even same-division prod
# secrets are unreachable from UAT AppRole
path "cash-equities/trading-apps/data/prod-only/*" {
  capabilities = ["deny"]
}

Policies are authored as HCL, stored alongside the application's infrastructure code, and applied through Terraform's Vault provider — never clicked into the UI. HashiCorp's programmatic-best-practices guidance reinforces this: manage Vault resources through Terraform, avoid long-lived secrets in Terraform state, and use Sentinel to keep changes inside the pipeline.[4] The hardening guide also cautions against heavy use of wildcards and globs in path patterns — explicit, narrow paths are easier to reason about and audit.[3]

05Identity, auth, and the request path

Humans and machines authenticate differently, but both resolve to a Vault entity scoped to exactly one division's namespace, and every credential Vault hands back is short-lived by default.

Fig. 3 — Authentication and dynamic-secret issuance, one division's namespace
HUMAN
engineer · OIDC via corporate IdP
CI/CD PIPELINE
AppRole — role_id + secret_id
K8S WORKLOAD
service-account JWT
namespace-scoped auth mounts
ENTITY + ALIAS
division ACL policy attached at auth time · token with bounded TTL
policy-gated access — division paths only
KV v2
app config, versioned
DB ENGINE
dynamic creds · TTL in minutes
PKI
short-lived certs, auto-rotated
every request logged
AUDIT DEVICE
who · which namespace · what path · when — shipped to SIEM
Nothing long-lived ever leaves Vault. Database credentials and certificates are minted per request with a TTL measured in minutes to a few hours — HashiCorp's baseline recommendation for guarding against credential compromise.[3] The audit device answers "who accessed what, from which division, and when" without cross-referencing a second system.

06Business continuity, disaster recovery, and fault tolerance

Vault sits in the critical path of every application that needs a credential — if Vault is down, deployments stall and dynamic credentials stop being issued. Resilience is therefore designed in three layers, each answering a different failure question.

Layer 1 — Fault tolerance within a cluster (node fails)

Each cluster runs Raft integrated storage across an odd number of nodes — five for PRD — spread across availability zones. Raft tolerates the loss of a minority of nodes: a 5-node cluster survives two node failures with no data loss and automatic leader election. PRD adds performance standby nodes so read traffic scales horizontally and a leader failover doesn't drop read capacity.

Layer 2 — Disaster recovery (cluster or region fails)

PRD replicates to a DR secondary cluster in a separate region or data center. DR replication mirrors everything — secrets, configuration, policies, and tokens/leases — so on promotion, applications keep working with the tokens they already hold. The secondary remains sealed and passive until deliberately promoted, with its own independent unseal device so the loss of the primary's KMS can never strand both clusters.[1]

Fig. 4 — PRD resilience topology: fault tolerance + DR
PRD Primary — Region A
AZ-1 · leader node
AZ-2 · follower + perf standby
AZ-3 · follower + perf standby
5-node Raft — survives 2 node losses
Auto-unseal: KMS-A (HSM-backed)
DR Secondary — Region B
Warm standby · sealed until promoted
DR replication: secrets + config + tokens/leases
Auto-unseal: KMS-B (independent)
Promotion = deliberate operator action
both clusters snapshot on schedule
RAFT SNAPSHOTS
automated, encrypted, stored in a third fault domain · restore-tested quarterly in an isolated environment — an untested backup is not a backup
Continuous async replication from primary to DR secondary. The independent unseal devices are the detail most designs miss: if both clusters depend on the same KMS, the KMS is a shared single point of failure that defeats the purpose of the second region.[1]

Layer 3 — Business continuity (process, not just infrastructure)

ControlPractice
RTO / RPO targetsDefine per environment. PRD: minutes-level RTO via DR promotion, near-zero RPO via continuous replication. Dev/UAT: snapshot-restore is usually acceptable — cheaper than a full DR pair.
Promotion runbookDR promotion is a documented, rehearsed operator procedure — who decides, who executes, how DNS/load-balancer cutover happens, and how clients re-resolve. Rehearsed on a schedule, not discovered during an outage.
Snapshot disciplineAutomated Raft snapshots on a schedule, encrypted, stored in a separate fault domain, and restore-tested regularly in an isolated environment to verify recovery objectives are actually met.[6]
Break-glass accessSealed-envelope recovery-key ceremony (quorum of named officers) for the scenario where auto-unseal infrastructure itself is the casualty. Recovery keys are held by different people than the KMS administrators.
Client-side resilienceVault Agent caching on application hosts, so a brief Vault outage doesn't immediately fail every credential lookup — apps ride through short blips on cached, still-valid leases.
Namespace-aware DR drillsEach division validates its own auth paths and secret access after every DR drill — a promotion that works for the platform team but breaks one division's OIDC mount is a failed drill.
Scope note: DR replication protects an environment against infrastructure loss — it is not a promotion path between environments. Dev, UAT, and PRD each get their own resilience posture; secrets never flow between them, even during recovery.

07The Dev → UAT → PRD promotion workflow

Vault configuration itself — policies, auth roles, secrets-engine mounts, namespace structure — is promoted the same way application code is: written once, applied to Dev automatically, and gated into UAT and PRD by review and a change record.

Fig. 5 — Configuration-as-code promotion pipeline
Dev
1
Engineer commits Vault policy / namespace Terraform to a feature branch.terraform plan output posted to the pull request for review
2
Merge to main auto-applies to the Dev cluster.division admins self-serve in Dev without a ticket
3
Automated policy tests run — expected allow/deny assertions against real auth paths.
Gate: change record approved for UAT?no → blocked, remediate in Dev
UAT
4
Same Terraform, UAT backend. No manual UI changes — ever.
5
Division validates in UAT — app teams exercise real auth and secret paths; business + security sign-off recorded.
Gate: second approval + change-record window matches execution time?no → blocked, remediate in UAT
PRD
6
Platform team executes the PRD apply.same module, PRD backend, named approver on record
7
Post-apply verification + audit-log spot check — confirm expected paths resolve and nothing unexpected changed.
One Terraform module targets three Vault backends via environment-specific variables. Division admins self-serve inside their own namespace in Dev; UAT and PRD promotion always carries a change record and a named approver.
Common failure mode to design against: letting "it worked in UAT" substitute for a PRD-specific review. UAT and PRD are separate clusters with separate unseal keys and separate audit sinks — a policy that behaves correctly in UAT still needs its own PRD apply and its own verification step.

08Putting it together: one request, start to finish

The diagrams above show each layer in isolation. Here's the same system as a single lifecycle — a new application onboarding into cash-equities/trading-apps and earning its way from a Dev secret to a PRD credential.

Fig. 6 — End-to-end workflow across environment, namespace, and promotion gates
1
Division admin requests app onboarding — platform team delegates cash-equities/trading-apps/<app>.
2
Engineer writes Terraform: policy + AppRole + KV/DB mounts, reviewed by division and platform team on the PR.
D
Dev cluster — auto-applied on merge; app authenticates via AppRole in its dev namespace; dynamic DB creds and KV config issued, TTL-bound; integration tests exercise real auth + secret paths.
Change record approved? No → remediate in Dev.
U
UAT cluster — same Terraform, UAT backend; app authenticates in its uat namespace; division runs business + security validation.
Second approval + CR window matches? No → remediate in UAT.
P
PRD cluster — platform team executes apply; app authenticates in its prd namespace on the HSM-backed cluster; short-lived DB creds and certs minted per request; every issuance logged — which app, on whose authority, when.
One namespace path (cash-equities/trading-apps/<app>) exists independently in all three clusters — the same relative structure, three separate physical boundaries. Nothing is copied between environments except the reviewed Terraform; secrets themselves never cross a cluster boundary.

09Governance controls worth setting on day one

These line up with HashiCorp's published production-hardening checklist — least-privilege policies, short TTLs, restricted root-token use, and tested disaster recovery are baseline recommendations for any production deployment, not extras.[3]

ControlPurpose
Namespace resource quotasCap lease count and rate limits per division so one team's runaway job can't exhaust cluster capacity for the others.
Dedicated audit device per environmentShip PRD audit logs to a separate, write-restricted sink so a PRD investigation isn't diluted by lower-environment noise.
Root-token lifecycleGenerate PRD root tokens only for a scoped, multi-operator, time-boxed ceremony — then revoke. Day-to-day admin runs through named, audited policies.[3]
Sentinel / OPA guardrailsEnforce structural rules platform-wide — e.g. "no policy may grant capabilities on sys/*" — so a division admin can self-serve without violating a firm-wide control.[4]
Namespace-scoped DR drillsTest failover per environment on a schedule — a Dev outage that takes a week to recover still stalls every division's delivery.

Takeaways

QuestionAnswer
How do we separate Dev / UAT / PRD?Separate Vault clusters, each independently sealed, each with its own audit sink. Never an environment-as-namespace shortcut inside one shared cluster.
How do we separate business divisions?Vault Enterprise namespaces, one per division inside each environment cluster, with delegated administration so divisions self-serve inside their own boundary.
How does config move between environments?Terraform-as-code against each cluster's backend, gated by pull-request review in Dev and a change record with named approval for UAT and PRD.
How does it survive failure?Raft quorum across AZs for node loss, a sealed DR secondary with an independent unseal device for region loss, and rehearsed promotion runbooks + restore-tested snapshots for everything else.
What ties it all together?Short-lived, dynamically issued credentials and per-request audit logging — the same identity-broker pattern that governs agentic-AI credentials applies directly to human and CI/CD access here.

Vault Enterprise NamespacesACL PoliciesAppRoleTerraformSentinel / OPARaft StorageDR ReplicationPerformance Standbys

References

This design pattern follows publicly documented HashiCorp guidance rather than a house-invented approach. Primary sources:

  1. Sealing best practices — Vault | HashiCorp Developer Reference-architecture guidance for independent unseal mechanisms and multi-cluster production topology.
  2. Namespace and secure multi-tenancy (SMT) support — Vault | HashiCorp Developer Namespaces as isolated tenant environments with delegated administration; see also the companion Secure multi-tenancy with namespaces tutorial.
  3. Production hardening — Vault | HashiCorp Developer Baseline hardening checklist: least privilege, explicit path policies over broad globs, short-lived credentials, restricted root-token use, immutable upgrades.
  4. Programmatic best practices — Vault | HashiCorp Developer Managing Vault resources through Terraform/IaC, protecting state, and using Sentinel to keep configuration changes inside the pipeline.
  5. Access controls with Vault policies — Vault | HashiCorp Developer RBAC model, least-privilege policy authorship, and restricting root-policy use to emergency scenarios.
  6. Extend Vault Enterprise for hybrid and multi-cloud deployments — HashiCorp Validated Patterns Deployment-plan workflows including baseline replication performance testing and IaC-driven multi-cluster management.
  7. How to use Vault namespaces — HashiCorp Solutions Engineering Blog Practitioner guidance on when namespaces do (and don't) fit a multi-tenant deployment.
© 2026 Brian Uckert · Be Digital Biz Inc.