Terraform @ Scale - Part 1b: Multi-Tenancy Architectural Example for Modular Cloud Infrastructures

In the previous part of this series, we explained the basics of the remote state concept in Terraform and how it can be used for information inheritance in multi-tenancy environments. Now we will illustrate this with a concrete architectural example.

Introduction

An effective multi-tenancy architecture should follow the organizational structure of a company while also considering cloud best practices. Modern cloud platforms offer various ways to map hierarchical organizational structures - whether through Compartments (OCI), Organizations and Accounts (AWS), Resource Groups (Azure), or Projects and Folders (GCP).

Introducing the Architectural Concept

In our architectural example, we see a "Global Customer Datacenter" as the top-level organizational unit. Within this unit, there are various areas:

Global Systems and Services: These include IAM, monitoring, billing, and other central services.
Regional Datacenters: Geographically distributed infrastructures for different requirements.
Organizational Subdivisions: For example, based on departments, security zones, or other criteria.

The key aspect of this model: Each organizational unit represents its own logical area with defined responsibility. These form natural boundaries between different tenants or organizational units while also providing a framework for inheriting permissions and configurations.

Hierarchical Organization as a Natural Multi-Tenancy Model

The hierarchical structuring of cloud infrastructure offers several advantages for multi-tenancy scenarios:

Hierarchical Organization: Units can be nested, similar to a folder structure.
Policy Inheritance: Permissions are inherited from higher to lower levels.
Resource Management: Limits can be defined per organizational unit.
Cost Tracking: Billing and metering are performed at the organizational unit level.

For example, a company could have a top-level unit for each business unit. Within these units, there could be additional subunits for production, development, and testing. At the lowest level, project-specific units may follow.
This hierarchy reflects the organizational structure while simplifying the management of access rights and resources. For instance, administrators can be granted access to a specific department unit without accessing other departments.

Different Segmentation Levels

We can segment our infrastructure along various dimensions:

Technical Segmentation: For example, "Location A Regional Datacenter" with different technical zones (DC1, DC2, DC3, DC4).
Organizational Segmentation: For example, "Location B Regional Datacenter" with different departments (Dept. 1, Dept. 2, etc.).
Security Zones: For example, "Location N Regional Datacenter" with high-security, medium, low, and public zones.
Value Streams: Business processes that can cut across other structures.

These different segmentation dimensions can overlap and complement each other. The challenge is to map them cleanly in Terraform while ensuring information flows between them.
Particularly important is the inheritance of data using remote state. Here we see how information flows from higher levels (Global Datacenter) to lower levels (regional datacenters and their subunits). This precisely aligns with the remote state pattern discussed in the previous section.

Practical Implementation Using Terraform Modules

Implementing such an architecture in Terraform requires a structured module approach. For each level of the hierarchy, we create dedicated Terraform projects.

Note: The exact configuration of the remote state data source varies depending on the backend used. Appropriate adjustments are required for common options such as S3, GCS, or Consul.

1. Root Module for Global Infrastructure:

# 1. Global Infrastructure (Root Level)
module "global_datacenter" {
  source = "./modules/organization"
  
  name        = "global-datacenter-customer"
  description = "Global Datacenter of the Customer"
  parent_id   = var.root_organization_id
}

module "global_network" {
  source = "./modules/network"
  
  organization_id = module.global_datacenter.id
  cidr_block      = "10.0.0.0/16"
  name            = "global-network"
}

# Essential Global Systems and Services
module "global_services" {
  source = "./modules/organization"
  
  name        = "global-services"
  description = "Required Global Systems and Services the Customer Needs"
  parent_id   = module.global_datacenter.id
}

# IAM, Monitoring, Billing Services Module would be implemented here...

output "global_datacenter_id" {
  value = module.global_datacenter.id
}

output "global_network_id" {
  value = module.global_network.id
}

output "global_services_id" {
  value = module.global_services.id
}

2. Regional Infrastructure by Technical Segmentation:

# 2. Regional Datacenter in Location A (Multiple DCs)
data "terraform_remote_state" "global" {
  backend = "remote"
  config = {
    organization = "my-org"
    workspaces = {
      name = "global-infrastructure"
    }
  }
}

module "location_a" {
  source = "./modules/organization"
  
  name        = "location-a-regional-datacenter"
  description = "Regional Datacenter Location A"
  parent_id   = data.terraform_remote_state.global.outputs.global_datacenter_id
}

# Provisioning the 4 DCs as Subunits
module "dc1" {
  source = "./modules/organization"
  
  name        = "dc1"
  description = "Datacenter 1"
  parent_id   = module.location_a.id
}

module "dc2" {
  source = "./modules/organization"
  
  name        = "dc2"
  description = "Datacenter 2"
  parent_id   = module.location_a.id
}

module "dc3" {
  source = "./modules/organization"
  
  name        = "dc3"
  description = "Datacenter 3"
  parent_id   = module.location_a.id
}

module "dc4" {
  source = "./modules/organization"
  
  name        = "dc4"
  description = "Datacenter 4"
  parent_id   = module.location_a.id
}

# Subnets of the Global Network for Location A
module "subnet_dc1" {
  source = "./modules/subnet"
  
  organization_id = module.dc1.id
  network_id      = data.terraform_remote_state.global.outputs.global_network_id
  cidr_block      = "10.0.1.0/24"
  name            = "subnet-dc1"
}

# More Subnets for DC2, DC3, DC4...

# Outputs for Use in Other Terraform Projects
output "location_a_id" {
  value = module.location_a.id
}

output "dc1_id" {
  value = module.dc1.id
}

output "subnet_dc1_id" {
  value = module.subnet_dc1.id
}

# Additional Outputs

3. Regional Infrastructure by Organizational Segmentation:

# 3. Regional Datacenter for Location B (Organizational Structure)
data "terraform_remote_state" "global" {
  backend = "remote"
  config = {
    organization = "my-org"
    workspaces = {
      name = "global-infrastructure"
    }
  }
}

module "location_b" {
  source = "./modules/organization"
  
  name        = "location-b-regional-datacenter"
  description = "Regional Datacenter Location B"
  parent_id   = data.terraform_remote_state.global.outputs.global_datacenter_id
}

# Organizational Departments
module "department_1" {
  source = "./modules/organization"
  
  name        = "department-1"
  description = "Department 1"
  parent_id   = module.location_b.id
}

module "department_2" {
  source = "./modules/organization"
  
  name        = "department-2"
  description = "Department 2"
  parent_id   = module.location_b.id
}

module "department_3" {
  source = "./modules/organization"
  
  name        = "department-3"
  description = "Department 3"
  parent_id   = module.location_b.id
}

module "department_4" {
  source = "./modules/organization"
  
  name        = "department-4"
  description = "Department 4"
  parent_id   = module.location_b.id
}

# Department-Specific Subnets
module "subnet_department_1" {
  source = "./modules/subnet"
  
  organization_id = module.department_1.id
  network_id      = data.terraform_remote_state.global.outputs.global_network_id
  cidr_block      = "10.0.10.0/24"
  name            = "subnet-department-1"
}

# Outputs for Further Terraform Projects
output "location_b_id" {
  value = module.location_b.id
}

output "department_1_id" {
  value = module.department_1.id
}

output "subnet_department_1_id" {
  value = module.subnet_department_1.id
}

4. Regional Infrastructure by Security Segmentation:

# 4. Regional Datacenter for Location n (Security Zones)
data "terraform_remote_state" "global" {
  backend = "remote"
  config = {
    organization = "my-org"
    workspaces = {
      name = "global-infrastructure"
    }
  }
}

module "location_n" {
  source = "./modules/organization"
  
  name        = "location-n-regional-datacenter"
  description = "Regional Datacenter Location n"
  parent_id   = data.terraform_remote_state.global.outputs.global_datacenter_id
}

# Security zones as subunits
module "high" {
  source = "./modules/organization"
  
  name        = "high"
  description = "High Security Zone"
  parent_id   = module.location_n.id
}

module "med" {
  source = "./modules/organization"
  
  name        = "med"
  description = "Medium Security Zone"
  parent_id   = module.location_n.id
}

module "low" {
  source = "./modules/organization"
  
  name        = "low"
  description = "Low Security Zone"
  parent_id   = module.location_n.id
}

module "pub" {
  source = "./modules/organization"
  
  name        = "pub"
  description = "Public Zone"
  parent_id   = module.location_n.id
}

# Security zone-specific subnets
module "subnet_high" {
  source = "./modules/subnet"
  
  organization_id       = module.high.id
  network_id            = data.terraform_remote_state.global.outputs.global_network_id
  cidr_block            = "10.0.20.0/24"
  name                  = "subnet-high"
  prohibit_public_ip    = true  
}

# Additional Security Resources like Security Lists and Network Security Groups...

# Outputs
output "location_n_id" {
  value = module.location_n.id
}

output "high_security_id" {
  value = module.high.id
}

output "subnet_high_id" {
  value = module.subnet_high.id
}

5. Department- or Team-Specific Infrastructure:

# 5. Example of resources in a department, which uses remote state information
data "terraform_remote_state" "location_b" {
  backend = "remote"
  config = {
    organization = "my-org"
    workspaces = {
      name = "regional-location-b"
    }
  }
}

module "vm_instances" {
  source = "./modules/compute"
  
  instances = {
    app_server_department_1 = {
      organization_id = data.terraform_remote_state.location_b.outputs.department_1_id
      subnet_id       = data.terraform_remote_state.location_b.outputs.subnet_department_1_id
      size            = "medium"
      image           = var.app_image_id
    }
    # Additional VMs ...
  }
}

Dynamic Module Configuration for Flexible Deployment Patterns

To further increase flexibility, we can also work with dynamic configurations. This is especially useful when managing a variety of similar resources:

module "regional_datacenters" {
  source   = "./modules/regional_datacenter"
  for_each = var.datacenter_configs
  
  name            = each.key
  description     = each.value["description"]
  parent_id       = data.terraform_remote_state.global.outputs.global_datacenter_id
  network_cidr    = each.value["network_cidr"]
  security_level  = each.value["security_level"]
  subunits        = each.value["subunits"]
}

# Usage example, how to call this module from a root module
# datacenter_configs = {
#   "europe-west" = {
#     description    = "European Datacenter West Region"
#     network_cidr   = "10.1.0.0/16"
#     security_level = "high"
#     subunits       = ["prod", "staging", "dev", "test"]
#   },
#   "us-east" = {
#     description    = "US East Coast Datacenter"
#     network_cidr   = "10.2.0.0/16"
#     security_level = "medium"
#     subunits       = ["prod", "dev"]
#   }
# }

Advantages of the Modular Structure

This modular structure allows us to translate the logical hierarchy of our organization into separate Terraform projects that can still exchange information:

Clear Responsibilities: Each team manages its own Terraform code and states.
Automatic Information Inheritance: The team managing the global infrastructure can make changes without requiring regional or department-level teams to adjust their code.
Isolated Deployments: Errors in one area do not automatically affect other areas.
Scalability: New organizational units can be added easily without modifying existing structures.
Consistent Governance: By inheriting outputs, consistent configurations are ensured across all levels.

Example of a Complete Multi-Tenant Setup

Here’s an example of how these patterns can be combined in a real project:

module "tenant_environments" {
  source = "./modules/tenant_environment"
  
  for_each = {
    for tenant in var.tenants : tenant.name => tenant
  }
  
  name            = each.key
  parent_id       = module.tenant_root_organization.id
  environments    = each.value["environments"]
  network_cidr    = each.value["network_cidr"]
  security_policy = each.value["security_policy"]
}

# The tenant_environments may then be used for different customers, 
# business units or projects.
# 
# Example:
# tenants = [
#   {
#     name = "finance-department"
#     environments = ["prod", "dev", "test"]
#     network_cidr = "10.10.0.0/16"
#     security_policy = "high"
#   },
#   {
#     name = "marketing-department"
#     environments = ["prod", "staging"]
#     network_cidr = "10.20.0.0/16"
#     security_policy = "medium"
#   }
# ]

With this structured approach, we can manage complex multi-tenancy environments in a way that is both technically robust and organizationally meaningful. The clear separation of responsibilities, combined with targeted information inheritance via remote states, enables us to work efficiently with Terraform even in large, distributed teams.

Security Considerations for Remote-State Information

As helpful as remote states are, they can potentially contain sensitive information. Particularly in multi-tenancy environments, protecting this information is essential.
To ensure that only authorized teams can access remote state outputs, you should implement the following measures:

• Access Restriction: Use the access controls provided by your backend (e.g., IAM policies for S3, ACLs for Consul, or role-based access control in Terraform Cloud/Enterprise).
• Minimization of Outputs: Define only the strictly necessary information as output. Each additional output attribute increases the risk of exposing sensitive data.
• Avoidance of Unnecessary Outputs: Outputs containing security-critical values (e.g., passwords, private keys) should ideally not be defined as outputs but managed through secret management services such as HashiCorp Vault or Secrets Manager.
• Consolidation of Information: Instead of providing individual database passwords, API keys, or other sensitive details as outputs, prefer structured outputs that bundle only the relevant parameters.
• Use of Proxy State Files: Filter the outputs of a remote state to include only information relevant to the respective tenant and provide this as a new state. The tenant should then only have access to the proxy state.

Error Handling with Remote-State Dependencies

Using remote states in multi-tenancy architectures presents unique challenges in error handling. Sources of errors such as unavailable backends, inconsistent outputs, or unexpected changes can lead to significant issues. To minimize these risks, you should implement the following measures:

• Availability Checks: Use Terraform's terraform_remote_state block in combination with query constructs like lookup() to mitigate errors when outputs are missing.
• Validation of Correct Outputs: Add tests to your modules and CI/CD pipelines to ensure that remote state outputs always provide the expected structures and contents.
• Fallback Mechanisms: If remote states are temporarily unavailable, using locally cached values (e.g., via environment variables) can serve as a temporary solution.

Example of a robust remote-state query with error handling:

data "terraform_remote_state" "network" {
  backend = "remote"
  config = {
    organization = "my-org"
    workspaces = {
      name = "global-infrastructure"
    }
  }
}

locals {
  private_subnet_id = lookup(data.terraform_remote_state.network.outputs.subnets, "private", null)
}

resource "example_vm" "app_server" {
  subnet_id = local.private_subnet_id != null ? local.private_subnet_id : var.default_subnet_id
}

In the next section, we will take a closer look at remote backends and examine how to best organize remote states hierarchically.