Terraform @ Scale - Part 1c: Practical Implementation of Remote State Data Flows

Through a combination of carefully structured remote backends, thoughtful output design, and targeted use of the terraform_remote_state data source, you can establish a controlled information flow between different tenant levels - all without compromising the isolation of individual tenants.

Effectively using remote state for information exchange between organizational units requires a well-thought-out configuration of the Terraform environment. Central to this is the selection and setup of a suitable storage backend for storing state data in what are known as state files.

Configuring a Remote Backend

Terraform offers various official backends for storing state data. The options officially supported by HashiCorp are:

 terraform {
  backend "consul" {
    address = "consul.example.com:8500"
    scheme  = "https"
    path    = "terraform/network/global"
    lock    = true
  }
}

 terraform {
  backend "remote" {
    hostname     = "terraform.example.com"
    organization = "ict-technology"
    
    workspaces {
      name = "network-global"
    }
  }
}

 terraform {
  backend "remote" {
    hostname     = "app.terraform.io"
    organization = "ict-technology"
    
    workspaces {
      name = "network-global"
    }
  }
}

 These storage backends offer several advantages over local state files:

• Centralized Storage: All team members work with the same state.
• Locking Mechanisms: Prevents parallel changes and associated conflicts.
• Versioning: Stores the history of changes.
• Access Control: Enables differentiated permission concepts.

When choosing a backend, you should consider your organization's requirements.

Consul is an excellent choice for companies using the free version of Terraform. 

Terraform Cloud and Terraform Enterprise on the other hand have a client/server architecture and already include a storage backend on the server side, so there is no need to worry about this - you can simply use the integrated storage. Terraform Cloud/Enterprise also offer additional features such as Policy-as-Code for implementing and enforcing policies, governance controls, and enhanced collaboration features.

There are also other backends supported by Terraform. These include backends like Amazon S3, HTTPS, various databases, and more.
However, there is a catch - HashiCorp only offers support for local state files, Consul, and Terraform Cloud/Enterprise.
You can use other backends, but in case of issues, you cannot expect support or problem resolution from HashiCorp, even with an existing Enterprise support contract.
In critical production environments, you should therefore exclusively use Consul or Terraform Cloud/Enterprise as remote state backends.

If you are using a backend that is not supported by HashiCorp, you should ensure that the backend offers a locking mechanism!
Otherwise, you risk that during parallel runs of a terraform apply command, the parallel Terraform instances overwrite each other's state files, which would leave your entire infrastructure inconsistent. For example, backends like an NFSv3 share or an HTTPS backend are likely questionable choices and may result in data loss. 

Security Mechanisms

The storage of state files is critical as these may contain sensitive information such as access credentials and infrastructure details.

Encryption 

In Terraform Cloud/Enterprise, state data is automatically encrypted at rest and during transmission. In Consul, you should enable TLS for the connection and Consul's built-in encryption features:

terraform {
  backend "consul" {
    address      = "consul.example.com:8500"
    access_token = var.consul_acl_token  
    scheme       = "https"
    path         = "terraform/network/global"
    ca_file      = "/etc/ssl/certs/ca.pem"
    cert_file    = "/etc/ssl/certs/cert.pem"
    key_file     = "/etc/ssl/private/key.pem"   
  }
}

 Access Control

In Terraform Cloud/Enterprise, the user management system offers detailed roles and permissions. For example, different roles can have different permissions:

• Team Network-Admins: Full access to workspace network-global.
• Team App-Developers: Read-only access to workspace network-global.

In Consul, you use ACLs that you assign to a role. Here’s an example of an ACL that grants the App-Developers team read access:

# Consul ACL policy
key_prefix "terraform/network/global" {
  policy = "read"
}

In production environments, you should also use an ACL token to prevent unauthorized access by other clients:

 data "terraform_remote_state" "global_network" {
  backend = "consul"
  config = {
    address = "consul.example.com:8500"
    path    = "terraform/network/global"
    scheme  = "https"
    token   = var.consul_acl_token
  }
}

Versioning and Recovery

Terraform Cloud/Enterprise automatically creates snapshots of your states after each successful run. These snapshots can be used to restore the state of an environment in case of failure. To do this, use either the UI or the CLI (terraform state pull) to view the version history and restore the desired version.

In Consul, you can configure highly available clusters with replicas in other data centers to prevent data loss. You can also create snapshots of the Raft storage.

State File Structure for Optimal Multi-Tenancy Management

The organization of states should reflect your organizational structure while also supporting information flows.

In Terraform Cloud/Enterprise, various configurations are organized as "Workspaces." In Consul, separation is done through "Paths," which correspond to a logical namespace.

A proven structure could look like this:

Such a structure enables clear responsibility separation and simplifies locating relevant states.

In Terraform Cloud/Enterprise, you can additionally use workspace tagging to group related workspaces. This can help create a logical connection between the infrastructure of an application and that of the associated team.

Code Examples for Data Inheritance Between Tenants

The actual data inheritance is achieved using the terraform_remote_state data source, which enables access to outputs from other Terraform configurations.

Example: Network Configuration in a Multi-Tenant Environment

Global Network Team (in Workspace global-infrastructure):

module "global_network" {
  source = "./modules/network"

  name            = "global-vcn"
  description     = "Global customer network"
  cidr_block      = "10.0.0.0/16"
  parent_id       = var.root_compartment_id
}

output "global_vcn_id" {
  value = module.global_network.id
}

Regional Team (in Workspace region-dc1):

data "terraform_remote_state" "global_network" {
  backend = "remote"
  config = {
    organization = "ict-technology"
    workspaces = {
      name = "network-global"
    }
  }
}

module "regional_network" {
  source = "./modules/network"

  name            = "regional-subnet-dc1"
  description     = "Subnet of europe region"
  parent_id       = data.terraform_remote_state.global_network.outputs.global_vcn_id
  cidr_block      = "10.0.1.0/24"
}

output "regional_subnet_id" {
  value = module.regional_network.id
}

 Customer Team (in Workspace customer-acme):

data "terraform_remote_state" "regional_network" {
  backend = "remote"
  config = {
    organization = "ict-technology"
    workspaces = {
      name = "region-europe"
    }
  }
}

module "customer_environment" {
  source = "./modules/environment"

  name            = "customer-instance"
  description     = "customer instance"
  parent_id       = var.customer_compartment_id
  subnet_id       = data.terraform_remote_state.regional_network.outputs.regional_subnet_id
  instance_shape  = "VM.Standard2.1"
}

 A particularly recommended pattern is the use of structured outputs. Instead of providing many individual values, a structured output can consolidate various pieces of information:

output "network_config" {
  value = {
    vcn_id     = module.global_network.id
    cidr_block = module.global_network.cidr_block
    dns_label  = module.global_network.dns_label
    subnets    = {
      public  = module.subnet_public.id
      private = module.subnet_private.id
    }
  }
}

This allows dependent configurations to selectively access the relevant information:

module "app_server" {
  source = "./modules/compute"

  name            = "app-server"
  description     = "application server in private subnet"
  parent_id       = var.customer_compartment_id
  subnet_id       = data.terraform_remote_state.network.outputs.network_config.subnets.private
  instance_shape  = "VM.Standard2.1"
}

 In the next chapter, we will explore pitfalls, best practices, and some tips for working with remote states.