Skip to content

State Management and Backends

Step 13 of 18

State Management and Backends

Introduction

This lesson covers the Terraform state management area of the Terraform Associate (004) exam, objectives 6a to 6d. The area holds four of the 37 objectives, about 11% of this course's mock questions (HashiCorp does not publish domain weights; that proportion is an OnLevelUp choice). The exam lasts one hour, tests Terraform 1.12, and mixes true/false, multiple-choice and multiple-answer questions.

State is the memory of Terraform. It maps each resource block in your code to a real object in the cloud, stores attributes that the provider returned, and records dependencies. Without state, Terraform could not know that aws_instance.web is the server i-0abc123, and every plan would try to create everything again. Because state can contain secrets and is shared by a team, how and where it is stored is one of the most important operational decisions in a Terraform project.

After this lesson you will be able to describe the local backend, explain state locking, configure remote state with a backend block, detect and handle resource drift, run refresh-only plans, and use moved and removed blocks to change state safely.

Learning objectives

  • Explain what state contains and why it exists.
  • Describe the local backend and its limits.
  • Explain state locking and what happens when a lock is stuck.
  • Configure remote state with a backend block, and migrate state with terraform init -migrate-state.
  • Define drift and handle it with terraform plan -refresh-only and terraform apply -refresh-only.
  • Use moved and removed blocks, and know when terraform state mv and terraform state rm still apply.

What state is and why it matters

By default state is a JSON document named terraform.tfstate. It records:

  • a mapping from every resource address (aws_instance.web) to its real ID;
  • all attributes the provider returned, including computed values;
  • metadata: Terraform version, a serial number, a lineage ID, dependencies;
  • the values of outputs and, sometimes, sensitive values in clear text.

State serves three purposes: mapping configuration to reality, performance (Terraform can use cached attributes instead of querying every object), and dependency tracking (it remembers the order in which resources were created so it can destroy them correctly).

Rules that follow from this:

  • Never edit the state file by hand. Use terraform state subcommands or configuration blocks.
  • Do not commit state to Git. It may hold passwords, private keys and tokens, and it is not designed for merges.
  • Treat the state as sensitive, even if outputs are marked sensitive. Marking a value sensitive hides it from CLI output, but the value remains in the state file.

6a. The local backend

A backend defines where state is stored and where operations run. If you do not configure anything, Terraform uses the local backend: it writes terraform.tfstate in the working directory and a backup terraform.tfstate.backup before changes.

HCL
terraform {
  backend "local" {
    path = "state/prod.tfstate"   # optional, default is terraform.tfstate
  }
}

Characteristics of the local backend:

Aspect Local backend
Storage A file on the machine that runs Terraform
Locking Yes, uses system file locks while an operation runs, but only on that machine
Sharing None: teammates cannot see your state
Durability Lost if the disk is lost; no versioning
Good for Learning, experiments, disposable environments

When several people work on the same infrastructure, the local backend is a risk: two engineers each hold a different copy of the state and each "fixes" the same resource, creating duplicates or destroying live objects.

6b. State locking

When a state-modifying operation (apply, destroy, import, state commands) starts, Terraform asks the backend to lock the state so that nobody else writes at the same time. If a second run tries to start, it fails with an error that shows the lock ID, the user, the operation and the time.

Key facts:

  • Locking is automatic on backends that support it. Use -lock=false to disable it, which is dangerous and only justified in emergencies.
  • -lock-timeout=5m makes Terraform wait for a lock to be released instead of failing immediately. This is useful in CI pipelines that may overlap.
  • Not every backend supports locking. The s3 backend uses native S3 locking through a lock file (use_lockfile = true; the older approach used a DynamoDB table); azurerm uses blob leases; gcs uses object locks; consul, pg and HCP Terraform also lock.
  • If a run crashes and leaves a stale lock, release it with terraform force-unlock <LOCK_ID>. Do this only after confirming that nobody is really running Terraform.

Worked example: a stuck lock in CI

Scenario. A pipeline job applying production infrastructure is cancelled mid-run. The next pipeline fails with Error acquiring the state lock and shows ID: 5f2c.... A teammate suggests deleting the state file and running apply again.

Reasoning, step by step.

  1. Deleting the state would make Terraform forget every resource, so the next apply would try to create duplicates. Bad idea.
  2. First confirm that the previous job is truly dead (check the CI system and the lock's "Who" and "Created" fields).
  3. If dead, the lock is stale. Release it with the lock ID shown in the message.

Solution.

Bash
terraform force-unlock 5f2c0a1e-9d7b-4c22-8f0e-1b6a4f6f0d11

Why not the alternatives. -lock=false removes the protection for every future run and could allow concurrent writes. Deleting the lock file by hand is backend-specific and error-prone, while force-unlock works the same way everywhere.

6c. Remote state with the backend block

A remote backend stores state in a shared, durable service. Typical choices are Amazon S3, Azure Blob Storage, Google Cloud Storage, HashiCorp Consul, PostgreSQL, or HCP Terraform.

HCL
terraform {
  required_version = ">= 1.9"

  backend "s3" {
    bucket       = "acme-terraform-state"
    key          = "prod/network/terraform.tfstate"
    region       = "eu-west-3"
    encrypt      = true
    use_lockfile = true
  }
}

Important rules about the backend block:

  • It lives inside the terraform block, and only one backend may be configured per root module.
  • It cannot use variables, locals or any expression: values must be literals, because Terraform needs the backend before it evaluates anything else. To keep values out of code, use partial configuration: leave arguments out of the file and supply them with terraform init -backend-config="bucket=..." or -backend-config=backend.hcl.
  • Credentials should come from environment variables or the provider's normal credential chain, not from the block, because .terraform/ and plan files may record backend settings.
  • A change to the backend block requires terraform init. Terraform detects the change and offers to migrate: terraform init -migrate-state copies existing state to the new backend, while -reconfigure ignores the old one and starts fresh with the new configuration.

Benefits of remote state: a single source of truth, locking across the team, encryption at rest, versioning of state objects (for instance S3 bucket versioning to recover from mistakes), and restricted access by IAM.

Sharing outputs between configurations

A different configuration can read the root outputs of another with the terraform_remote_state data source:

HCL
data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-terraform-state"
    key    = "prod/network/terraform.tfstate"
    region = "eu-west-3"
  }
}

resource "aws_instance" "app" {
  subnet_id = data.terraform_remote_state.network.outputs.private_subnet_id
  # ami and instance_type omitted
}

Only root-module outputs are exposed. The reader needs read access to the entire state, which can contain secrets, so teams often prefer a purpose-built data source or a published output channel for tighter control.

Comparison: backends

Backend Locking Notes
local File lock on one machine Default; not shared
s3 Lock file in the bucket (use_lockfile), or legacy DynamoDB table Enable bucket versioning and encryption
azurerm Blob lease Storage account and container
gcs Object-level locking Bucket versioning recommended
pg Advisory locks Needs a PostgreSQL server
cloud block (HCP Terraform) Yes, built in Remote runs, state history, access control

6d. Resource drift and refresh-only runs

Drift happens when the real infrastructure no longer matches the state, usually because someone changed a resource in the cloud console or another tool changed it. Terraform detects drift during plan: it first refreshes, reading the current attributes of each resource through the provider, and then compares configuration, refreshed state and prior state.

If a resource was modified outside Terraform, the plan shows the difference and normally proposes to bring the object back to the configured value. If the object was deleted, the plan proposes to recreate it.

Refresh-only mode

Sometimes you want to accept reality into the state without changing infrastructure, for example when an administrator legitimately added a tag. Use:

Bash
terraform plan  -refresh-only
terraform apply -refresh-only

A refresh-only plan only updates state to match real objects and shows the differences that would be recorded. It never creates, changes or destroys infrastructure. The older command terraform refresh does the same without showing a plan first, and the documentation recommends apply -refresh-only instead because it lets you review the changes. Note that a normal plan and apply also refresh; use -refresh=false to skip that read for speed.

After accepting drift, your configuration may still differ from the new state. Update the code so that the next normal plan is empty, or the next apply will revert the change.

Worked example: accepting a legitimate manual change

Scenario. The security team added the tag Owner = "secops" to a production bucket from the console. The configuration has only Environment = "prod". The next scheduled terraform plan shows ~ tags being removed.

Step by step.

  1. Decide whether the change is legitimate. Here it is.
  2. Run terraform plan -refresh-only. The output says the object changed outside Terraform and lists the added tag.
  3. Run terraform apply -refresh-only and confirm. State now contains the tag; infrastructure is unchanged.
  4. Add Owner = "secops" to the configuration.
  5. Run terraform plan: it reports no changes.

Why other options are weaker. A normal apply would remove the tag, which the security team wants to keep. Editing the state file by hand is unsafe. Ignoring the change with lifecycle { ignore_changes = [tags] } hides all future tag drift, which may not be desired.

Changing addresses safely: moved, removed, and the state commands

State maps addresses to objects. When you refactor code you change addresses; you need to tell Terraform that the object is the same.

moved block

HCL
moved {
  from = aws_instance.web
  to   = aws_instance.frontend
}

The block says that the object formerly tracked at from now belongs at to. On the next plan Terraform shows a "moved" action instead of destroying and creating. Typical uses: rename a resource, move a resource into or out of a module, change from single instance to count or for_each (aws_instance.web to aws_instance.web[0]). You can keep the block in the code for a while so that others who run the module also migrate; remove it after everyone has applied.

removed block

The removed block (Terraform 1.7 and later) removes a resource from state without destroying it, or declares that it should be destroyed:

HCL
removed {
  from = aws_s3_bucket.legacy

  lifecycle {
    destroy = false     # keep the real bucket, just stop managing it
  }
}

With destroy = false, Terraform forgets the object and leaves the infrastructure untouched. Without the lifecycle setting, the default is to destroy it. Remember to delete the original resource block when you add removed; having both is an error.

The terraform state command family

Command Purpose
terraform state list List addresses in state
terraform state show ADDRESS Show the attributes of one resource
terraform state mv SRC DEST Rename or move an address in state
terraform state rm ADDRESS Remove an object from state without destroying it
terraform state pull / push Download or upload the raw state
terraform force-unlock ID Release a stale lock

state mv and state rm are imperative and change state immediately, while moved and removed are declarative, reviewable in a pull request and applied consistently on every workspace. Prefer the blocks; the CLI commands remain valid and are still part of the exam.

Common pitfalls and exam tips

  • Exam tip: the default backend is local. If a question says "no backend is configured", state is terraform.tfstate on the local disk.
  • Exam tip: a backend block cannot interpolate variables. Use partial configuration with -backend-config.
  • Exam tip: marking a value sensitive does not remove it from state. Protect the state itself.
  • Pitfall: committing terraform.tfstate to Git.
  • Pitfall: disabling locking in CI to "make the pipeline faster".
  • Pitfall: running terraform state rm and then forgetting to delete the resource block, so the next apply recreates a duplicate.
  • Pitfall: renaming a resource without moved, which destroys and recreates it.
  • Exam tip: a refresh-only run never changes real infrastructure.
  • Exam tip: terraform_remote_state exposes root module outputs only.

Mini case study

Situation. Two engineers, Maya and Leo, share a Terraform project for a payments platform. State is local. Last week Leo applied a change and Maya, working from an older copy, applied another one the same day; a database subnet was destroyed.

Questions. (1) What is the root problem? (2) What setup do you propose? (3) How do you move the existing local state?

Answer.

  1. State is stored per machine, so each engineer had a different view and locks could not protect them.
  2. Use a remote backend with locking and versioning, for example S3 with use_lockfile = true, encrypt = true and bucket versioning, with IAM restricting who can read the bucket. CI then becomes the only place that runs apply.
  3. Add the backend "s3" block, run terraform init -migrate-state, answer yes to copy the existing state, check with terraform state list and terraform plan (it should show no changes), then delete the stale local copies.

Key terms

  • State: the file mapping configuration to real infrastructure.
  • Backend: where state is stored and operations run.
  • Locking: preventing concurrent state writes.
  • Drift: difference between state and real infrastructure.
  • Refresh-only: a mode that updates state without changing infrastructure.
  • moved / removed: declarative blocks that change state addresses or stop managing an object.
  • Partial configuration: supplying backend arguments at init time.

Summary checklist

  • I can say what state contains and why it must be protected and never committed.
  • I know the local backend is the default and unsuitable for teams.
  • I can explain locking, -lock-timeout and force-unlock.
  • I can write a backend block and know it cannot use variables.
  • I can migrate state with terraform init -migrate-state.
  • I can explain drift and use plan / apply -refresh-only.
  • I can use moved and removed, and I know the terraform state subcommands.

Official sources