Simple wrapper for Terraform

Terragrunt — Keeping your Terraform code DRY

Enhance your IAC experience

Photo by Cuong Do on Unsplash

Problems with bare Terraform

Terraform is a tool which allows you to write code which defines your infrastructure but there are few known issues with using terraform. The first one is terraform state file, the second one, which we will try to solve today is problem with keeping terraform code DRY (Do Not Repeat Yourself) — especially if you are using multiple environments.

There are few solutions — like workspaces, paying for terraform cloud, scripts with variables. We will focus on flexible and elegant solution which uses wrapper for terraform — Terragrunt.

Terragrunt folder structures

Terragrunt (in short) allows you to select terraform code directory and populate variables with easy state managment . Inputs for terraform variables are defined in .hcl files which can reference each other and more.

This is example folder structure which uses terragrunt for AWS kubernetes stack:

.
├── common.hcl
├── dev
│ └── terragrunt.hcl
├── prod
│ └── terragrunt.hcl
├── staging
│ └── terragrunt.hcl
└── tf_files
 ├── envs
 │ ├── asg.tf
 │ ├── eks.tf
 │ ├── iam.tf
 │ ├── kubernetes.tf
 │ ├── lb.tf
 │ ├── main.tf
 │ ├── s3.tf
 │ ├── security_groups.tf
 │ ├── variables.tf
 │ └── vpc.tf

• common.hcl — inputs and settings passed to each enviornment.
• tf_files — terraform code.
• dev/prod/staging terragurnt.hcl — inputs used only for specified environment.

Now to deploy environment use command terragrunt apply in one of environment directories (dev/staging/prod) and the magic happens. Lets boil down how .hcl files allows to be more flexible with terrafrom inputs.

Terragrunt config file structure

Terragrunt config contains blocks and attributes. Let’s start with common.hcl file. It contains definition for things every environment will use:

# common.hcl
# block locals
locals {
  region         = "us-east-1"
  bucket         = "example"
  key            = "terraform/${path_relative_to_include()}.tfstate"
  dynamodb_table = "terraform_locks"
  profile        = "my-aws-profile"
}
# attributes local
inputs = {
}
# block remote_state
remote_state {
  backend = "s3"
  config = {
    bucket         = local.bucket
    region         = local.region
    key            = local.key
    dynamodb_table = local.dynamodb_table
    profile        = local.profile
    encrypt        = true
  }
}

Locals block should be quite obvious — you can easily refer them later in terragrunt.

The most important code line is :

key = “terraform/${path_relative_to_include()}.tfstate”

This will ensure that the state will be saved to path which includes path in which .hcl lives. With that you can quickly assure that every environment state will be managed in different state file.

Now to make sure this will be referenced in other terragrunt files you need to add this:

# dev/terragrunt.hcl
include {
  path = find_in_parent_folders("common.hcl")
}
terraform {
  source = "${get_parent_terragrunt_dir()}//tf-files/envs/"
}
inputs {
  my_var = "var"
}

Use include block to add common.hcl file. Then use terraform to point directory with terraform code using relative path returned from function get_parent_terragrunt_dir() . To run develop terragrunt.hcl go to directory and run terragrunt apply . State will be saved in bucket as:

“terraform/dev.tfstate”

Each environment configuration is wrapped into single .hcl file. Of course if you want to keep states in different buckets it also possible. Just change bucket name to use function like state key.

Terragrunt commands

Most of terragrunt commands are wrappers around terraform commands:

terragrunt plan -> terraform plan
terragrunt apply -> terraform apply

But some of them proceed with additional changes. terragrunt init automaticaly creates bucket or DynamoDB (if using AWS).

Generating terraform code with Terragrunt

You can generate terraform code to skip some manual work. Example user case: you want to generate identical provider for each environment with availability to change AWS profile and region.

Easy as adding additonal block to you common.hcl :

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents = <<EOF
provider "aws" {
  version = ">= 2.57.0"
  profile = var.aws_profile
  region  = var.aws_region
}
EOF
}

Merging multiple Terragrunt files

Sometimes creating multiple resources with detailed configuration can require using dictionaries with many keys or nested blocks. For example:

# ETL
glue_pythonshell_jobs = {
     match-trainingfile = {
      default_arguments = {
        "--extra-py-files"      = "s3://extra-py-files/pyfile.py"
        "--job-bookmark-option" = "job-bookmark-disable"
        "--enable-metrics"                   = "true"
      }
      role_arn        = "arn:aws:iam::<redacted>:role/<redacted>"
      script_location = "s3://scripts/match_trainingfile.py"
      name            = "pythonshell"
      timeout         = "2880"
    },
     order-records = {
      default_arguments = {
        "--job-bookmark-option" = "job-bookmark-disable"
        "--job-language"        = "python"
        "--TempDir"    = "s3://pipeline/workspace/temporary/"
        "--enable-metrics"                   = "true"
      }
      role_arn        = "arn:aws:iam::<redacted>:role/<redacted>"
      script_location = "s3://scripts/order_records.py"
      name            = "pythonshell"
      timeout         = "2880"
    },
...
# Kubernetes
...

This is a dictionary in inputs attribute of terragrunt config file. If you ever want to manage also Kubernetes or VPC or anything more, image how big terragrunt can grow with that many inputs. Luckily terragrunt comes with solution to sort your inputs into files and merge them in main terragrunt file.

You can move all inputs for Glue jobs and ETL which I showed to file named etl.hcl in the same directory and put them into inputs attribute:

inputs = {
  ##
  ## Glue
  ##
 ....
}

And now refer to that file in our main terragrunt.hcl :

locals {
  etl_vars = read_terragrunt_config("${get_terragrunt_dir()}/etl.hcl", {inputs = {}})
}
inputs = merge(
  local.etl_vars.inputs,
{
  project_name = "test"
  project_env  = "dev"
  aws_profile  = "test-dev"
  aws_region   = "us-east-1"
  vpc_cidr_block     = "10.0.0.0/16"
  vpc_subnet_newbits = 8 # vpc_cidr_block mask bits +      vpc_subnet_newbits = subnet size
  # Kubernetes
  ...
}
)

This allows to manage terraform code which needs detailed inputs. As you can see there is a # Kubernetes section — we can move it to separate file too, if it grows to big and merge after like this:

locals {
  etl_vars = read_terragrunt_config("${get_terragrunt_dir()}/etl.hcl", {inputs = {}})
  k8s_vars = read_terragrunt_config("${get_terragrunt_dir()}/k8s.hcl", {inputs = {}})
}
inputs = merge(
  local.etl_vars.inputs, 
  local.k8s_vars.inputs,
{
  project_name = "test"
  project_env  = "dev"
  aws_profile  = "test-dev"
  aws_region   = "us-east-1"
vpc_cidr_block     = "10.0.0.0/16"
  vpc_subnet_newbits = 8 # vpc_cidr_block mask bits +      vpc_subnet_newbits = subnet size
}
)

Everything stays organized and easy to read.

Photo by Cesar Carlevarino Aragon on Unsplash

Final thoughts

Terraform is a great tool but I could not image working with it now without Terragrunt. This article is only a quick introduction and does not dive deeply into more hidden features. And yes, Terragrunt supports working with modules.

Of course Terraform has official solution to multi-environment management — Terraform Cloud. But if you are not convinced to enterprise solutions and costs — here comes Terragrunt.

Try it yourself. Could be a game changer for you.