Hello, I’m keke from the SRE team.

At Eukarya Inc., we use Terraform to manage resources such as Google Cloud. Re:Earth is a general-purpose WebGIS platform that not only powers the managed service Re:Earth Cloud but also hosts systems like the Ministry of Land, Infrastructure, Transport and Tourism’s PLATEAU project. Moreover, since it is deployed across multiple environments such as production, development, and testing, and includes other components like Re:Earth CMS, we have many reusable Terraform files.

To improve reproducibility and maintainability, we abstracted these Terraform files using Terraform Modules and manage all Terraform files in a single repository (Monorepo).

Recently, we implemented CI/CD with GitHub Actions to automatically execute terraform plan and terraform apply, allowing developers to start interacting with Terraform and managing resources. Currently, the limited SRE members are still refining this setup, navigating through trial and error.

In this article, I will discuss an issue we encountered during this process, where terraform apply did not work properly after modifying a Terraform Module, and how we resolved it.

Directory Structure in the Monorepo

Before diving into the issue, let me explain the directory structure of our Terraform repository.

Here’s a simplified version of the actual Monorepo structure we use. Each terraform/product/deployment-environment directory corresponds to a Google Cloud project.

.
├─ modules
│  ├─ reearth
│  └─ reearth-cms
└─ terraform
   ├─ cloud
   │  ├─ dev
   │  │  ├─ reearth.tf
   │  │  └─ reearth_cms.tf
   │  └─ prod
   │     ├─ reearth.tf
   │     └─ reearth_cms.tf
   ├─ plateau
   │  ├─ dev
   │  │  ├─ reearth.tf
   │  │  └─ reearth_cms.tf
   │  └─ prod
   │     ├─ reearth.tf
   │     └─ reearth_cms.tf
   └─ ...

For example, the reearth.tf file specifies a Terraform Module as follows. In the production environment, we pin the version using tags, while in the development environment, we allow applying changes to the Module at any time.

# Development environment
module "reearth" {
     source = "../../modules/reearth"
     ...
}

# Production environment
module "reearth" {
     source = "<GITHUB_REPOSITORY_URL>//modules/reearth?ref=v0.1.0"
     ...
}

By maintaining these Terraform Modules, we can apply consistent changes to resources deployed across various environments, reducing operational burden. This setup also ensures unified implementation and naming conventions for security and best practices.

CI/CD Using GitHub Environments and changed-files

Next, let me explain the CI/CD workflow for Terraform. Using GitHub Actions, we execute terraform plan on pull requests (PRs) and post the results as comments on the PRs.

name: Terraform Plan
on:
  pull_request:

jobs:
  changes:
    runs-on: ubuntu-latest
    permissions:
      contents: read # To checkout
    outputs:
      terraform-modified: ${{ steps.terraform-changed-files.outputs.changed }}
      terraform-modified-files: ${{ steps.terraform-changed-files.outputs.changed_files }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - id: changed-files
        uses: tj-actions/changed-files@v44
        with:
            dir_names_max_depth: 3
            files: terraform/**/*.{hcl,tf,yaml}
            matrix: true

  plan:
    if: needs.changes.outputs.terraform-modified == 'true'
    runs-on: ubuntu-latest
    timeout-minutes: 30 # mins. Default is 360 mins.
    needs:
      - changes
    permissions:
      contents: read # To checkout
      id-token: write # To authenticate with GCP using OIDC
      pull-requests: write # To comment on PRs
    defaults:
      run:
        working-directory: ${{ matrix.working-directory }}
    environment: ${{ matrix.working-directory }}-plan
    strategy:
      fail-fast: false
      matrix:
        working-directory: ${{ fromJson(needs.changes.outputs.terraform-modified-files) }}
    steps:
      - uses: actions/checkout@v4

      - uses: google-github-actions/auth@v2
        with:
          service_account: ${{ secrets.GOOGLE_CLOUD_SERVICE_ACCOUNT_EMAIL }}
          workload_identity_provider: ${{ secrets.GOOGLE_CLOUD_WORKLOAD_IDENTITY_PROVIDER }}
      
      - run: tfcmt -var "target:${{ matrix.working-directory }}" plan -- terraform plan

Terraform plan results are posted as comments, allowing quick review of changes.

After the PR is merged into the main branch, a similar workflow runs terraform apply.

We use the tj-actions/changed-files GitHub Action to detect which directories (environments) have changes. Combined with GitHub Environments, this setup ensures Terraform executes with the appropriate credentials for each environment (e.g., Google Cloud service accounts).

This approach optimizes GitHub Actions usage and reduces waiting time while isolating credentials per environment. It also allows us to support virtually unlimited environments by simply adding directories without modifying YAML (workflow) files—highly flexible and efficient.

Issue: Module Changes Not Applied Automatically

As mentioned earlier, we heavily use Terraform Modules with relative paths in the source. However, GitHub Actions only detects changes in specific environments and doesn’t trigger Terraform when only a Module is updated, leaving dependent environments unchanged.

├─ modules
│  ├─ reearth <- Changes here...
│  └─ ...
└─ terraform
   ├─ cloud
   │  └─ dev
   │     └─ reearth.tf <- ...won’t trigger Terraform execution here
   └─ ...

This creates a problem for developers who want to modify a Module and apply it to the development environment within a single PR. Instead, they must create two separate PRs—one for modifying the Module and another for applying it. This is cumbersome, especially when debugging or fixing bugs, as it requires repeating the two-step process.

Moreover, since we use relative paths, developers are forced to make dummy changes like adding blank lines to trigger GitHub Actions.

module "reearth" {
     source = "../../modules/reearth"
     ....
}
+ # Forcing a change to trigger GitHub Actions...

To avoid this, we needed a mechanism to propagate Module changes to dependent environments.

Defining Dependencies as Side Effects

To handle Module changes as "side effects," we implemented the GitHub Action KeisukeYamashita/side-effect-changed-files.

This action allows us to define which environments (directories) depend on which Modules via an external YAML file or inline configuration.

- uses: tj-actions/changed-files@v44
  id: raw-changed-files
  files: **/*.{hcl,tf,yaml}

- name: Get changed files
  id: changed-files
  uses: KeisukeYamashita/side-effect-changed-files@v1
  with: 
    bypass: terraform/**/*.{hcl,tf,yaml} # Each environment is changed as usual and nothing is done.
    dir_names: true # Grouping by directory
    files: ${{ steps.raw-changed-files.outputs.files }}
    mapping: |
      # If there are changes in reearth module,
      # Terraform will be executed in cloud and plateau development environments
      terraform/{cloud,plateau}/dev/*:
        - modules/reearth/*.tf
     matrix: true

This configuration treats changes in the reearth Module as if changes occurred in the cloud and plateau environments, triggering Terraform execution.

.
├─ modules
│  ├─ reearth <- If there are changes...
│  └─ reearth-cms
└─ terraform
   ├─ cloud
   │  └─ dev <- Triggers Terraform execution here as a side effect
   │     ├─ reearth.tf
   │     └─ reearth_cms.tf
   ├─ plateau
   │  └─ dev <- Also triggers Terraform execution here
   │     ├─ reearth.tf
   │     └─ reearth_cms.tf
   └─ ...

Although maintaining these mappings requires effort, the frequency of adding or removing Modules or environments is low, so the overhead is minimal. If a mapping is missed, Terraform simply doesn’t run, making it a safe and noticeable issue for developers.

This setup enables developers to modify Modules and apply changes to the development environment within a single PR, preventing configuration drift and adhering to GitOps principles (e.g., the main branch reflects the latest resource state). Additionally, developers receive quicker feedback on plan failures, improving the development cycle.

Conclusion

In this article, we introduced how to manage Terraform utilizing Monorepo and GitHub Actions. In particular, we implemented a mechanism to effectively reflect Module changes to each environment, thereby improving development efficiency and preventing configuration drifts. This initiative has enabled more secure and efficient infrastructure management, leading to increased productivity of the development team. We will continue to make ongoing improvements to achieve better DevOps practices.