Unless you paid extra attention when creating a new feed in ProGet 2024.20+, you probably didn’t notice the new Terraform Modules Feed Type. And if you’re like most of our users, you don’t use Terraform so you really don’t care. But, you might find the technical challenges and solutions involved in building a new feed type to be interesting.

In theory, building a new feed type should just be a matter of “implementing the documented specs” for the server API and package format. But in reality, “accurate specs” aren’t actually a thing and the server/package model often doesn’t align with ProGet’s feed model.

This was especially the case with Terraform Modules, and in this article I’ll give a little background on Terraform itself and talk about the some behind-the-scenes engineering that went into building the new feed type. Like all of our feed types, we developed Terraform Feeds based on user conversations starting in the forums, so if you’d like us to develop a new feed type, just let us know!

What is Terraform?

According to Hashicorp (the creator), Terraform helps you automate infrastructure by “codifying cloud APIs into declarative configuration files.” A Terraform Module lets you reuse these configuration files.

Beyond that, I’m not entirely sure how Terraform is used. Like many feed types in ProGet, we don’t actually use Terraform ourselves, so we have to rely on research and what our users say. From this, my general impression was that Terraform is a kind of scripting language to build application environments with AWS, Azure, and GCP resources.

But after developing this feed, I got the impression that Terraform has some alternative use cases involving “alternative” (non-cloud) providers. There’s also a forked project (OpenTofu) that seems to be fully API-compatible. I’m not sure how popular these alternatives are, but I think ProGet’s Terraform feeds will also work fine to support them.

Three Challenges in Building a Private Terraform Module Registry

Terraform’s module registries were designed with registry.terraform.io as the “source of truth” for free and open-source Terraform content.

Private registry support has since been added, but the registry model is still heavily influenced by the original design. This made building a feed in ProGet a bit more challenging than other ecosystems.

Challenge 1: Git-based Modules

From the registry’s perspective, a “module” is essentially a Git repository and a “module version” is a tag or commit in that repository. For example, when you specify terraform-aws-modules/vpc/aws as your project’s source, the Terraform client simply clones that module’s GitHub repository and uses a specific tag from it.

While this makes things simpler for publishing content to Hashicorp’s free and open-source registry, it’s not a good fit for organizations that rely on reliable and repeatable processes. For example, the GitHub repository could be deleted or changed, meaning the module will suddenly no longer be available.

In addition, when it comes to private/internal modules, the Git-based approach is more challenging to administer. It’s also not a great fit for artifact-driven CI/CD pipelines. But most importantly, ProGet is a package-based system and our users want a package-based approach.

Challenge 2: Hardwired to Hashicorp’s Registry

The Terraform client is “hardwired” to work with registry.terraform.io as the sole and single source of truth. Unlike most package management systems, there is no concept of a “package source”. This means you can’t add, modify, delete, or otherwise specify multiple registry URLs for the Terraform client. It’s always meant to be connected to Hashicorp’s server.

Fortunately, the Terraform client does have a concept of “external modules”. When you prefix a module’s name with a hostname (e.g. terraform-registry.corp.local/my-corp/my-module/aws), the Terraform client will connect to terraform-registry.corp.local instead of Hashicorp’s server.

This feels like a poor design decision on Hashicorp’s part, but it’s also how Docker repositories work. So, users are somewhat accustomed to it. In either case, that’s just how Terraform works.

Challenge 3: One Registry Per Domain

When you specify an external module, the Terraform client assumes that there is only a single private module registry on that server. This is a poor assumption on their part because of multi-feed / multi-registry tools like ProGet.

The Docker client makes the same assumption but allows you to specify multiple/nested namespaces. For example, when you specify proget.local/my-feed/my-groupmy-sub-group/my-app as your container image, ProGet will use the first namespace segment as the feed.

Unfortunately, Terraform modules have a much more restrictive naming convention. They must always be formatted as «namespace»/«name»/«provider» – and no special characters are allowed. This makes it challenging to add a feed name.

ProGet’s Terraform Module Solution

Overcoming Terraform’s single-registry problem was fairly simple. Even if you’ve never used Terraform before, you can probably figure out how we addressed it:

module "example_module" {
  source = "proget.corp/internal-terraform__my-company/my-module/aws"
  version = "4.20.0"
}

It feels a bit hacky, but as long as you prefix modules with «proget-host-name»/«feed-name»__, then ProGet can process module requests for the right feed. The real challenge was dealing with the Git-based Modules.

Terraform Module Packages in ProGet

In the Terraform ecosystem, a Terraform Module is simply a collection of related .tf files in a folder. They’re typically stored in a GitHub repository, but they don’t have to be. Versions of these files are handled using Git’s tagging mechanism or with creative folder naming conventions.

A package, on the other hand, is an archive file (e.g. .zip). In addition to the “packaged” content files, it also contains a manifest file that provides the package’s name, version, author, and other key metadata. Packages are portable and independent from external resources, and most importantly, cryptographically sealed with a hash.

There’s a reason nearly all package management systems use package files instead of random files in folders. Regardless, to get Terraform Modules working in a feed, they need to be packaged. This is where ProGet’s Universal Packages come in.

The Creating Terraform Module Packages documentation provides more details, but it basically involves using the upack CLI to create a package file with a manifest similar to this:

{
  "group": "my-company",
  "name": "my-module.aws",
  "version": "4.20.0"
}

Once this Terraform Module Package is in a Terraform feed, the Terraform client can access it just like any other private module registry.

Automatically Converting Git-based Modules

In the Terraform ecosystem, a Terraform Module Registry basically just provides the Terraform client with a “pointer” to download content. While there are technically HTTP archive pointers, most private registries (including Hashicorp’s official registry) only provide pointers to Git Repositories at GitHub.com.

When you connect a Terraform feed to another module registry, ProGet needs to decode and follow those pointers to download and package the module’s content. Unlike the Terraform CLI, ProGet does not use the Git client to download the module files from GitHub.

Instead, ProGet uses a rudimentary string replacement to download the repository from GitHub’s archive endpoint. It feels hacky, but it seems to work.

For example, the Hashicorp registry responds with this pointer when requesting a module:

X-Terraform-Get: git::https://github.com/sourcegraph/terraform-aws-executors?ref=d9d6b1db18013b7dbd01edce457980c58e7a1c90

That’s not technically a valid GitHub.com URL; it’s just what a Git-based “pointer” looks like. When ProGet sees a pointer that starts with git::https://github.com, it will be transformed into a usable GitHub URL:

https://github.com/sourcegraph/terraform-aws-executors/archive/d9d6b1db18013b7dbd01edce457980c58e7a1c90.zip

With that, ProGet can download the contents and automatically package it in the feed. We could certainly add support for other Git-based registries, but everything seems to just use that.

If you run into any issues with specific modules from the Hashicorp registry or another one, please let us know so we can research further.

Feedback is Always Welcome

Terraform Feeds were an interesting technical challenge and I think we came up with an interesting technical solution. But was this article interesting?

Let me know what you think! We mostly write code and/or documentation that makes our products better and easier to use, but sometimes it’s interesting to take a break and share the behind-the-scenes engineering efforts over here.