Terraform, Terraform Cloud and Terraform Registry, and AWS multi-regional deployments (Public + China)
Sharing my experiences and knowledge around Terraform, Terraform Cloud and Terraform Registry while working with AWS deployments into Ireland, North Virginia and China.
Introduction
A little bit about me first, I am a Lead DevOps Engineer specialising in Cloud Native solutions. I primarily work with Kubernetes, however I am not against using solutions such as AWS ECS or AWS Fargate. I am a big fan of solutions offered by AWS cloud provider and I am also a certified solutions architect.
This story will focus on my experience around Terraform, Terraform Cloud and Terraform Registry, and well it works with AWS deployments. I will write about my recent achievements and work I recently conducted as part of my current professional work.
If you decide you want to know more about me, you can read about my experience -> https://www.linkedin.com/in/marcincuber
AWS multi-regional deployments
AWS and GCP are the only cloud platforms I worked with, however AWS is my home. I have been using it non-stop for past five years.
Until this year, I only had a pleasure to work across regions in public partition of AWS. AWS public partition includes regions such as Ireland, Frankfurt, London, North Virginia and Oregon. More about regions can be found in official AWS documentation https://aws.amazon.com/about-aws/global-infrastructure/regions_az/
In my latest challenge I managed to work on setting AWS China, a challenging partition to work with. In order to deploy to China, I used Terraform and most of the things worked fine. However, it wasn’t a smooth process.
Terraform
In general, Terraform AWS provider isn’t yet fully supporting AWS China. For example, VPC Endpoints did cause some issues which meant that I couldn’t use the official VPC module. In simple terms, Terraform doesn’t source correctly the endpoints which are required to have an AWS partition mentioned. So instead of aws commercial partition, there should be a Chinese one, which isaws-cn. Also, there is change in service definition where the endpoint has to use .com.cn instead of .com. I have raised a Github issue to get this resolved and you can find it here -> https://github.com/hashicorp/terraform-provider-aws/issues/17640
Terraform Registry
While deploying things into China, additional problem I faced was Terraform Registry itself. In order to deploy Terraform modules to China, you require a pipeline which is also based in AWS China. This is necessary in order to assume a correct IAM Role which will have relevant permissions to deploy everything. This is again related to the fact that you are no longer using aws partition but rather aws-cn. So here is a surprise for you, using Terraform Registry modules don’t work from China pipeline. To clarify this further here is an example:
# Reference that doesn't work and will fail to download
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "2.77.0"
...
}# Working reference
module "vpc" {
source = "git@github.com:terraform-aws-modules/terraform-aws-vpc.git?ref=v2.77.0"
...
}
It is a shame that this is failing because of Terraform Registry implementation and this is based on the debug logs are saw. They are all related to the same problem which is logged out as “gnutls_handshake() failed: Error in the pull function”. I have raised the problem with Terraform Support but it doesn’t look like it is a priority at all. For a month I haven’t even received a single response to High Severity issue… Well, be aware to be careful with Terraform Registry as it is not the first time that it caused issues for me.
Terraform Cloud
I am using Terraform Cloud to store all terraform states in a place which is isolated from AWS accounts. To my surprise this is the only service that didn’t cause any issues. I have to say that it has proven to be very solid and stable. Perhaps there are not many releases to update it which makes a lot of sense. Note that I am a big fan of simplicity and the way Terraform Cloud is implemented. I am still using a free version of it as I don’t require any paid packaged. I was interested in upgrading to Terraform Business plan, however, after hearing that costs start at £40k a month to have audit logs, SAML integration and some runners, I decided to pass on this fantastic deal lol.
Conclusion
There are a lot of positives in using Terraform, Terraform Registry and Terraform Cloud when they work as expected. In case of deploying to AWS China, this is work in progress but I am confident that at some point everything will be working smoothly.
Project I worked on required me to implement AWS China account from scratch and I have to say that after a month I managed to accomplish everything as expected. This is definitely a step forward for me and for my current employer.
Last part of the project was to deploy all our APIs and make sure they all work in China. This has been a big challenge but I also managed to resolve all the issues and working configurations are deployed. One thing that is still outstanding is the Chinese ICP licence which still be progressed.
AWS China is missing some great services which are supported in a public partition but not in China. The major ones are:
- ALB Global Accelerator,
- ECS Insights
- WAFv2
- SES
It is absolutely crazy that some crucial services are not available. Especially when you consider the fact that ICP licence process requires you to deploy services with static public IPs. Static IPs have to be then provided to the government for approval and they are regularly checked. When running containerised applications behind ALB, the lack of Global Accelerator complicates things.
Finally, I am happy to share that I have implemented and deployed APIs which are running across three different regions (EU, US, China). They are all deployed using the same Terraform modules and the same pipeline is handling entire deployment from development to prod environments.
Contact
In case you require any support/consultation in a similar type project then please find me on LinkedIn. https://www.linkedin.com/in/marcincuber/
I can design, implement and deploy. ECS or Kubernetes, it doesn’t matter :). If you interested in other stories, please check out my marcincuber.medium.com
Sponsor Me
Like with any other story on Medium written by me, I performed the tasks documented. This is my own research and issues I have encountered.
Thanks for reading everybody. Marcin Cuber
