Terraform state corruption?

How to Use GitLab Self-managed
1

Problem to solve

I’m having issue with Terraform state on self hosted Gitlab. I think it’s corrupted.

varun@qa-docker-1:~/git/ansible-terraform/terraform/ovh$ TF_LOG=DEBUG terraform plan
2025-04-24T16:02:50.794-0400 [INFO]  Terraform version: 1.11.3
2025-04-24T16:02:50.794-0400 [DEBUG] using github.com/hashicorp/go-tfe v1.70.0
2025-04-24T16:02:50.794-0400 [DEBUG] using github.com/hashicorp/hcl/v2 v2.23.0
2025-04-24T16:02:50.794-0400 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2025-04-24T16:02:50.794-0400 [DEBUG] using github.com/zclconf/go-cty v1.16.0
2025-04-24T16:02:50.794-0400 [INFO]  Go runtime version: go1.23.3
2025-04-24T16:02:50.794-0400 [INFO]  CLI args: []string{"terraform", "plan"}
2025-04-24T16:02:50.794-0400 [DEBUG] Attempting to open CLI config file: /home/varun/.terraformrc
2025-04-24T16:02:50.794-0400 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2025-04-24T16:02:50.794-0400 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2025-04-24T16:02:50.794-0400 [DEBUG] ignoring non-existing provider search directory /home/varun/.terraform.d/plugins
2025-04-24T16:02:50.794-0400 [DEBUG] ignoring non-existing provider search directory /home/varun/.local/share/terraform/plugins
2025-04-24T16:02:50.794-0400 [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2025-04-24T16:02:50.794-0400 [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2025-04-24T16:02:50.794-0400 [DEBUG] ignoring non-existing provider search directory /var/lib/snapd/desktop/terraform/plugins
2025-04-24T16:02:50.796-0400 [INFO]  CLI command args: []string{"plan"}
2025-04-24T16:02:50.847-0400 [DEBUG] checking for provisioner in "."
2025-04-24T16:02:50.851-0400 [DEBUG] checking for provisioner in "/usr/bin"
2025-04-24T16:02:50.852-0400 [INFO]  backend/local: starting Plan operation
2025-04-24T16:02:50.852-0400 [DEBUG] POST https://gitlab.transnomis.com/api/v4/projects/17/terraform/state/ovh/lock
2025-04-24T16:02:51.045-0400 [DEBUG] GET https://gitlab.transnomis.com/api/v4/projects/17/terraform/state/ovh
2025-04-24T16:02:51.157-0400 [DEBUG] DELETE https://gitlab.transnomis.com/api/v4/projects/17/terraform/state/ovh/lock
╷
│ Error: error loading state: Failed to read remote state: stream error: stream ID 3; INTERNAL_ERROR; received from peer

Issue is only with one single Terraform state. I had the issue come up after a Ceph issue so it could be corruption. It was working on/off and now it completely stopped working.

Requesting steps to resolve issue or try to download Terraform state file and migrate it to another state hosted on Gitlab.

Steps to reproduce

Run terraform plan or terraform apply.

Configuration

Provide screenshots from the GitLab UI showing relevant configuration, if applicable.
On self-managed instances, add the relevant configuration settings or changes.

image
image1429×558 34.7 KB

Versions

Please add an x whether options apply, and add the version information.

  • [x ] Self-managed
  • GitLab.com SaaS
  • Dedicated

Versions

  • GitLab (Web: /help or self-managed system information sudo gitlab-rake gitlab:env:info):
gitlab-rake gitlab:env:info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   3.2.5
Gem Version:    3.6.6
Bundler Version:2.6.5
Rake Version:   13.0.6
Redis Version:  7.2.7
Sidekiq Version:7.3.9
Go Version:     unknown

GitLab information
Version:        17.11.1
Revision:       78fb986046b
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     16.8
URL:            https://gitlab.transnomis.com
HTTP Clone URL: https://gitlab.transnomis.com/some-group/some-project.git
SSH Clone URL:  ssh://git@gitlab.transnomis.com:2222/some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: saml

GitLab Shell
Version:        14.41.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      17.11.1
- default Git Version:  2.49.0.gl1

Helpful resources

  1. Check the FAQ for helpful documentation, issues/bugs/feature proposals, and troubleshooting tips.
  2. Before opening a new topic, make sure to search for keywords in the forum search
  3. Check the GitLab project for existing issues. If you encounter a bug, please create a bug report issue.
  4. Review existing troubleshooting docs.

Thanks for taking the time to be thorough in your request, it really helps! :blush:

2

I tried migrating state to another state name once it was working. But it stopped working again with same error. Wondering if I can dump the state file and manually upload it to a new name.

3

Tried dumping TF state file but got gibberish instead. Can someone explain me what I’m doing wrong?

gitlab-rails console
irb(main):001:0> project = Project.find_by_full_path('infra/ansible-terraform') 
=> #<Project id:17 infra/ansible-terraform>>
irb(main):002:0> project.terraform_states.map(&:name)
=> ["proxmox-new", "proxmox-ovh", "zerotier", "godaddy", "ovh_infra"]
irb(main):003:0> state = project.terraform_states.fifth
=> 
#<Terraform::State:0x00007bf8cdf40320
...

File.open("/tmp/terraform.tfstate3", "wb") do |f|
  f.write(state.latest_file.open.read)
end