Reset Terraform state when using GitLab as backend?

I decided to try using Gitlab as a backend for Terraform. For one reason or another, I needed to manually remove something from my state because it was breaking my CI pipeline. So from my command line, I run terraform state rm the_resource.my_resource.

Problem: I am running Terraform 0.12.29 locally, but the latest Terraform container image for GitLab CI is Terraform 0.12.28. (I could try to use someone else’s container image, but then the tail is wagging the dog–the point was to make things simpler!) Anyway, after I do that, the pipeline begins failing with a much more problematic message:

Error: Error loading state: state snapshot was created by Terraform v0.12.29, which is newer than current v0.12.28; upgrade to Terraform v0.12.29 or greater to work with this state

Well, I can’t do that! So I decide, “whatevs, let me just delete the state file.” And then I realize I have no idea how to do that, either, since I have only interacted with it via API. The only thing I can think of, aside from giving up on the GitLab Terraform backend, is deleting the repository and starting over. That is not a palatable option.

Is there another way to reset the backend state?

This isn’t anything to do with Gitlab, so you may get better answers from Terraform folks elsewhere.

However, I have some experience, and as far as I know, once a state file has been used with a later version, it can’t be “downgraded” to a lower one - the reason being that a later version may change the format of it, and so older versions would not be able to read them.

That said, you may be able to hack this because it’s “only” a single dot release difference. You should make sure you’ve got lots of good backups of everything before you start though.

The general process would be:

  • terraform pull the state from wherever it’s stored.
  • Edit the state file in a text editor and change the place(s) where the Terraform binary version is specified
  • terraform push the state back to wherever it is stored normally
  • (possibly delete your .terraform directory?)
  • terraform init and terraform apply again

Please note: none of this is “supported”, so you’re very much on your own with it. It’s probably easier to upgrade the Terraform binary you’re using.

It actually is a GitLab issue. Sorry if I wasn’t clear in my original post; I see now that I spent too much time on background and not enough on the fundamental issue.

GitLab provides a REST API to manipulate a set of Terraform state files that reside on its servers. According to the repo for the GitLab Terraform image, the URL for this API takes the form of:

{GITLAB_API_URL}/projects/{PROJECT_ID}/terraform/state/{ANY_STRING_YOU_WANT}

So a trivial solution, which I ended up using, was to change {ANY_STRING_YOU_WANT} to something new. But there is no obvious way to delete previously created state via this API. Presumably, the API is manipulating a data store on GitLab somewhere, but I don’t know where that data store is or how to alter it.

So the fundamental question is one of the following:
(1) Is there a REST command that I can send to the above API that would reset the state?
(2) Alternatively, is there some way to directly access the underlying data store so I can delete or alter what is stored there?

Again, if it’s a Terraform backend, you can use terraform pull and terraform push to get the data (and to put it back again, in edited form).

I can’t immediately see a way to delete the state entirely, but I guess if you just empty it out that would achieve much the same thing. Maybe there’s a UI way to see the files you have there which has a delete capability? (or perhaps there will be in the future).

1 Like

Didn’t think of that! Thank you.

1 Like

I haven’t found a documentation for this, however you can actually just delete the state via a curl:

curl -X DELETE -u username:password "{GITLAB_API_URL}/projects/{PROJECT_ID}/terraform/state/{ANY_STRING_YOU_WANT}"