Optimal configuration for Docker, GitLab Runner, and GitLab CI

I just came back from a few days of vacation and apparently our CI/CD has not been working (it used to work pretty well).

The first error message I got was this:

ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (executor_docker.go:968:0s)

I was able to fix this by running sudo systemctl start docker.

However our CI/CD still doesn’t work - I now get this error:

Fetching changes…
remote: You are not allowed to download code from this project.
fatal: unable to access ‘http://gitlab-ci-token:[MASKED]@domain.com/group/project.git/’: The requested URL returned error: 403

So I was investigating this one and it turns out that the registration token has apparently changed without us having to manually reset it?

I’m looking at a few documentations and I can’t find what I’m looking for. I need to update the registration token that was registered on our GitLab Runner - but from the documentation, it seems like I have to unregister the existing one and register another.

In the end, I have a few questions so that stuff like this never happens again.

  • How do I make sure that the docker service is always running even after the server restarts (I’m assuming our server rebooted that’s why docker wasn’t running from the first error)?
    • I saw the command docker run --restart=always <container> but I’m not sure if that does what I think it does
  • How do I just update the registration token without having to unregister a GitLab Runner?
  • Has there been any reported cases where the registration token just changed by itself?

I really hope you guys can help me out here.

For my first question I guess I can just run (taken from https://askubuntu.com/questions/733469/what-is-the-difference-between-systemctl-start-and-systemctl-enable)

sudo systemctl enable docker

I still can’t find an answer to my other 2 questions though.