Debugging CI by ssh-ing into the running container

I want to debug a CI job from inside the server it’s running on.

This is a feature of other CI/CD systems, and I’m having trouble telling whether its supported on gitlab. Basically, I want to run a specific job, then before it finishes, suspend it, and then ssh into the container and “look around.”

This is a private gitlab instance.

I’m looking at the documentation here: https://docs.gitlab.com/ee/ci/ssh_keys/#ssh-keys-when-using-the-docker-executor

I’ve done all the steps described, the part I’m unsure of is how to target the container. (I’m also not 100% sure that the documentation linked is what I’m trying to do). In short:

  1. Create a new public/private key on my computer
  2. Add the private key as a variable to the project
  3. install the ssh-agent, and use the following commands in the before_script:
    - 'command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )'
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
  1. profit?

Also, aren’t these steps backward?

- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh

Should I be doing ssh https://gitlab.mycooldomain.com/bsn/mycoolproject/-/pipelines/82343?

Hey there claytron,

The guide you are following may not be doing exactly what you are expecting.

Based on the beginning:

I want to debug a CI job from inside the server it’s running on.

I’m assuming you have created some server that is a registered gitlab-runner as a docker executor, and you want to actually SSH onto that gitlab-runner instance while the CI job is running to look around the job container.

The guide above is an explanation of how to deliver code during a CD step to production servers by SSHing to them.

Just a few thoughts:

  1. I’m not sure how well SSHing from inside a docker container to the host that container is running on will work. If you’ve tried it and it works, that’s good. Although I would think you’d likely end up in an odd loop since you’re trying to look around the current container you’re running in by SSHing and then a docker exec ing into that same container.

  2. I believe the order of those commands are correct. I believe you don’t need the ~/.ssh/ directory to exist for ssh agent and you’re not adding an ssh key into a file. That directory I believe is more for the known_hosts file.

Suggestions:
To preface all of this, I believe gitlab runner on the docker executor will name containers based off of their project id. We mostly use kubernetes as our runners and I know pods are named as such. That could be something to look for. If not I recommend simply watching a list of running docker containers on your gitlab runner and see what changes when you startup a new build

  1. Manually investigate your containers. In your gitlab-ci.yml, add a sleep 1800 to allow you to ssh into the executor manually and try to find the container and look around

  2. Register a new shell executor on that gitlab runner and run a parallel job (in the same stage) that simply monitors the running docker containers and can exec into the correct pod. This option will require you to investigate how docker containers are named and correctly identify the right container.

Hope this helps some!

Cheers,
Thomas