How to debug SSH issues

Similar question there: Problem configuring Gitlab CI -> ssh: handshake failed

Here is debug process for this situation.

Verify you can logging to remote ssh server manually using same local user and same key file than gitlab-runner

sudo su - USER

eval `ssh-agent -s`
ssh-add YOUR_SSH_KEY_FILE
ssh REMOTEU_SER@REMOTE_SERVER

You should solve any issue here before continue.

Run gitlab-runner with log in debug mode

sudo vi /etc/gitlab-runner/config.toml

Add log_level = "debug"

# Sample
concurrent = 1
check_interval = 0
log_level = "debug"
...

Then restart gitlab-runner

sudo gitlab-runner restart

Retry to run buggy job.

If it does not help go to next step.

Launch remote sshd in debug mode

Run SSH daemon on another port as a part of the command (then you don’t need to lost your existing session, if already connect thought ssh)

sudo /usr/sbin/sshd -d -p 2200
# sudo /usr/sbin/sshd -dd -p 2200 - if you need more logs
# sudo /usr/sbin/sshd -ddd -p 2200 - if you need all logs

Warn: You should run this for each ssh connection.

Then modify gitlab-runner configuration (on gitlab runner host)

sudo vi /etc/gitlab-runner/config.toml

Add port = "2200"

...
[[runners]]
  name = "XXXXX"
  url = "https://XXXXX/"
  token = "XXXXX"
  executor = "ssh"
  [runners.ssh]
    user = "XXXXX"
    host = "XXXXX"
    port = "2200"
    identity_file = "XXXXX"
...

Retry to run buggy job.

On console sshd console you are able to see all gitlab-runner actions and you should be able to understand what is the issue.

1 Like

Thanks so much @cClaude for putting this how to together. I’ve now moved it to the tutorials section and pinned it, so that it’s easy to reference and other community members can benefit from it.

Great work! :slight_smile: :clap: