Gitlab-ci runner.ssh ssh: handshake failed: ssh: unable to authenticate,

Hello ,

Sorry for my approximative langage in my issue description.

I’m running a gitlab server (centos7.3) with CE Omnibus gitlab-ce-8.16.5-ce.0.el7.x86_64
and another machine also centos7.3 with gitlab-ci-multi-runner-1.11.4-1.x86_64

I want to execute a script test define in .gitlab-ci.yml on the machine running the gitlab-ci-multi-runner

I first setup a runner with the shell method and then a runner.ssh using a password in the config.toml
when i modify my project/repository its OK the test is PASSED

Then I generate an ssh key with the command : ssh-keygen -t rsa

If I try to setup the runner.ssh with the identity_file = “/home/user/.ssh/id_rsa” or /root/.ssh/id_rsa instead of password then the gitlab server return Failed with this message:

Running with gitlab-ci-multi-runner 1.11.4 (7e2b646)
on SpokeSSH (596bbba1)
Using SSH executor…
ERROR: Preparation failed: ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none password], no supported methods remain
Will be retried in 3s …
etc…

I have this type of message in /var/log/messages on the runner machine:

Jun 13 09:20:06 beurk gitlab-runner[4293]: time=“2017-06-13T09:20:06+02:00” level=error msg=“Job failed (system failure): ssh: handshake failed: ssh:
unable to authenticate, attempted methods [none password publickey], no supported methods remain” build=60 project=1 runner=596bbba1 #012

This is an exemple of config.toml

[[runners]]
name = “SpokeSSH”
url = “https://gitlab1.domain.top/ci
token = “596bbte1c20ed436tceffbe8b51ec74”
executor = “ssh”
[runners.ssh]
user = “root”
host = “gitlabci.domain.top”
port = “22”
identity_file = “/root/.ssh/id_rsa”
[runners.cache]

I try to manipulate various settings on the sshd_config file … but still hung with the handshake failed!

… Any Idea around to help? , thanks
plap

2 Likes

Similar question there: GitLab CI fails to perform SSH Handshake

Here is debug process for this situation.

Verify you can logging to remote ssh server manually using same local user and same key file than gitlab-runner

sudo su - USER

eval `ssh-agent -s`
ssh-add YOUR_SSH_KEY_FILE
ssh REMOTEU_SER@REMOTE_SERVER

You should solve any issue here before continue.

Run gitlab-runner with log in debug mode

sudo vi /etc/gitlab-runner/config.toml

Add log_level = "debug"

# Sample
concurrent = 1
check_interval = 0
log_level = "debug"
...

Then restart gitlab-runner

sudo gitlab-runner restart

Retry to run buggy job.

If it does not help go to next step.

Launch remote sshd in debug mode

Run SSH daemon on another port as a part of the command (then you don’t need to lost your existing session, if already connect thought ssh)

sudo /usr/sbin/sshd -d -p 2200
# sudo /usr/sbin/sshd -dd -p 2200 - if you need more logs
# sudo /usr/sbin/sshd -ddd -p 2200 - if you need all logs

Warn: You should run this for each ssh connection.

Then modify gitlab-runner configuration (on gitlab runner host)

sudo vi /etc/gitlab-runner/config.toml

Add port = "2200"

...
[[runners]]
  name = "XXXXX"
  url = "https://XXXXX/"
  token = "XXXXX"
  executor = "ssh"
  [runners.ssh]
    user = "XXXXX"
    host = "XXXXX"
    port = "2200"
    identity_file = "XXXXX"
...

Retry to run buggy job.

On console sshd console you are able to see all gitlab-runner actions and you should be able to understand what is the issue.

2 Likes

I have ran all these steps. And still get the same issue.

The CI log show the right name of my local runner, but fails:

Running with gitlab-runner 16.2.0 (782e15da) 
on mpm-staging Pe59iRx3Y, system ID: s_bd8db0f11468 
Preparing the "ssh" executor  
Using SSH executor...
ERROR: Preparation failed: ssh command Connect() error: ssh Dial() error: ssh: handshake failed: knownhosts: key mismatch

Using the debug ssh server shows nothing. Like it’s not attepting the connection. If I see the local runner log I see the same than in gitlab.com so I believe the issue relies in other config.