Gitlab-runner ssh executor. ssh: handshake failed: knownhosts: key mismatch

GitLab CI/CD
, ,
1

I’m trying to use gitlab-runner to deploy my site on digital ocean with the ssh executor.

In my droplet I have these users:

  • root: account with log in disabled only accesed by web console.
  • deploy: user account that actually runs the application.
  • gitlab-runner: created with gitlab’s setup script.

This is the content of /home/gitlab-runner/.gitlab-runner/config.toml :

concurrent = 1
check_interval = 0
log_level = "debug"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "mpm-staging"
  url = "https://gitlab.com"
  id = 27295584
  token = "redacted"
  token_obtained_at = 2023-08-28T17:25:05Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "ssh"
  #known_hosts_file = "/home/gitlab-runner/.ssh/known_hosts"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.ssh]
    user = "deploy"
    password = "redacted"
    host = "redacted.com"
    port = "22"
    identity_file = "/home/gitlab-runner/.ssh/id_rsa"
    disable_strict_host_key_checking = true

The gitlab-runner use can login successfully as deploy@redacted.com.

At gitlab.com CI console I see the same output than that of running gitlab-runner run locally on my droplet.
I’m at a loss here. I don’t know how to debug this. I’ve used this setup in the past without trouble.

2

This config is commented out. Is that intentional?

If it is actually used, suggest editing and verifying that all SSH key entries match the relevant IP/Hosts. Sometimes, domains/IPs change, thus leading to errors. Could also be that someone malicious plays man-in-the-middle, pretending to be the target host with faked DNS resolval.

3

It would be interesting to see job output and where exactly you get that error. GitLab Runner running on the same machine where you deploy should not SSH to any other host. I suppose you just copy files around?

4

By default it comes without that section. I added it then commented it out. The effect is the same.
I understand the security implications, I assume it’s using the default now.

The complete log is:

Running with gitlab-runner 16.2.0 (782e15da)
  on mpm-sapp-01 PMGVbwn7z, system ID: s_bd8db0f11468
Preparing the "ssh" executor
Using SSH executor...
ERROR: Preparation failed: ssh command Connect() error: ssh Dial() error: ssh: handshake failed: knownhosts: key mismatch
Will be retried in 3s ...
Using SSH executor...
ERROR: Preparation failed: ssh command Connect() error: ssh Dial() error: ssh: handshake failed: knownhosts: key mismatch
Will be retried in 3s ...
Using SSH executor...
ERROR: Preparation failed: ssh command Connect() error: ssh Dial() error: ssh: handshake failed: knownhosts: key mismatch
Will be retried in 3s ...
ERROR: Job failed (system failure): ssh command Connect() error: ssh Dial() error: ssh: handshake failed: knownhosts: key mismatch
5

Can you share the GitLab CI/CD configuration that matches the shown error log?

The docs mention that the SSH executor does not support all features, and recommend other types. Maybe the config unveils a problem.

An alternative approach could be using the shell executor, and connecting to the remote host inside the CI/CD config. SSH keys etc. can be passed as variables. Using SSH keys with GitLab CI/CD | GitLab This is a common scenario on GitLab.com SaaS shared runners where users cannot modify the executor type.

6

Ah, I looked at the config.toml again and you are SSHing to a remote host. Well, then it’s fairly easy. I’d say the Host key for the server changed (for whatever reason, double check this! Just to be sure there is no bad guys doing bad things). So you need to fix that in SSH’s known_hosts file. Delete the incorrect line and run ssh-keyscan remote_host to save the new key.

1 Like