Having trouble integrating CD (Continuous Delivery) with docker + ssh

I’m having trouble integrating CD (Continuous Delivery) in my GitLab v16.2.2-ee.

I’ve followed the steps and made several attempts to use it, but it always fails in one way or another.

$ eval $(ssh-agent -s)
Agent pid 3062
$ echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
Identity added: (stdin) (root@gitlab.mydomain.tld)
$ mkdir -p ~/.ssh
$ chmod 700 ~/.ssh
$ ssh-keyscan central.mydomain.tld >> ~/.ssh/known_hosts
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

Procedures

Generate the SSH key

Following the steps from Use SSH keys to communicate with GitLab, I created my SSH key to use on the server where I want to perform the deployment.

root@gitlab:~# ssh-keygen -t ed25519 -C "root@gitlab.mydomain.tld"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/root/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_ed25519
Your public key has been saved in /root/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:YB7wROaejtxCmrue8d6e+4h5Z6/vjGBrqBEwb5E17pE root@gitlab.mydomain.tld
The key's randomart image is:
+--[ED25519 256]--+
|    +.+          |
|   + O           |
|o o E *          |
| + o = +         |
|  + o + S        |
| . * +           |
|  = +.=          |
|   *.*o=oo       |
| .*+=+B=+==      |
+----[SHA256]-----+

Variables

In the project > Settings > CI/CD > Variables, I created the variable SSH_PRIVATE_KEY and entered the private key in the Value field.

I also created the variables LIVE_SERVER_IP and LIVE_SERVER_USERNAME.

As another attempt, I also created SSH_KNOWN_HOSTS by following the steps.

However, to simplify and intensify debugging, in my .gitlab-ci.yml, I changed the variables to real values.

GitLab Runner on the GitLab server

gitlab-runner --version
Version:      16.2.0
Git revision: 782e15da
Git branch:   16-2-stable
GO version:   go1.20.5
Built:        2023-07-21T22:52:35+0000
OS/Arch:      linux/amd64

.gitlab-ci.yml

Obtained from the official GitLab example for CD with SSH.

image: ubuntu

before_script:
  ##
  ## Install ssh-agent if not already installed, it is required by Docker.
  ## (change apt-get to yum if you use an RPM-based image)
  ##
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'

  ##
  ## Run ssh-agent (inside the build environment)
  ##
  - eval $(ssh-agent -s)

  ##
  ## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
  ## We're using tr to fix line endings which makes ed25519 keys work
  ## without extra base64 encoding.
  ## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556
  ##
  - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -

  ##
  ## Create the SSH directory and give it the right permissions
  ##
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh

  ##
  ## Use ssh-keyscan to scan the keys of your private server. Replace gitlab.com
  ## with your own domain name. You can copy and repeat that command if you have
  ## more than one server to connect to.
  ##
  - ssh-keyscan central.mydomain.tld >> ~/.ssh/known_hosts
  - chmod 644 ~/.ssh/known_hosts

  ##
  ## Alternatively, assuming you created the SSH_SERVER_HOSTKEYS variable
  ## previously, uncomment the following two lines instead.
  ##
  ##- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
  ##- chmod 644 ~/.ssh/known_hosts

  ##
  ## You can optionally disable host key checking. Be aware that by adding that
  ## you are suspectible to man-in-the-middle attacks.
  ## WARNING: Use this only with the Docker executor, if you use it with shell
  ## you will overwrite your user's SSH config.
  ##
  #- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

  ##
  ## Optionally, if you will be using any Git commands, set the user name and
  ## email.
  ##
  - git config --global user.email "abelkarim.mateos@mydomain.tld"
  - git config --global user.name "Abkrim"

deploy_production:
  stage: deploy
  script:
    - ssh -p 2244 user@176.1.1.228 "cd /home/user/web/user.deploy.domain/user && git pull && yarn install && yarn prod"
  environment: production

GitLab Runner

root@gitlab:~# cat  /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
log_level = "debug"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "user"
  url = "https://gitlab.mydomain.tld"
  id = 27
  token = "glrt-EfM

nd_473N6n48rMUw-i"
  token_obtained_at = 2023-08-02T05:52:25Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "ubuntu:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Failures

No matter what variations I try, it always fails somewhere, and I can’t figure out how to debug this issue.

Running hooks in /etc/ca-certificates/update.d...
done.
$ eval $(ssh-agent -s)
Agent pid 3063
$ echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
Identity added: (stdin) (root@gitlab.mydomain.tld)
$ mkdir -p ~/.ssh
$ chmod 700 ~/.ssh
$ ssh-keyscan central.mydomain.tld >> ~/.ssh/known_hosts
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

On the log there is nothing about debugging of GitLab-runner.

Hello,

I hope you have solved your problem since then, but I have the same problem. How did you solve this?
Do you have any leads?