Using Passphrase in pipeline to rsync to external server

GitLab CI/CD
, , ,
1

Has anyone been able to export via rsync or ssh artifacts in a CI/CD pipeline using an SSH key with a passphrase to a remote host?

The pipeline logs the following failure:

$ rsync -rzvvvhWP --delete -e "ssh -i /root/.ssh/artifacts.remote.server_key -o HostKeyAlgorithms=ssh-rsa" ./public/ luser@artifacts.remote.server:/var/www/html
opening connection using: ssh -i /root/.ssh/artifacts.remote.server_key -o HostKeyAlgorithms=ssh-rsa -l luser artifacts.remote.server rsync --server -vvvWrze.iLsfxCIvu --delete --partial . "/var/www/html/"  (15 args)
luser@artifacts.remote.server: Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(231) [sender=3.2.7]
[sender] _exit_cleanup(code=12, file=io.c, line=231): about to call exit(12)
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

Here is the single online example I could find which did not work for me:

Here is how I have been forced to do it, and yet I am stuck with a strange host key negotiation standoff.

Using this example gitlab-ci.yml file and myriad incarnations I never am able to use the passphrase

# This GitLab CI configuration is designed to build and deploy a Hugo site.

# Define global variables
variables:
  HUGO_ENV: production
  THEME_URL: github.com/google/docsy@v0.7.1
  DART_SASS_VERSION: 1.63.6
  HUGO_VERSION: 0.118.2
  NODE_VERSION: 20.x
  GIT_DEPTH: 1
  GIT_STRATEGY: clone
  GIT_SUBMODULE_STRATEGY: recursive
  TZ: Europe/Vienna
  CI_DEBUG_TRACE: 'false'

# Define the Docker image to be used for the CI jobs
image:
  name: golang:1.21.1-bookworm

before_script:
  # Set up SSH for deployment
  - echo ${PRIVATE_KEY}
  - mkdir -p ~/.ssh
  - ssh-keyscan artifact.remote.server >> ~/.ssh/known_hosts
  - chmod 644 ~/.ssh/known_hosts
  - touch ~/.ssh/config
  - echo "Host artifact.remote.server" >> ~/.ssh/config
  - echo "  IdentityFile ~/.ssh/artifact.remote.server_key" >> ~/.ssh/config
  - echo "  IdentitiesOnly yes" >> ~/.ssh/config
  - chmod 600 ~/.ssh/config
  - export SSH_CONFIG=~/.ssh/config

  # Install necessary tools
  - apt-get update -y
  - apt-get install -y openssh-client rsync expect

  # Initialize the SSH agent
  - eval $(ssh-agent -s)

  # Diagnostic: Check the value of PRIVATE_KEY variable (first and last few characters for security)
  - echo "$PRIVATE_KEY" | head -n 1
  - echo "$PRIVATE_KEY" | tail -n 1

  - echo "$PRIVATE_KEY" | tr -d '\r' > ~/.ssh/artifact.remote.server_key
  - chmod 600 ~/.ssh/artifact.remote.server_key
  - if [ ! -f ~/.ssh/artifact.remote.server_key ]; then echo "Key file not found!"; exit 1; fi
  
  # Diagnostic: Display the key format
  - cat ~/.ssh/artifact.remote.server_key | head -n 1

  - if grep -- "-----BEGIN OPENSSH PRIVATE KEY-----" ~/.ssh/artifact.remote.server_key; then ssh-keygen -p -N "" -m pem -f ~/.ssh/artifact.remote.server_key; fi
  
  # Diagnostic: Check the key format after potential conversion
  - cat ~/.ssh/artifact.remote.server_key | head -n 1
  - ls -la ~/.ssh/
  - ssh-add /root/.ssh/artifact.remote.server_key
  - echo 'spawn ssh-add ~/.ssh/artifact.remote.server_key' > add_key.exp
  - echo 'expect "Enter passphrase for ~/.ssh/artifact.remote.server_key:"' >> add_key.exp
  - echo 'send "$env(SSH_PASSPHRASE)\r"' >> add_key.exp
  - echo 'expect eof' >> add_key.exp
  - chmod +x add_key.exp
  - ls -la ~/.ssh/
  - echo "$SSH_AGENT_PID"
  - ps -ef | grep $SSH_AGENT_PID
  - expect add_key.exp

  # Diagnostic: List the keys currently managed by ssh-agent
  - ssh-add -L

# Define the pages job
pages:
  script:
    # Install necessary tools for building the site
    - apt-get install -y brotli
    - curl -LJO https://github.com/sass/dart-sass/releases/download/${DART_SASS_VERSION}/dart-sass-${DART_SASS_VERSION}-linux-x64.tar.gz
    - tar -xf dart-sass-${DART_SASS_VERSION}-linux-x64.tar.gz
    - cp -r dart-sass/* /usr/local/bin
    - rm -rf dart-sass*
    - curl -LJO https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_linux-amd64.deb
    - apt-get install -y ./hugo_extended_${HUGO_VERSION}_linux-amd64.deb
    - rm hugo_extended_${HUGO_VERSION}_linux-amd64.deb
    - curl -fsSL https://deb.nodesource.com/setup_${NODE_VERSION} | bash -
    - apt-get install -y nodejs

    # Build the site
    - "[[ -f package-lock.json || -f npm-shrinkwrap.json ]] && npm ci || true"
    - hugo mod get $THEME_URL
    - export NODE_PATH=$NODE_PATH:`npm root -g`
    - hugo mod graph
    - npm install -g postcss postcss-cli autoprefixer
    - hugo --gc --minify

    # Compress the site's files
    - find public -type f -regex '.*\.\(css\|html\|js\|txt\|xml\)$' -exec gzip -f -k {} \;
    - find public -type f -regex '.*\.\(css\|html\|js\|txt\|xml\)$' -exec brotli -f -k {} \;

    # Deploy the site using rsync
    - rsync -rzvvvhWP --delete ./public/ luser@artifact.remote.server:~/public_html/customer2023/

  # Define the artifacts for the pages job
  artifacts:
    paths:
    - public/
  rules:
  - if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
    when: always

# Include the Auto-DevOps template
include:
  - template: Auto-DevOps.gitlab-ci.yml
I expect to see verbose rsync copy data to the remote server.

  • *This is GitLab.com system with ultimate features enabled using some automaticallfy assigned runner.

  • Troubleshooting steps taken:

I have added diagnostic steps to the pipeline to verify the CI/CD variables are found and being implemented as one would expect.

I have tried using these variables as file because that is the original instructions for gitlab CI/CD ssh variables. I have tried creating a similar local setup to verify the script steps work using these keys, and they do work sending data via rsync and passphrase with zero interaction.

I’ve updated the paths to be explicit and refined the expect script.

I am aware that Gitlab CI/CD SSH Keys are supposed to have a carriage return at the end. I have included logging to show that the first line does indeed contain data from the keys themselves.

I have tried myriad alternatives with CI/CCD variables. What I currently have are all set as variables and not files, not masked, not protected using the

The auth.log of the remote system seems to indicate that the gitlab server still fails to offer an RSA key, even when I explicitly use a HostKeys entry in ~/.ssh/config:

Sep 21 14:55:57 doc0123 sshd[13286]: Unable to negotiate with 17.74.21.149 port 55462: no matching host key type found. Their offer: sk-ecdsa-sha2-nistp256@openssh.com [preauth]
Sep 21 14:55:57 doc0123 sshd[13287]: Connection closed by 17.74.21.149 port 55456 [preauth]
Sep 21 14:55:57 doc0123 2sshd[13288]: Unable to negotiate with 17.74.21.149 port 55458: no matching host key type found. Their offer: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 [preauth]
Sep 21 14:55:57 doc0123 sshd[13289]: Unable to negotiate with 17.74.21.149 port 55460: no matching host key type found. Their offer: ssh-ed25519 [preauth]
Sep 21 14:55:57 doc0123 sshd[13290]: Unable to negotiate with 17.74.21.149 port 55464: no matching host key type found. Their offer: sk-ssh-ed25519@openssh.com [preauth]
Sep 21 14:57:35 doc0123 sshd[13299]: Connection closed by authenticating user luser 17.74.21.149 port 55564 [preauth]

This suggests the culprit is the keyscan command.
I now believe we have a disagreement between client and server over the hostkey algorithm we need to agree upon. Using ECDSA now and am seeing in auth.log:

Sep 21 16:52:54 doc01 sshd[14540]: Unable to negotiate with 17.73.135.57 port 38930: no matching host key type found. Their offer: ecdsa-sha2-nistp256 [preauth]

And the relevant section of the pipeline job log:

debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: (no match)
Unable to negotiate with 63.250.59.146 port 22: no matching host key type found. Their offer: ssh-rsa
scp: Connection closed

Relevant sshd_config on server is:

#       $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

Include /etc/ssh/sshd_config.d/*.conf

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
PermitRootLogin no
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile     .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem sftp  /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#       X11Forwarding no
#       AllowTcpForwarding no
#       PermitTTY no
#       ForceCommand cvs server
PasswordAuthentication yes
HostKeyAlgorithms sk-ecdsa-sha2-nistp256@openssh.com