Trouble deploying Docker Container to EC2 via SSH

Hi everyone,

I’m currently trying to deploy my NodeJS API by dockerising it and put it onto an EC2, however I’ve head a dead end with this error: load pubkey "/root/.ssh/id_rsa": invalid format

Things I’ve tried

  • Converting my Private RSA Key from PSK1 to PSK8
  • Setting protected variable to false
    (and various combinations of the above)
  • Using apt-get install openssl-client and converting the Private RSA Key from within the docker (but apparently I’m missing apt-get so I’ve also tried apk which also didn’t work so… ???)

Questionable:

  • The error mentions pubkey, but all tutorials have pointed to using my pem key which is generated from my EC2 setup. So perhaps I’m missing a pubkey? But I can’t find any material saying otherwise.

gitlab-ci.yml

image: docker:19.03.12
variables:
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_DRIVER: overlay2
services:
  - docker:19.03.0-dind
before_script:
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
stages:
  - build
  - deploy
build:
  stage: build
  script:
    # - docker build --tag $CI_REGISTRY/$CI_PROJECT_PATH:latest . 
    # - docker push $CI_REGISTRY/$CI_PROJECT_PATH:latest
    - docker pull $CI_REGISTRY_IMAGE:latest || true
    - docker build --cache-from $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY/$CI_PROJECT_PATH:latest . 
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY/$CI_PROJECT_PATH:latest
deploy:
  stage: deploy
  only:
    - master
  script:
    ###### Login to deployment server using SSH #####
    - mkdir -p ~/.ssh
    - echo "$DEPLOY_SERVER_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 700 ~/.ssh/id_rsa
    - '[[ -f ./dockerenv ]] && echo -e "Host *\n\tStrickHostKeyChecking no\n\n" > ~/.ssh/config'
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - eval "$(ssh-agent -s)"
    - ssh-add ~/.ssh/id_rsa
    - ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts  
    ###### Pull the registry to deployment server and recereate the docker service #####    
    - ssh $DEPLOYMENT_SERVER_USERNAME@$DEPLOYMENT_SERVER_IP docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY 
    - ssh $DEPLOYMENT_SERVER_USERNAME@$DEPLOYMENT_SERVER_IP docker pull $CI_REGISTRY/$CI_PROJECT_PATH:latest
    - ssh $DEPLOYMENT_SERVER_USERNAME@$DEPLOYMENT_SERVER_IP docker run $CI_REGISTRY/$CI_PROJECT_PATH:latest

gitlab-runner config.toml

[[runners]]
  name = "docker-runner"
  url = "https://gitlab.com/"
  token = "secretkey"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "docker:19.03.12"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache"]
    shm_size = 0

CI runner output

$ ssh $DEPLOYMENT_SERVER_USERNAME@$DEPLOYMENT_SERVER_IP docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
load pubkey "/root/.ssh/id_rsa": invalid format
Login Succeeded
$ ssh $DEPLOYMENT_SERVER_USERNAME@$DEPLOYMENT_SERVER_IP docker pull $CI_REGISTRY/$CI_PROJECT_PATH:latest
load pubkey "/root/.ssh/id_rsa": invalid format

Thanks in advanced,
Geoff

Hey @gyfchong,
we were live analysing your Problem so here is our solution:

  1. encode your ssh-key as "base64 string "
    cat $HOME/.ssh/id_rsa | base64

  2. paste this your output in your CI/CD variable “$DEPLOY_SERVER_PRIVATE_KEY”

Afterwards your need to edit your gitlab-ci.yml

echo -n "$DEPLOY_SERVER_PRIVATE_KEY" | base64 --decode > ~/.ssh/id_rsa

Your .gitlab-ci.yml should look like the following than afterwards

script:
- echo -n "$DEPLOY_SERVER_PRIVATE_KEY" | base64 --decode > ~/.ssh/id_rsa
- chmod 700 ~/.ssh
- chmod 600 ~/.ssh/id_rsa

Here is also a short explanation of why you should doing this.

I hope this helps if you want to check the german recording you follow the following link below:

2 Likes

Hi,

Thanks for live analysing my problem!

I followed the steps and, here’s the new output:

$ echo -n "$DEPLOY_SERVER_PRIVATE_KEY" | base64 --decode > ~/.ssh/id_rsa
base64: unrecognized option: decode
BusyBox v1.31.1 () multi-call binary.
Usage: base64 [-d] [FILE]
Base64 encode or decode FILE to standard output
	-d	Decode data
ERROR: Job failed: exit code 1

Cheers,
Geoff

Hi,

seems that decode in the Docker image is not available for base64, or it uses different flags.

Is there a specific reason to use this image for every job? I would leave this to the job which runs DinD and builds the image.

There’s two approaches I would recommend:

  1. Change the --decode flag to the shorter -d and try again.
  2. Modify the image for the deployment stage, e.g. to debian:latest. Do not use alpine, this is the base image layer for docker:19.03.12.

Explanation

Short local tests with Docker.

Docker 19.03.12

$ docker run -ti docker:19.03.12 sh

/ # base64 --help
BusyBox v1.31.1 () multi-call binary.

Usage: base64 [-d] [FILE]

Base64 encode or decode FILE to standard output
	-d	Decode data

Alpine

$ docker run -ti alpine:latest sh

/ # base64 --help
BusyBox v1.31.1 () multi-call binary.

Usage: base64 [-d] [FILE]

Base64 encode or decode FILE to standard output
	-d	Decode data

Debian

$ docker run -ti debian:latest sh

# base64 --help
Usage: base64 [OPTION]... [FILE]
Base64 encode or decode FILE, or standard input, to standard output.

With no FILE, or when FILE is -, read standard input.

Mandatory arguments to long options are mandatory for short options too.
  -d, --decode          decode data
  -i, --ignore-garbage  when decoding, ignore non-alphabet characters
  -w, --wrap=COLS       wrap encoded lines after COLS character (default 76).
                          Use 0 to disable line wrapping

      --help     display this help and exit
      --version  output version information and exit

The data are encoded as described for the base64 alphabet in RFC 4648.
When decoding, the input may contain newlines in addition to the bytes of
the formal base64 alphabet.  Use --ignore-garbage to attempt to recover
from any other non-alphabet bytes in the encoded stream.

GNU coreutils online help: <https://www.gnu.org/software/coreutils/>
Report base64 translation bugs to <https://translationproject.org/team/>
Full documentation at: <https://www.gnu.org/software/coreutils/base64>
or available locally via: info '(coreutils) base64 invocation'

Cheers,
Michael

Hey @gyfchong & @dnsmichi,

I thought that I send my answer already. The problem here is that the provided base64 binary isn’t the normal GNU base64. So in Debian it’s included by default. If you want to stick to your current image mostly all docker:* images are based on alpine linux. So you need only to install the coreutils package and everything should work as excepted.

You can use my example if you want to

Example

before_script:
 - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
 - apk add --no-cache coreutils bash

If you got problems to execute shell scripts in alpine images install also the bash package. This is also a common error mostly.

References

1 Like