Docker+Machine runner CA issue - x509: certificate signed by unknown authority

GitLab CI/CD
1

I am experiencing an issue with a docker+machine runner which is running a gitlab-ci.yml that uses a docker image to run CI tools on docker containers such as terraform. It gets to the docker login and fails with “x509: certificate signed by unknown authority”.

I’m perplexed as the CA certificate has been installed on the docker image using two ways:

In my (sanitised) config.toml file I have:

pre_build_script = """
    export HTTP_PROXY=http://192.168.0.1:3128/
    export HTTPS_PROXY=http://192.168.0.1:3128/
    export NO_PROXY=<internal domain>,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,docker
    apk update >/dev/null
    apk add ca-certificates > /dev/null
    rm -rf /var/cache/apk/*
    wget http://<CA Webserver>certs/ca-cert.pem -O /usr/local/share/ca-certificates/ca-cert.pem
    update-ca-certificates --fresh > /dev/null
"""

and in my gitlab-ci.yml:

    before_script:
        - |
            set -xv
            http_proxy=$http_proxy
            https_proxy=$https_proxy
            no_proxy=${no_proxy},docker
            apk add --no-cache ca-certificates
            wget http://<ca webserver>/certs/ca-cert.pem -O /usr/local/share/ca-certificates/ca-cert.pem && update-ca-certificates

Now I believe these are superflous as, looking at the CI output, it appears they run on the same docker container.

Can anyone suggest why the addition of the CA is not working?

Here is my config.toml in full:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "docker-machine-runner"
  url = "https://gitlab.<internal domain>"
  token = "syzrstC3iD36x97iTXgX"
  executor = "docker+machine"
  environment = [
    "GIT_SSL_NO_VERIFY=1"
  ]
  pre_build_script = """
  export HTTP_PROXY=http://192.168.0.1:3128/
  export HTTPS_PROXY=http://192.168.0.1:3128/
  export NO_PROXY=<internal domain>,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,docker
  apk update >/dev/null
  apk add ca-certificates > /dev/null
  rm -rf /var/cache/apk/*
  wget http://<CA webserver>/certs/ca-cert.pem -O /usr/local/share/ca-certificates/ca-cert.pem
  update-ca-certificates --fresh > /dev/null
  """
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/certs/client", "/var/run/docker.sock:/var/run/docker.sock"]
    network_mode = "host"
    shm_size = 0
  [runners.machine]
    IdleCount = 0
    IdleTime = 60
    IdleScaleFactor = 0.0
    IdleCountMin = 0
    MachineDriver = "vmwarevsphere"
    MachineName = "gitlab-docker-machine-runner-%s"
    MachineOptions = ["vmwarevsphere-username=administrator@vsphere.local", "vmwarevsphere-password=<password>", "vmwarevsphere-vcenter=vc", "vmwarevsphere-vcenter-port=443", "vmwarevsphere-datacenter=<DC>", "vmwarevsphere-folder=DevOps/Tools/Gitlab", "vmwarevsphere-datastore=fastnas_DS1_2TB", "vmwarevsphere-cpu-count=2", "vmwarevsphere-memory-size=2048", "vmwarevsphere-disk-size=2048", "vmwarevsphere-disk-size=10000", "vmwarevsphere-network=DSwitch0_VLAN-100-Mgmt", "vmwarevsphere-boot2docker-url=http://192.168.0.1/yum/boot2docker/boot2docker.iso", "engine-env=HTTP_PROXY=http://192.168.0.1:3128/", "engine-env=HTTPS_PROXY=http://192.168.0.1:3128/", "engine-env=NO_PROXY=.<internal domain>,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"]

and my gitlab-ci.yml:

stages:
    - terraform_plan

merge review:
    stage: terraform_plan
    image: docker:20.10.16
    variables:
        DOCKER_TLS_CERTDIR: ""
    before_script:
        - |
            set -xv
            http_proxy=$http_proxy
            https_proxy=$https_proxy
            no_proxy=${no_proxy},docker
            apk add --no-cache ca-certificates
            wget http://<CA webserver>/certs/ca-cert.pem -O /usr/local/share/ca-certificates/ca-cert.pem && update-ca-certificates
            docker info
            docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    script:
        - |
            set -xv
            echo -en "http_proxy=${http_proxy}\n\
            https_proxy=${https_proxy}\n\
            no_proxy=${no_proxy},docker\n\
            CI_REGISTRY=${CI_REGISTRY}\n\
            TF_IMAGE=${TF_IMAGE}\n\
            TF_IMAGE_TAG=${TF_IMAGE_TAG}\n\
            TF_VAR_vsphere_user=${TF_VAR_vsphere_user}\n\
            TF_VAR_vsphere_password='${TF_VAR_vsphere_password}'\n\
            TF_VAR_dns_key_secret=${TF_VAR_dns_key_secret}\n\
            TERRAFORM_TFSTATE_NFS_HOST=${TERRAFORM_TFSTATE_NFS_HOST}\n\
            TERRAFORM_TFSTATE_NFS_OPTS=${TERRAFORM_TFSTATE_NFS_OPTS}\n\
            TERRAFORM_TFSTATE_NFS_PATH=${TERRAFORM_TFSTATE_NFS_PATH}\n\
            TERRAFORM_TFSTATE_MOUNTPOINT=${TERRAFORM_TFSTATE_MOUNTPOINT}\n" > .env
            echo -en "terraform {\n  backend \"local\" {\n    path = \"${TERRAFORM_TFSTATE_MOUNTPOINT}/terraform.tfstate\"\n  }\n}\n" > tfstate.tf
            docker compose run --rm terraform init
            docker compose run --rm terraform plan --out=${TERRAFORM_TFSTATE_MOUNTPOINT}/$PLAN | tee -a $PLAN.txt
            echo \`\`\` >> ${PLAN}.txt
            sed -i -e 's/  +/+/g' ${PLAN}.txt
            sed -i -e 's/  ~/~/g' ${PLAN}.txt
            sed -i -e 's/  -/-/g' ${PLAN}.txt
            MESSAGE=$(cat ${PLAN}.txt)
            curl -X POST -g -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}" --data-urlencode "body=${MESSAGE}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests/${CI_MERGE_REQUEST_IID}/discussions"
    artifacts:
        name: plan
        paths:
            - ${TERRAFORM_TFSTATE_MOUNTPOINT}/$PLAN

Any suggestions?

2

Please format your post properly as stated in the FAQ section: Community, first steps: Code, config, log block formatting in topics and replies

It’s a nightmare for anyone to read when people post and cannot be bothered to use the post formatting tools. If you want people to put in the effort to help, then put in the effort to post properly.

1 Like
3

There you go

4

Perhaps add some usage [in the before-script section on gitlab-ci.yml] of openssl s_client to verify the CA certs? Just to perhaps shed some additional light on what’s happening?

I do recall from the update-ca-certificates man page, Certificates must have a .crt extension in order to be included by update-ca-certificates. Perhaps your *.pem is being ignored?

5

The issue is because I installed the certificate on the docker container whereas in actuality it needed to be installed on the Docker Machine VM.

I don’t know how to get the certificates to be included in the Docker Machine VM (boot2docker), is there any kind of post start script that can be set up in the runner that will execute and copy in the certs?

6

Latest attempt failed:

Set tls_cert_path in config.toml and put ca cert in that directory, same error.

Is there nobody that can give me a clear cut answer to how I put a CA certificate on the boot2docker VM when started by gitlab-runner? I can find lots of answers on how to do this using ssh but since this is being executed by gitlab-runner those do not apply. It’s beginning to get somewhat infuriating how there is a lack of information around this. The doco says what “tls_cert_path” is, but not how to use it, no examples, nothing. Just that it is useful for boot2docker.

7

Maybe this topic will help as this person also used boot2docker.

8

More things I tried:

Set volumes in runners.docker section:

  1. to provide the CA cert in
    /etc/docker/certs.d/<hostname>/
    then when that failed
  2. provide daemon.json file with insecure registry as a workaround - failed.
9

I already have that in the environment of the [[runner]] section or do you mean setting it in a different way?

environment = [
“GIT_SSL_NO_VERIFY=1”
]

10

Perhaps next time check who the author of that post was first lol

It was my own post from a while back and unrelated issue

11

Yeah ok Sorry. I’ll just ignore the posts from now then instead of attempting to help.

12

Solved!

The solution was to build a new boot2docker.iso containing the CA certificates.

I had tried it previously but ran into an issue with “kernel too old” errors. That, it turned out was the local kernel on the system (Centos 7) not the image itself and was resolved by vomits building the image on Windows vomits. Once I ran it the pipeline just worked.

1 Like