Upgrading from 11.11.1-ce.0 to 12.0.2-ce.0 broke AutoDevops (CI/CD)

As the title says, I’m having issues running AutoDevops after upgrading to the latest release.

this what shows:

Running with gitlab-runner 10.5.0 (10.5.0)
  on Runner for all Projects xN1xTmPD
Using Docker executor with image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image/master:stable ...
Starting service docker:stable-dind ...
Pulling docker image docker:stable-dind ...
Using docker image sha256:ac662ef38e72ab3e0d1850cb47b8d9026602c4b6d0d6a350c84b5a2830248ede for docker:stable-dind ...
Waiting for services to be up and running...

*** WARNING: Service runner-xN1xTmPD-project-209-concurrent-0-docker-0 probably didn't start properly.

service runner-xN1xTmPD-project-209-concurrent-0-docker-0-wait-for-service did timeout

2019-06-28T08:38:32.877937400Z mount: permission denied (are you root?)
2019-06-28T08:38:32.877986880Z Could not mount /sys/kernel/security.
2019-06-28T08:38:32.877991930Z AppArmor detection and --privileged mode might break.
2019-06-28T08:38:32.878982155Z mount: permission denied (are you root?)

*********

Pulling docker image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image/master:stable ...
Using docker image sha256:0a33f6e09cc48a84379b8e091cfeef1d944b6629afe52736ed6518ceaa47075f for registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image/master:stable ...
Running on runner-xN1xTmPD-project-209-concurrent-0 via sowlutions-git-01...
Fetching changes...
HEAD is now at 3ce9fc8 asda
Checking out 3ce9fc85 as master...
Skipping Git submodules setup
$ [[ "$TRACE" ]] && set -x # collapsed multi-line command
$ if [[ -z "$CI_COMMIT_TAG" ]]; then # collapsed multi-line command
$ /build/build.sh
Logging to GitLab Container Registry with CI credentials...
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Get https://registry-1.docker.io/v2/: unauthorized: incorrect username or password
ERROR: Job failed: exit code 1

any help would be appreciated

Hi @M034B5, welcome to the GitLab community forum!

This issue is likely caused by the GitLab runner version being out of sync with your current GitLab version.

The GitLab Runner Compatibility with GitLab versions documentation reads:

The GitLab Runner version should be in sync with the GitLab version. While older Runners may still work with newer GitLab versions, and vice versa, in some cases, features may be not available or work properly if there’s a version difference.

According to the job output, it seems you’re running gitlab-runner version 10.5.0 with GitLab version 12.0.2. With two major versions difference between your runner and your GitLab version its likely that there will be breakage or errors like what you’re encountering. What you’ll want to do is upgrade the runner to version 12.0 to stay in sync with the GitLab instance version.

As you’re using the docker executor, you’ll want to follow the documentation to upgrade your gitlab-runner version.

If you still have problems after upgrading, please follow up with any error messages you’re seeing.

Best Regards,
Greg

Hello Greg,

Thanks for the prompt reply.
A thing to mention that I have gitlab-runner installed from the default ubuntu repos (not dockerized):

when I do apt-get update and apt-get upgrade it shows nothing .
when i try apt-get install gitlab-runner it shows:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
gitlab-runner is already the newest version (10.5.0+dfsg-2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

lsb_release -a shows:

Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

I do have Docker version 18.09.7, build 2d0083d installed for the runner (AutoDevOps) to work.

Meanwhile i’m following this: https://docs.gitlab.com/runner/install/linux-repository.html

will update the topic as soon as i get some results

update:

got the same result:

Using docker image sha256:ac662ef38e72ab3e0d1850cb47b8d9026602c4b6d0d6a350c84b5a2830248ede for docker:stable-dind ...
Waiting for services to be up and running...

*** WARNING: Service runner-xN1xTmPD-project-209-concurrent-0-docker-0 probably didn't start properly.

Health check error:
service "runner-xN1xTmPD-project-209-concurrent-0-docker-0-wait-for-service" timeout

Health check container logs:


Service container logs:
2019-07-01T08:46:38.700319662Z mount: permission denied (are you root?)
2019-07-01T08:46:38.700420365Z Could not mount /sys/kernel/security.
2019-07-01T08:46:38.700428038Z AppArmor detection and --privileged mode might break.
2019-07-01T08:46:38.701447582Z mount: permission denied (are you root?)

*********

Will try to check around.

update2

whenever i try to start the service it shows:

:~# gitlab-runner start
Runtime platform                                    arch=amd64 os=linux pid=22597 revision=0e5417a3 version=12.0.1
FATAL: Failed to start gitlab-runner: "systemctl" failed: exit status 5, Failed to start gitlab-runner.service: Unit gitlab-runner.service not found.

even /etc/init.d/gitlab-runner start shows the same error as above

@gitlab-greg any help would be really appreciated

Hi @M034B5,

Sorry to hear you’re still having problems.

Can you confirm that you’ve added the gitlab-runner Linux repository and updated your gitlab-runner version?

If so, can you describe what setup/configuration you had previously working with gitlab-runner and AutoDevOps? Are you using the docker exector for AutoDevOps?

Since you have Docker already installed, your best bet might be to use a Dockerized gitlab-runner in privelaged mode. This should fix any compatibility and privilege problems. Note that this would require registering the runner

Hi @gitlab-greg,

Thanks for your reply.
Just to be clear, I have both gitlab and the runner installed from gitlab repos:

Gitlab: deb https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic main
Runner: deb https://packages.gitlab.com/runner/gitlab-runner/ubuntu/ bionic main

Both are installed using apt-get running on Ubuntu 18.04.2 LTS x64

I only installed docker as a service for deployments for runner.
I did not install runner using docker.

I hope this clears out what’s vague.

In my previous message, systemctl can’t find the runner service.
So do you know by any chance how can I solve that?

Thanks in advance!

Update, after a lot of inspections, it turned out that the project I was testing on (which coincidentally was created the same date gitlab released ver 12.0) didn’t have the .gitlab-ci.yml file pushed. That’s why runner couldn’t find or run the service properly.

Sorry for this inconvenience