Gitlab Runner randomly crashes with exit code 1

  • What are you seeing, and how does that differ from what you expect to see?
    the test job exits with code 1 in the middle of downloading a dependency which i verified beeing online
  • Consider including screenshots, error messages, and/or other helpful visuals
    gitlab ci file:
test:
  image:
    name: golang
  stage: test
  environment:
    name: Production
  except:
    - tags
  only:
    - master
  tags:
    - docker
  script:
    - go test "./..." -coverprofile="coverage.out" -covermode=count -json > report.json
  artifacts:
    paths:
      - coverage.out
      - report.json

Logs:

e[0KRunning with gitlab-runner 14.3.0 (b37d3da9)e[0;m
e[0K  on DockerShared tx_iefsze[0;m
section_start:1632788128:prepare_executor
e[0Ke[0Ke[36;1mPreparing the "docker" executore[0;me[0;m
e[0KUsing Docker executor with image golang ...e[0;m
e[0KStarting service docker:dind ...e[0;m
e[0KPulling docker image docker:dind ...e[0;m
e[0KUsing docker image sha256:8e5d106d317b2ddf45e60b5ba90a53082d306fac5c96e36ef3f2ad48e9719adb for docker:dind with digest docker@sha256:1efacb06edbb1e2bd4a39dcc69883866c7dd98562a49fa97d7d29c2da1c3974f ...e[0;m
e[0KWaiting for services to be up and running...e[0;m

e[0;33m*** WARNING:e[0;m Service runner-txiefsz-project-8-concurrent-0-14da5803acdc5850-docker-0 probably didn't start properly.

Health check error:
service "runner-txiefsz-project-8-concurrent-0-14da5803acdc5850-docker-0-wait-for-service" timeout

Health check container logs:


Service container logs:
2021-09-28T00:15:30.417972585Z time="2021-09-28T00:15:30.416693921Z" level=info msg="Starting up"
2021-09-28T00:15:30.422295780Z time="2021-09-28T00:15:30.422120507Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
2021-09-28T00:15:30.423583153Z time="2021-09-28T00:15:30.423410555Z" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://0.0.0.0:2375"
2021-09-28T00:15:30.423612160Z time="2021-09-28T00:15:30.423453653Z" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://0.0.0.0:2375"
2021-09-28T00:15:31.424081306Z time="2021-09-28T00:15:31.423706934Z" level=warning msg="Binding to an IP address without --tlsverify is deprecated. Startup is intentionally being slowed down to show this message" host="tcp://0.0.0.0:2375"
2021-09-28T00:15:31.424579818Z time="2021-09-28T00:15:31.424363381Z" level=warning msg="Please consider generating tls certificates with client validation to prevent exposing unauthenticated root access to your network" host="tcp://0.0.0.0:2375"
2021-09-28T00:15:31.424606432Z time="2021-09-28T00:15:31.424415262Z" level=warning msg="You can override this by explicitly specifying '--tls=false' or '--tlsverify=false'" host="tcp://0.0.0.0:2375"
2021-09-28T00:15:31.424616693Z time="2021-09-28T00:15:31.424434552Z" level=warning msg="Support for listening on TCP without authentication or explicit intent to run without authentication will be removed in the next release" host="tcp://0.0.0.0:2375"
2021-09-28T00:15:46.432100842Z time="2021-09-28T00:15:46.431790931Z" level=info msg="libcontainerd: started new containerd process" pid=36
2021-09-28T00:15:46.432635333Z time="2021-09-28T00:15:46.432336077Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2021-09-28T00:15:46.432655328Z time="2021-09-28T00:15:46.432386864Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2021-09-28T00:15:46.432852046Z time="2021-09-28T00:15:46.432696338Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2021-09-28T00:15:46.433127584Z time="2021-09-28T00:15:46.432937339Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2021-09-28T00:15:46.504668492Z time="2021-09-28T00:15:46.503993428Z" level=info msg="starting containerd" revision=e25210fe30a0a703442421b0f60afac609f950a3 version=v1.4.9
2021-09-28T00:15:46.548472825Z time="2021-09-28T00:15:46.548209261Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
2021-09-28T00:15:46.549153712Z time="2021-09-28T00:15:46.549003635Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
2021-09-28T00:15:46.563537360Z time="2021-09-28T00:15:46.563246402Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
2021-09-28T00:15:46.564445414Z time="2021-09-28T00:15:46.564240863Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
2021-09-28T00:15:46.564488727Z time="2021-09-28T00:15:46.564312485Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
2021-09-28T00:15:46.564612175Z time="2021-09-28T00:15:46.564473734Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
2021-09-28T00:15:46.564650727Z time="2021-09-28T00:15:46.564514191Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
2021-09-28T00:15:46.564971228Z time="2021-09-28T00:15:46.564786767Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
2021-09-28T00:15:46.565827573Z time="2021-09-28T00:15:46.565668184Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
2021-09-28T00:15:46.566430300Z time="2021-09-28T00:15:46.566191795Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
2021-09-28T00:15:46.566452068Z time="2021-09-28T00:15:46.566249880Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
2021-09-28T00:15:46.566461451Z time="2021-09-28T00:15:46.566375435Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
2021-09-28T00:15:46.566468608Z time="2021-09-28T00:15:46.566403140Z" level=info msg="metadata content store policy set" policy=shared
2021-09-28T00:15:46.572593417Z time="2021-09-28T00:15:46.572364908Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
2021-09-28T00:15:46.572630533Z time="2021-09-28T00:15:46.572499928Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
2021-09-28T00:15:46.573103779Z time="2021-09-28T00:15:46.572955223Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
2021-09-28T00:15:46.573243734Z time="2021-09-28T00:15:46.573135248Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
2021-09-28T00:15:46.573255192Z time="2021-09-28T00:15:46.573174554Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
2021-09-28T00:15:46.573266219Z time="2021-09-28T00:15:46.573205509Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
2021-09-28T00:15:46.573377142Z time="2021-09-28T00:15:46.573263964Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
2021-09-28T00:15:46.573388162Z time="2021-09-28T00:15:46.573321085Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
2021-09-28T00:15:46.573460096Z time="2021-09-28T00:15:46.573367083Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
2021-09-28T00:15:46.573466826Z time="2021-09-28T00:15:46.573402816Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
2021-09-28T00:15:46.573564759Z time="2021-09-28T00:15:46.573469973Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
2021-09-28T00:15:46.574149016Z time="2021-09-28T00:15:46.573997507Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
2021-09-28T00:15:46.574448577Z time="2021-09-28T00:15:46.574340701Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
2021-09-28T00:15:46.576302468Z time="2021-09-28T00:15:46.576163430Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
2021-09-28T00:15:46.576493085Z time="2021-09-28T00:15:46.576367111Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
2021-09-28T00:15:46.577221588Z time="2021-09-28T00:15:46.577059327Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
2021-09-28T00:15:46.577506501Z time="2021-09-28T00:15:46.577168750Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
2021-09-28T00:15:46.577521330Z time="2021-09-28T00:15:46.577323192Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
2021-09-28T00:15:46.577526378Z time="2021-09-28T00:15:46.577365283Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
2021-09-28T00:15:46.577530617Z time="2021-09-28T00:15:46.577407577Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
2021-09-28T00:15:46.577663098Z time="2021-09-28T00:15:46.577439373Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
2021-09-28T00:15:46.577686839Z time="2021-09-28T00:15:46.577495938Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
2021-09-28T00:15:46.577713118Z time="2021-09-28T00:15:46.577518351Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
2021-09-28T00:15:46.577721528Z time="2021-09-28T00:15:46.577540497Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
2021-09-28T00:15:46.578422014Z time="2021-09-28T00:15:46.578190668Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
2021-09-28T00:15:46.578438675Z time="2021-09-28T00:15:46.578258749Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
2021-09-28T00:15:46.578443492Z time="2021-09-28T00:15:46.578291481Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
2021-09-28T00:15:46.578472524Z time="2021-09-28T00:15:46.578337941Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
2021-09-28T00:15:46.580684267Z time="2021-09-28T00:15:46.580502890Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
2021-09-28T00:15:46.581222382Z time="2021-09-28T00:15:46.581045268Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
2021-09-28T00:15:46.581576249Z time="2021-09-28T00:15:46.581438911Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
2021-09-28T00:15:46.585349577Z time="2021-09-28T00:15:46.584579699Z" level=info msg="containerd successfully booted in 0.090940s"
2021-09-28T00:15:46.647263934Z time="2021-09-28T00:15:46.647016555Z" level=info msg="Setting the storage driver from the $DOCKER_DRIVER environment variable (overlay2)"
2021-09-28T00:15:46.648535703Z time="2021-09-28T00:15:46.648327398Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2021-09-28T00:15:46.648558709Z time="2021-09-28T00:15:46.648396788Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2021-09-28T00:15:46.648565316Z time="2021-09-28T00:15:46.648437529Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2021-09-28T00:15:46.648574903Z time="2021-09-28T00:15:46.648457436Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2021-09-28T00:15:46.653570003Z time="2021-09-28T00:15:46.652230310Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2021-09-28T00:15:46.653612699Z time="2021-09-28T00:15:46.652414993Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2021-09-28T00:15:46.653621887Z time="2021-09-28T00:15:46.652586931Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2021-09-28T00:15:46.653629697Z time="2021-09-28T00:15:46.652621357Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2021-09-28T00:15:46.720085689Z time="2021-09-28T00:15:46.719777128Z" level=warning msg="Your kernel does not support swap memory limit"
2021-09-28T00:15:46.720130499Z time="2021-09-28T00:15:46.719887862Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
2021-09-28T00:15:46.720140522Z time="2021-09-28T00:15:46.719899694Z" level=warning msg="Your kernel does not support cgroup blkio weight"
2021-09-28T00:15:46.720148129Z time="2021-09-28T00:15:46.719910819Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
2021-09-28T00:15:46.720803093Z time="2021-09-28T00:15:46.720654751Z" level=info msg="Loading containers: start."
2021-09-28T00:15:46.866278877Z time="2021-09-28T00:15:46.866018499Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address"
2021-09-28T00:15:46.932353287Z time="2021-09-28T00:15:46.932108685Z" level=info msg="Loading containers: done."
2021-09-28T00:15:46.985883332Z time="2021-09-28T00:15:46.985550533Z" level=info msg="Docker daemon" commit=75249d8 graphdriver(s)=overlay2 version=20.10.8
2021-09-28T00:15:46.986252835Z time="2021-09-28T00:15:46.986087117Z" level=info msg="Daemon has completed initialization"
2021-09-28T00:15:47.071664297Z time="2021-09-28T00:15:47.071105048Z" level=info msg="API listen on /var/run/docker.sock"
2021-09-28T00:15:47.084039722Z time="2021-09-28T00:15:47.083729683Z" level=info msg="API listen on [::]:2375"

e[0;33m*********e[0;m

e[0KPulling docker image golang ...e[0;m
e[0KUsing docker image sha256:ec365f06285d63175d6ae474a6ddc9d1d22ebf4625903b54b4c54836d2a323a9 for golang with digest golang@sha256:285cf0cb73ab995caee61b900b2be123cd198f3541ce318c549ea5ff9832bdf0 ...e[0;m
section_end:1632788163:prepare_executor
e[0Ksection_start:1632788163:prepare_script
e[0Ke[0Ke[36;1mPreparing environmente[0;me[0;m
Running on GitLabRunnerWorker via 39301228e795...
section_end:1632788164:prepare_script
e[0Ksection_start:1632788164:get_sources
e[0Ke[0Ke[36;1mGetting source from Git repositorye[0;me[0;m
e[32;1mFetching changes with git depth set to 50...e[0;m
Reinitialized existing Git repository in /build/tx_iefsz/0/awesomedb/awesomekv/.git/
e[32;1mChecking out 53e240d8 as master...e[0;m
Removing coverage.out
Removing report.json
Removing src/internal/authentication/data/

e[32;1mSkipping Git submodules setupe[0;m
section_end:1632788167:get_sources
e[0Ksection_start:1632788167:step_script
e[0Ke[0Ke[36;1mExecuting "step_script" stage of the job scripte[0;me[0;m
e[0KUsing docker image sha256:ec365f06285d63175d6ae474a6ddc9d1d22ebf4625903b54b4c54836d2a323a9 for golang with digest golang@sha256:285cf0cb73ab995caee61b900b2be123cd198f3541ce318c549ea5ff9832bdf0 ...e[0;m
e[32;1m$ go test "./..." -coverprofile="coverage.out" -covermode=count -json > report.jsone[0;m
go: downloading github.com/spf13/viper v1.8.1
go: downloading gitlab.ho-me.zone/awesomedb/awesomedb-dto v1.1.10
go: downloading gitlab.ho-me.zone/conamu/base-tools v0.0.0-20210730120135-afaa63b62129
go: downloading gitlab.ho-me.zone/conamu/base-tools/v2 v2.0.3
go: downloading go.mongodb.org/mongo-driver v1.7.2
go: downloading golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
go: downloading github.com/mitchellh/mapstructure v1.4.1
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading github.com/fsnotify/fsnotify v1.4.9
go: downloading github.com/magiconair/properties v1.8.5
go: downloading github.com/pelletier/go-toml v1.9.3
go: downloading github.com/spf13/afero v1.6.0
go: downloading github.com/spf13/cast v1.4.0
go: downloading github.com/spf13/jwalterweatherman v1.1.0
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/subosito/gotenv v1.2.0
go: downloading gopkg.in/ini.v1 v1.62.0
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c
go: downloading golang.org/x/text v0.3.6
go: downloading github.com/go-stack/stack v1.8.0
section_end:1632788198:step_script
e[0Ksection_start:1632788198:cleanup_file_variables
e[0Ke[0Ke[36;1mCleaning up project directory and file based variablese[0;me[0;m
section_end:1632788200:cleanup_file_variables
e[0Ke[31;1mERROR: Job failed: exit code 1
e[0;m

All the errors and warnings before the runner starts are the same as the ones when it worked
The errors seemed to begin randomly 3 days ago, as i never updated or changed anything.

I already tried:

  • download newest gitlab and gitlab runner images
  • clearing build and build cache folders
  • using older images of golang or docker:dind

Some help or nudges in the right direction are appretiated greatly!

  • *GitLab: 14.3.0 Self managed *
  • Runner: 14.3.0

Resolved, apparently gitlab runner doesnt show failed test result and just crashes… so it was my stupidity forgetting to test locally :frowning:

1 Like