Gitlab runners stopped working

Hi all,

I have a project using middleman app (static website generator - ruby) set up on gitlab pages which doesn’t work any more. Did anything changed to gitlab Runners ?how can I begin to debug this ?

Running with gitlab-runner 11.9.0-rc2 (227934c0)
on docker-auto-scale ed2dce3a
Using Docker executor with image ruby:2.3.1 ...
Pulling docker image ruby:2.3.1 ...
Using docker image sha256:ffe8239a147c666621476db01004e6d525b9a3b0db2deace8861afba2aef3001 for ruby:2.3.1 ...
Running on runner-ed2dce3a-project-11042130-concurrent-0 via runner-ed2dce3a-srm-1553589370-7b7f2629...
Initialized empty Git repository in /builds/nedzen/scytale/.git/
Fetching changes...
Created fresh repository.
From https://gitlab.com/nedzen/scytale
 * [new branch]      master     -> origin/master
 * [new branch]      wip        -> origin/wip
Checking out c6d1ceb7 as master...

Skipping Git submodules setup
Checking cache for default-5...
FATAL: file does not exist                         
Failed to extract cache
$ export LC_ALL=C.UTF-8
$ export LANG=en_US.UTF-8
$ export LANGUAGE=en_US.UTF-8
$ apt-get update -yqqq
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/main/binary-amd64/Packages  404  Not Found [IP: 151.101.248.204 80]

E: Some index files failed to download. They have been ignored, or old ones used instead.
ERROR: Job failed: exit code 1
3 Likes

Hi,

same here. I’m building an Angular app with dind in gitlab CI/CD.
This worked yesterday:

Status: Downloaded newer image for node:9.4
 ---> c888d933885c
Step 2/20 : RUN apt-get update     && apt-get install -y wget
 ---> Running in dcc27cf6571f
Get:1 http://security.debian.org jessie/updates InRelease [44.9 kB]
Ign http://deb.debian.org jessie InRelease
Ign http://deb.debian.org jessie-updates InRelease
Get:2 http://deb.debian.org jessie Release.gpg [2420 B]
Ign http://deb.debian.org jessie-updates Release.gpg
Get:3 http://deb.debian.org jessie Release [148 kB]
Ign http://deb.debian.org jessie-updates Release
Err http://deb.debian.org jessie-updates/main amd64 Packages
  
Err http://deb.debian.org jessie-updates/main amd64 Packages
  
Err http://deb.debian.org jessie-updates/main amd64 Packages
  
Err http://deb.debian.org jessie-updates/main amd64 Packages
  
Err http://deb.debian.org jessie-updates/main amd64 Packages
  404  Not Found
Get:4 http://deb.debian.org jessie/main amd64 Packages [9098 kB]
Get:5 http://security.debian.org jessie/updates/main amd64 Packages [822 kB]
Fetched 10.1 MB in 9s (1093 kB/s)
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/main/binary-amd64/Packages  404  Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.
The command '/bin/sh -c apt-get update     && apt-get install -y wget' returned a non-zero code: 100
ERROR: Job failed: exit code 100
1 Like

It seems that it comes from the removal of the repository jessie-updates by debian (see this current discussion https://www.mail-archive.com/debian-user@lists.debian.org/msg740473.html) although I don’t know if it’s intended or an error.

As a temporary solution, I removed the line deb http://deb.debian.org/debian/ jessie-updates main from the /etc/apt/sources.list of the docker image in which the CI process runs, by putting this in my .gitlab-ci.yml

before_script:
  - echo -e "deb http://deb.debian.org/debian jessie main\ndeb http://security.debian.org jessie/updates main" > /etc/apt/sources.list

This is only to get my CI running today, but I hope this will resolved either by the debian team (fixing their repository), or by the docker image maintainer to adjust to these changes

2 Likes

I have the exact same issue.
Your solution almost worked for me. However I need to remove the backports from the sources as well.

Err http://deb.debian.org jessie-backports/main amd64 Packages
  404  Not Found [IP: 151.101.248.204 80]
Fetched 10.1 MB in 5s (1690 kB/s)
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-backports/main/binary-amd64/Packages  404  Not Found [IP: 151.101.248.204 80]

E: Some index files failed to download. They have been ignored, or old ones used instead.
ERROR: Job failed: exit code 1

How can I remove it?

deb http://deb.debian.org/debian jessie-backports main

didn’t helped

well this line - echo -e "deb http://deb.debian.org/debian jessie main\ndeb http://security.debian.org jessie/updates main" > /etc/apt/sources.list is supposed to replace whatever your sources.list was with just

deb http://deb.debian.org/debian jessie main
deb http://security.debian.org jessie/updates main

If you still have references to jessie-backports then I guess it’s still configured somewhere (maybe under sources.list.d ?), it will depends on your docker image

Hi!
I’ve got the same issue… Does this come from Debian CDN ?
Damien

The before-script from @quentin.richaud fixed the issue for me.
Is there an issue in the tracker for this?

Added an upstream issue against the docker package:
[Critical] debian repositories broken · Issue #64 · TetraWeb/docker

Adding the before_script fixed the issue for me.

@quentin.richaud thanks I got mine working with before_script code

It seems that debian jessie is going to be deprecated and the better long term solution is to refrain from using jessie based images. For example I was using docker image node:8.10, and switching to node:8.10-stretch solved this.

@quentin.richaud, adding the before_script fixed a part of the issue for me. But I still add an error when I was building a Scala app. So I also add rm /etc/apt/sources.list.d/jessie-backports.list to my before_script.

1 Like

@prlherisson adding rm /etc/apt/sources.list.d/jessie-backports.list works for me as well, thanks!

Also added a gitlab.com docs issue.

Docs feedback: recommended tetraweb/php is broken (#61059) · Issues · GitLab.org / GitLab Community Edition · GitLab