Pulling from Gitlab Registry from Bitbucket Pipeline is Failing

I submitted this to Bitbucket and they said to submit this to Gitlab support.

We cannot deploy or merge any code at the moment because our build is failing. The docker login command which works in my terminal just fine fails when run in Bitbucket pipelines. We have changed nothing on Bitbucket. I’ve verified that the Gitlab registry credentials are all correct as the same commands connect perfectly locally and the env vars (Gitlab username and Gitlab api token) in bitbucket match them exactly.

This is affecting all our active repos that use pipelines cyno-api and cyno-web-app are the most critical.

echo $GITLAB_API_TOKEN | docker login registry.gitlab.com/cynoteam --username $GITLAB_USERNAME --password-stdin

This is the command that is failing with an error today.

The error is:
Error response from daemon: Get https://registry.gitlab.com/v2/: error parsing HTTP 403 response body: invalid character ‘e’ looking for beginning of value: “error code: 1020”

Again, I’ve verified the environment variables and the credentials themselves are correct.

It’s like something has changed with how pipelines work.

Full docker error log:

?
time=“2020-06-04T18:16:04.994822502Z” level=warning msg=“could not change group /var/run/docker.sock to docker: group docker not found”
time=“2020-06-04T18:16:04.994994433Z” level=warning msg="[!] DON’T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON’T KNOW WHAT YOU’RE DOING [!]"
time=“2020-06-04T18:16:05.016891274Z” level=warning msg=“failed to load plugin io.containerd.snapshotter.v1.btrfs” error=“path /var/lib/docker/165536.165536/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter”
time=“2020-06-04T18:16:05.023594893Z” level=warning msg=“failed to load plugin io.containerd.snapshotter.v1.aufs” error=“modprobe aufs failed: “ip: can’t find device ‘aufs’\nmodprobe: can’t change directory to ‘/lib/modules’: No such file or directory\n”: exit status 1”
time=“2020-06-04T18:16:05.024219942Z” level=warning msg=“failed to load plugin io.containerd.snapshotter.v1.zfs” error=“path /var/lib/docker/165536.165536/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter”
time=“2020-06-04T18:16:05.024316417Z” level=warning msg=“could not use snapshotter zfs in metadata plugin” error=“path /var/lib/docker/165536.165536/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter”
time=“2020-06-04T18:16:05.024334348Z” level=warning msg=“could not use snapshotter btrfs in metadata plugin” error=“path /var/lib/docker/165536.165536/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter”
time=“2020-06-04T18:16:05.024342385Z” level=warning msg=“could not use snapshotter aufs in metadata plugin” error=“modprobe aufs failed: “ip: can’t find device ‘aufs’\nmodprobe: can’t change directory to ‘/lib/modules’: No such file or directory\n”: exit status 1”
time=“2020-06-04T18:16:05.133108889Z” level=warning msg=“Running modprobe bridge br_netfilter failed with message: ip: can’t find device ‘bridge’\nbridge 167936 1 br_netfilter\nstp 16384 1 bridge\nllc 16384 2 bridge,stp\nip: can’t find device ‘br_netfilter’\nbr_netfilter 24576 0 \nbridge 167936 1 br_netfilter\nmodprobe: can’t change directory to ‘/lib/modules’: No such file or directory\n, error: exit status 1”
time=“2020-06-04T18:16:05.139535848Z” level=warning msg=“Running modprobe nf_nat failed with message: ip: can't find device 'nf_nat'\nnf_nat_ipv6 20480 1 ip6table_nat\nnf_nat_ipv4 16384 2 ipt_MASQUERADE,iptable_nat\nnf_nat 32768 3 nf_nat_ipv6,xt_nat,nf_nat_ipv4\nnf_conntrack 139264 8 nf_nat_ipv6,nf_conntrack_netlink,xt_nat,xt_conntrack,ip_vs,ipt_MASQUERADE,nf_nat_ipv4,nf_nat\nlibcrc32c 16384 3 ip_vs,nf_nat,nf_conntrack\nmodprobe: can't change directory to '/lib/modules': No such file or directory, error: exit status 1”
time=“2020-06-04T18:16:05.171346235Z” level=warning msg=“Running modprobe xt_conntrack failed with message: ip: can't find device 'xt_conntrack'\nxt_conntrack 16384 23 \nnf_conntrack 139264 8 nf_nat_ipv6,nf_conntrack_netlink,xt_nat,xt_conntrack,ip_vs,ipt_MASQUERADE,nf_nat_ipv4,nf_nat\nmodprobe: can't change directory to '/lib/modules': No such file or directory, error: exit status 1”
time=“2020-06-04T18:16:12Z” level=info msg=“Pipelines plugin request authorization.” allowed=true method=GET plugin=pipelines uri=/_ping
time=“2020-06-04T18:16:12Z” level=info msg=“Pipelines plugin request authorization.” allowed=true method=GET plugin=pipelines uri=/v1.39/info
time=“2020-06-04T18:16:12Z” level=info msg=“Pipelines plugin request authorization.” allowed=true method=POST plugin=pipelines uri=/v1.39/auth
time=“2020-06-04T18:16:12.658168741Z” level=error msg="Handler for POST /v1.39/auth returned error: Get https://registry.gitlab.com/v2/: error parsing HTTP 403 response body: invalid character ‘e’ looking for beginning of value: “error code: 1020"”
time=“2020-06-04T18:16:12.658740068Z” level=error msg="Handler for POST /v1.39/auth returned error: Get https://registry.gitlab.com/v2/: error parsing HTTP 403 response body: invalid character ‘e’ looking for beginning of value: “error code: 1020"”

I just checked and the issue has randomly resolved itself. I’m not sure what was causing it… It was not working all day yesterday.