Waiting for deployment "production-rollout" rollout to finish: 0 of 1 updated replicas are available

Hi,

any pointers what this could be?
In GKE it looks like production-rollout 0 of 1 updated replicas available - ImagePullBackOff

more details:
Warning Failed 30s (x3 over 56s) kubelet, gke-cluster-1-default-pool-31243a54-64vj Error: ImagePullBackOff Normal Pulling 15s (x3 over 57s) kubelet, gke-cluster-1-default-pool-31243a54-64vj Pulling image “registry.gitlab.com/XXXXXX/master:da9f36d81a2829ea24bd48fcf67df972f07995a6” Warning Failed 15s (x3 over 56s) kubelet, gke-cluster-1-default-pool-31243a54-64vj Failed to pull image “[registry.gitlab.com/](http://registry.gitlab.com/XXXXXXX/master:da9f36d81a2829ea24bd48fcf67df972f07995a6”: rpc error: code = Unknown desc = Error response from daemon: manifest for registry.gitlab.com/XXXXXX/master:da9f36d81a2829ea24bd48fcf67df972f07995a6 not found Warning Failed 15s (x3 over 56s) kubelet, gke-cluster-1-default-pool-31243a54-64vj Error: ErrImagePull

gitlab when trying to deploy onto GKE and getting:

Deploying new rollout release…
Release “production-rollout” has been upgraded.
LAST DEPLOYED: Fri Apr 24 06:49:01 2020
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME AGE
production-rollout-7ccc665565-pj54c 2s
production-rollout-7d85cc78ff-nbdk2 41m

==> v1beta1/Deployment
NAME AGE
production-rollout 116m

NOTES:
Application should be accessible at

http://XXX.XXX.XXX.XXX
Waiting for deployment “production-rollout” rollout to finish: 0 of 1 updated replicas are available…
Running after_script
WARNING: Failed to inspect build container 283b571f2a586bb36f65eead4d2c9ae697c457b6741ef6cf129c6195edab2b6c context deadline exceeded (docker_command.go:77:0s)
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v0.13.0
Uploading artifacts for failed job
Pulling docker image gitlab/gitlab-runner-helper:x86_64-6c8c540f …
ERROR: Job failed: execution took longer than 1h0m0s seconds

any pointers?
thanks for your time.
pavel

Application should be accessible at

http://steamru-fnm-test-piankov.34.70.159.229.nip.io

  • [[ -z ‘’ ]]

  • kubectl rollout status -n fnm-test-piankov-18297570-production -w deployment/production-rollout

Waiting for deployment “production-rollout” rollout to finish: 0 of 1 updated replicas are available…

Running after_script

00:00

WARNING: Failed to inspect build container 761971b6a52180fd5ec5793268f2d48a0766c629687c49aad7fa6c7ee13e4b3c context deadline exceeded (docker_command.go:77:0s)

Authenticating with credentials from job payload (GitLab Registry)

Pulling docker image registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v0.13.0

Uploading artifacts for failed job

00:00

Pulling docker image gitlab/gitlab-runner-helper:x86_64-6c8c540f …

ERROR: Job failed: execution took longer than 1h0m0s seconds

and here is my gitlab-ci.yaml:

image: alpine:latest

variables:
ROLLOUT_RESOURCE_TYPE: deployment
POSTGRES_ENABLED: “false”
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: “”
TRACE: set -x

stages:

  • build
  • test
  • deploy
  • review
  • dast
  • staging
  • canary
  • production
  • incremental rollout 10%
  • incremental rollout 25%
  • incremental rollout 50%
  • incremental rollout 100%
  • performance
  • cleanup

.auto-deploy:
image: “registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v0.13.0

review:
extends: .auto-deploy
stage: deploy
script:
- auto-deploy download_chart
- auto-deploy ensure_namespace
- auto-deploy initialize_tiller
- auto-deploy create_secret
#- sed ‘s/dockercfg/dockerconfigjson/g’
#- sed ‘s/dockercfg/dockerconfigjson/g’ | kubectl replace -n “$KUBE_NAMESPACE” --force -f -
- auto-deploy deploy
- auto-deploy persist_environment_url
environment:
name: review/$CI_COMMIT_REF_NAME
url: http://$CI_PROJECT_ID-$CI_ENVIRONMENT_SLUG
on_stop: stop_review
artifacts:
paths: [environment_url.txt]
only:
refs:
- branches
- tags
kubernetes: active
except:
refs:
- master
variables:
- $REVIEW_DISABLED

stop_review:
extends: .auto-deploy
stage: .pre
variables:
GIT_STRATEGY: none
script:
- auto-deploy initialize_tiller
- auto-deploy delete
environment:
name: review/$CI_COMMIT_REF_NAME
action: stop
dependencies:
when: manual
allow_failure: true
only:
refs:
- branches
- tags
kubernetes: active
except:
refs:
- master
variables:
- $REVIEW_DISABLED

Staging deploys are disabled by default since

continuous deployment to production is enabled by default

If you prefer to automatically deploy to staging and

only manually promote to production, enable this job by setting

STAGING_ENABLED.

staging:
extends: .auto-deploy
stage: deploy
script:
- auto-deploy download_chart
- auto-deploy ensure_namespace
- auto-deploy initialize_tiller
- auto-deploy create_secret
#- sed ‘s/dockercfg/dockerconfigjson/g’
#- sed ‘s/dockercfg/dockerconfigjson/g’ | kubectl replace -n “$KUBE_NAMESPACE” --force -f -
- auto-deploy deploy
environment:
name: staging
url: http://$CI_PROJECT_PATH_SLUG-staging
only:
refs:
- master
kubernetes: active
variables:
- $STAGING_ENABLED

production: &production_template
extends: .auto-deploy
stage: deploy
script:
- auto-deploy download_chart
- auto-deploy ensure_namespace
- auto-deploy initialize_tiller
- auto-deploy create_secret
#- sed ‘s/dockercfg/dockerconfigjson/g’
#- sed ‘s/dockercfg/dockerconfigjson/g’ | kubectl replace -n “$KUBE_NAMESPACE” --force -f -
- auto-deploy deploy
- auto-deploy delete canary
- auto-deploy delete rollout
- auto-deploy persist_environment_url
environment:
name: production
url: http://$CI_PROJECT_PATH_SLUG
artifacts:
paths: [environment_url.txt]

production:
<<: *production_template
only:
refs:
- master
kubernetes: active
except:
variables:
- $STAGING_ENABLED
- $INCREMENTAL_ROLLOUT_ENABLED
- $INCREMENTAL_ROLLOUT_MODE

production_manual:
<<: *production_template
when: manual
allow_failure: false
only:
refs:
- master
kubernetes: active
variables:
- $STAGING_ENABLED
except:
variables:
- $INCREMENTAL_ROLLOUT_ENABLED
- $INCREMENTAL_ROLLOUT_MODE

This job implements incremental rollout on for every push to master.

rollout: &rollout_template
extends: .auto-deploy
script:
- auto-deploy download_chart
- auto-deploy ensure_namespace
- auto-deploy initialize_tiller
- auto-deploy create_secret
#- sed ‘s/dockercfg/dockerconfigjson/g’
#- sed ‘s/dockercfg/dockerconfigjson/g’ | kubectl replace -n “$KUBE_NAMESPACE” --force -f -
- auto-deploy deploy rollout ROLLOUT_PERCENTAGE - auto-deploy scale stable ((100-ROLLOUT_PERCENTAGE))
- auto-deploy persist_environment_url
environment:
name: production
#url: http://$CI_PROJECT_PATH_SLUG
#url: http://34.71.115.159
url: http://$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN
artifacts:
paths: [environment_url.txt]

.manual_rollout_template: &manual_rollout_template
<<: *rollout_template
stage: production
when: manual

This selectors are backward compatible mode with $INCREMENTAL_ROLLOUT_ENABLED (before 11.4)

only:
refs:
- master
kubernetes: active
variables:
- $INCREMENTAL_ROLLOUT_MODE == “manual”
- $INCREMENTAL_ROLLOUT_ENABLED
except:
variables:
- $INCREMENTAL_ROLLOUT_MODE == “timed”

.timed_rollout_template: &timed_rollout_template
<<: *rollout_template
when: delayed
start_in: 5 minutes
only:
refs:
- master
kubernetes: active
variables:
- $INCREMENTAL_ROLLOUT_MODE == “timed”

timed rollout 10%:
<<: *timed_rollout_template
stage: deploy
variables:
ROLLOUT_PERCENTAGE: 10

timed rollout 25%:
<<: *timed_rollout_template
stage: deploy
variables:
ROLLOUT_PERCENTAGE: 25

timed rollout 50%:
<<: *timed_rollout_template
stage: deploy
variables:
ROLLOUT_PERCENTAGE: 50

timed rollout 100%:
<<: *timed_rollout_template
<<: *production_template
stage: deploy
variables:
ROLLOUT_PERCENTAGE: 100

rollout 10%:
<<: *manual_rollout_template
stage: deploy
variables:
ROLLOUT_PERCENTAGE: 10

rollout 25%:
<<: *manual_rollout_template
stage: deploy
variables:
ROLLOUT_PERCENTAGE: 25

rollout 50%:
<<: *manual_rollout_template
stage: deploy
variables:
ROLLOUT_PERCENTAGE: 50

rollout 100%:
<<: *manual_rollout_template
<<: *production_template
allow_failure: false

ErrImagePull seems to suggest that GKE could not access the container image. Perhaps you can check that https://docs.gitlab.com/ee/user/project/deploy_tokens/#gitlab-deploy-token is enabled ?

it was enabled and reenabled multiple times - no use - it simply does not work as it should.
and eventually google people suggested that gitlab is broken - specifically - ABAC should be used. How to do that?

  • Maybe there is a networking problem? Do you perhaps have a firewall rule blocking access to external resources such as the registry.gitlab.com?
  • Maybe the image was not built correctly? Are you able to pull it locally?

definitely there is no network problem. I have tried to
docker login registry.gitlab.com/steamru/fnm-test-piankov/master:6065cbf6315deaafd2631984cac0405e66b049a2
Username: gitlab+deploy-token-164546
Password:
Error response from daemon: Get https://registry.gitlab.com/v2/: unauthorized: HTTP Basic: Access denied

from localhost - same result as from GKE.
and changing Deploy tokens and rerunning ci.yaml does not fix this auth problem.

– it is definitely not a firewall issue.

– I am running with TRACE: set -x and there are no errors on building.

– pull locally where? I can not even access it.