Not a whole lot of detail to give, I don’t know what log files to check out since nothing is displayed on the web ui and no log files exist in /var/opt/gitlab/gitlab-ci/builds/
Here is the .gitlab-ci.yml file:
stages:
- packer:engineering
- terraform-plan:engineering
- terraform-apply:engineering
- recycle:engineering
- packer:production
- terraform-plan:production
- terraform-apply:production
- recycle:production
variables:
engineering_terraform_bucket: vandelay-engineering-terraform
engineering_terraform_path: infrastructure/terraform/common/website
production_terraform_bucket: vandelay-terraform
production_terraform_path: infrastructure/terraform/common/website
GIT_SUBMODULE_STRATEGY: normal
GIT_STRATEGY: clone
packer:engineering:
image: docker.vandelay.engineering/costanza/ci/packer-ansible:latest
stage: packer:engineering
script:
- git submodule update --remote
- cd infrastructure
- packer build packer/website.engineering.json
variables:
AWS_ACCESS_KEY_ID: $engineering_packer_key
AWS_SECRET_ACCESS_KEY: $engineering_packer_secret
only:
- master
terraform-plan:engineering:
image: docker.vandelay.engineering/costanza/ci/terraform:0.10.2
stage: terraform-plan:engineering
script:
- git submodule update --remote
- cd ${engineering_terraform_path}
- terraform init -input=false -backend-config bucket=${engineering_terraform_bucket}
- terraform plan -input=false -var-file=engineering.tfvars -out engineering.plan
variables:
TF_WORKSPACE: engineering
AWS_ACCESS_KEY_ID: $engineering_terraform_key
AWS_SECRET_ACCESS_KEY: $engineering_terraform_secret
artifacts:
paths:
- ${engineering_terraform_path}/engineering.plan
expire_in: 5m
only:
- master
terraform-apply:engineering:
image: docker.vandelay.engineering/costanza/ci/terraform:0.10.2
stage: terraform-apply:engineering
script:
- git submodule update --remote
- cd ${engineering_terraform_path}
- terraform init -input=false -backend-config bucket=${engineering_terraform_bucket}
- terraform apply engineering.plan
variables:
TF_WORKSPACE: engineering
AWS_ACCESS_KEY_ID: $engineering_terraform_key
AWS_SECRET_ACCESS_KEY: $engineering_terraform_secret
dependencies:
- terraform-plan:engineering
only:
- master
recycle:engineering:
image: docker.vandelay.engineering/costanza/instance-recycler:latest
stage: recycle:engineering
script:
- recycle-instances --asg website
variables:
AWS_ACCESS_KEY_ID: $engineering_terraform_key
AWS_SECRET_ACCESS_KEY: $engineering_terraform_secret
only:
- master
packer:production:
image: docker.vandelay.engineering/costanza/ci/packer-ansible:latest
stage: packer:production
script:
- git submodule update --remote
- cd infrastructure
- packer build packer/website.production.json
variables:
AWS_ACCESS_KEY_ID: $production_packer_key
AWS_SECRET_ACCESS_KEY: $production_packer_secret
only:
- master
terraform-plan:production:
image: docker.vandelay.engineering/costanza/ci/terraform:0.10.2
stage: terraform-plan:production
script:
- ls -alh
- git submodule update --remote
- cd ${production_terraform_path}
- terraform init -input=false -backend-config bucket=${production_terraform_bucket}
- terraform plan -input=false -var-file=production.tfvars -out production.plan
- >-
curl
-X POST https://hooks.slack.com/services/ -d '
{
"channel": "#deploy",
"username": "GitlabCI",
"text": "The latest build for `website` is ready for deployment at <'"$CI_PROJECT_URL"'/pipelines/'"$CI_PIPELINE_ID"'>",
"icon_emoji": ":clippy:"
}
'
-H "Content-Type: application/json"
variables:
CI_DEBUG_TRACE: "true"
TF_WORKSPACE: production
AWS_ACCESS_KEY_ID: $production_terraform_key
AWS_SECRET_ACCESS_KEY: $production_terraform_secret
artifacts:
paths:
- ${production_terraform_path}/production.plan
expire_in: 1h
only:
- master
terraform-apply:production:
image: docker.vandelay.engineering/costanza/ci/terraform:0.10.2
stage: terraform-apply:production
script:
- git submodule update --remote
- cd ${production_terraform_path}
- terraform init -input=false -backend-config bucket=${production_terraform_bucket}
- terraform apply production.plan
variables:
TF_WORKSPACE: production
AWS_ACCESS_KEY_ID: $production_terraform_key
AWS_SECRET_ACCESS_KEY: $production_terraform_secret
dependencies:
- terraform-plan:production
only:
- master
when: manual
allow_failure: false
recycle:production:
image: docker.vandelay.engineering/costanza/instance-recycler:latest
stage: recycle:production
script:
- recycle-instances --asg website
variables:
AWS_ACCESS_KEY_ID: $production_terraform_key
AWS_SECRET_ACCESS_KEY: $production_terraform_secret
only:
- master
In the middle of the day the terraform-plan:oho stopped working. Things go from pending to failed quite quickly but then fail with a duration of 0 seconds.
Where should I start looking for further troubleshooting?