Packer Job Fails Whenever Run Via Schedule (works when run manually)

I have a perplexing problem, which doesn’t look like it’s .gitlab-ci.yml or runner related, or similar to any other problem I can see reported.

I have a scheduled CI job which runs Packer. It fails every time the schedule starts it, but works just fine if I run it myself (even if I set CI_PIPELINE_SOURCE=schedule). The weird thing is that the problem is inside Packer - it’s during an apt update task inside Packer - so running on an AWS server, completely unrelated to Gitlab, or its runner.

The general flow is:

  • Gitlab kicks off the pipeline
  • The first step in the pipeline is to get some AWS temporary credentials - this step works
  • The second step is to run Packer and give it some tasks
    • Packer start up correctly
    • Packer creates an AWS EC2 instance, and waits for it to start up
    • Packer SSHes onto the instance
    • Packer runs apt update → This fails on a schedule, but works manually
  • Since a pipeline step has failed, a third step runs to cleanup (which works correctly)

The gitlab-ci.yml config for the packer step is:

packer-job:   
  stage: packer  
  image:
    name: hashicorp/packer:latest
    entrypoint: [""]
  script:
    - set
    - apk add gpg
    - cd ubuntu-20.04
    - cp ${NEXUS_APT_PACKAGE_SIGNING_KEY_FILE} apt-key
    - gpg --dearmour apt-key
    - packer init .
    - packer build -var ci_build_id=${CI_PIPELINE_ID} .
  needs:
    - job: aws-authentication-job
      artifacts: true
  tags:
    - docker

(it’s failing in the packer build... step)

The pipeline log looks like this:

...
$ packer build -var ci_build_id=${CI_PIPELINE_ID} .
ubuntu-20.04-golden.amazon-ebs.ubuntu: output will be in this color.
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Prevalidating any provided VPC information
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Prevalidating AMI Name: iothic_ubuntu_20.04_golden_20220610_15.03
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Found Image ID: ami-0d2a4a5d69e46ea0b
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Creating temporary keypair: packer_62a35d47-2521-67dd-b015-1de9178cddef
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Creating temporary security group for this instance: packer_62a35d4a-870e-5cab-e4eb-70469e99620c
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Launching a source AWS instance...
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Adding tag: "CIBuildId": "1511"
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Instance ID: i-0ecded0731b50c16a
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Waiting for instance (i-0ecded0731b50c16a) to become ready...
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Using SSH communicator to connect: 10.0.36.102
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Waiting for SSH to become available...
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Connected to SSH!
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Uploading apt-key.gpg => /tmp/apt-key.gpg
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Uploading apt-key => /tmp/apt-key
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Uploading iops.list-focal => /tmp/iops.list
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Provisioning with shell script: /tmp/packer-shell1856619453
==> ubuntu-20.04-golden.amazon-ebs.ubuntu:
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
==> ubuntu-20.04-golden.amazon-ebs.ubuntu:
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:5 https://nexus.services.iothic.dev/repository/devops-apt-focal focal InRelease [2109 B]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [8628 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:7 http://archive.ubuntu.com/ubuntu focal/universe Translation-en [5124 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 c-n-f Metadata [265 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:9 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [144 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:10 http://archive.ubuntu.com/ubuntu focal/multiverse Translation-en [104 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:11 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 c-n-f Metadata [9136 B]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [928 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.8 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:15 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:16 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7336 B]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [596 B]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:18 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [44.5 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:19 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.9 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:20 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [980 B]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:21 http://archive.ubuntu.com/ubuntu focal-backports/restricted amd64 c-n-f Metadata [116 B]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:22 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [23.7 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:23 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.9 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [860 B]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:25 http://archive.ubuntu.com/ubuntu focal-backports/multiverse amd64 c-n-f Metadata [116 B]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:26 https://nexus.services.iothic.dev/repository/devops-apt-focal focal/main amd64 Packages [12.9 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:27 https://nexus.services.iothic.dev/repository/devops-apt-focal focal/main all Packages [462 B]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:28 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1544 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [264 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:30 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1001 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [142 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:32 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [707 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:33 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [127 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:34 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.5 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:35 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [22.2 kB]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:36 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5376 B]
    ubuntu-20.04-golden.amazon-ebs.ubuntu: Get:37 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [512 B]
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: E: Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal-updates_restricted_cnf_Commands-amd64 - open (2: No such file or directory)
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: Traceback (most recent call last):
==> ubuntu-20.04-golden.amazon-ebs.ubuntu:   File "/usr/lib/cnf-update-db", line 27, in <module>
==> ubuntu-20.04-golden.amazon-ebs.ubuntu:     col.create(db)
==> ubuntu-20.04-golden.amazon-ebs.ubuntu:   File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 95, in create
==> ubuntu-20.04-golden.amazon-ebs.ubuntu:     self._fill_commands(con)
==> ubuntu-20.04-golden.amazon-ebs.ubuntu:   File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 141, in _fill_commands
==> ubuntu-20.04-golden.amazon-ebs.ubuntu:     raise subprocess.CalledProcessError(returncode=sub.returncode,
==> ubuntu-20.04-golden.amazon-ebs.ubuntu: subprocess.CalledProcessError: Command '/usr/lib/apt/apt-helper cat-file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal-updates_restricted_cnf_Commands-amd64' returned non-zero exit status 100.
...

I’m really lost how this could even be a thing, let alone where to look. All I can think is that maybe there’s a different environment variable or something, but even then I can’t see how such a thing would get into an EC2 instance started by a process which is running on a runner…!?

I appreciate this may not be gitlab directly, but any clues or ideas would be most welcome!