Please help fill in this template with all the details to help others help you more efficiently. Use formatting blocks for code, config, logs and ensure to remove sensitive data.
Problem to solve
I’m trying to build a multi-platform image (eg linux/amd64, linux/arm64/v8). I could post my gitlab-cicd.yml file but that will not be helpful. I’ve tried every single combination of suggestions online from the last 5 years. This includes setting driver to docker-container, building custom contexts, using --push and/or --pull in the docker buildx build command, and upgrading to docker:25.0.3 and others.
No matter what I try I cant get the image (which apparently builds fine) to either show in the local registry or push to the GitLab container registry. EVERY single piece of help online I’ve found where someone says ‘All I had to do was…’ has not worked for me. Can someone post a hello world .gitlab-ci.yml file that builds multi-arch images and posts them to a registry or gets them to show in a docker images command. I’m truly at a loss as to what is going on
BTW, using latest Docker Desktop on my Mac M3 … I can get multi-arch images to build and appear in a ‘docker images’ command. What is going on?
Describe your question in as much detail as possible:
What are you seeing, and how does that differ from what you expect to see?
Consider including screenshots, error messages, and/or other helpful visuals
Steps to reproduce
I tried to paste my .gitlab-ci.yml file here… but got a 422 error. I tried to upload/attach the file and still no luck. If you email swsides@txcorp.com I can send it directly. Sorry about that…
Add the CI/CD configuration from .gitlab-ci.yml and other configuration if relevant (e.g. docker-compose.yml). Alternatively, create a public GitLab.com example project that provides all necessary files to reproduce the question.
Versions
Please select whether options apply, and add the version information.
build:
stage: build
script:
- docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
- docker buildx create --name mybuilder --use
- docker buildx build --platform linux/amd64,linux/arm64/v8 --tag $CI_REGISTRY_IMAGE:latest --push .
I can suggest a basic setup for your .gitlab-ci.yml file that should help with building and pushing images that work on different types of computers (like Intel and ARM processors). This setup assumes that you already have Docker and a GitLab Runner ready for handling builds for different platforms.
Here’s a simplified explanation of what to do:
Use the latest version of Docker along with Docker-in-Docker to make sure you can use the newest features.
Before you start building your Docker images, sign in to the GitLab Container Registry using the login details GitLab provides automatically.
Use a special container called multiarch/qemu-user-static to allow you to build ARM images even if you’re not using an ARM computer.
Set up a new Docker buildx builder named mybuilder and start using it.
Build your Docker image so it can work on both amd64 (standard Intel and AMD processors) and arm64/v8 (ARM processors) and then push this image to the GitLab Container Registry.
Make sure your GitLab Runner is set up to run Docker commands and has the right permissions. Also, check that your Docker Desktop app is up to date and set up correctly to work with GitLab CI/CD. This setup is a starting point, and you might need to adjust it based on the specific errors you encounter or the particular needs of your project.
Thank you so much for the reply. The qemup-user-static container sounds familar… I have downloaded it already, but perhaps I didnt use it correctly. I’ll try it again with your other suggestions. If I build correctly with your suggestions… will the image appear if I do ‘docker images’? And… will I then be able to do an explicit ‘docker push …’? Or… should the ‘docker buildx build…’ line include ‘–push’ ?
Also… I have the latest Docker Desktop and builds there (on my mac) definitely work without too much trouble. Why did you mention getting Docker Desktop for fixing my GitLab pipeline on a shared GitLab runner?
And like all my other attempts… the image apparently builds but it pushes an ‘empty’ image to the registry. This is draining me of the will to live. Help?
From what I gather, there could be a few things worth double-checking:
First, It’s crucial to make sure your Dockerfile and the whole setup are ready to handle builds for different types of processors, like Intel and ARM. Sometimes, this means you have to pick base images or dependencies that play nice with both kinds of tech.
You’ll want to ensure that when you tell Docker to build your image, the path you give it includes everything your application needs. If the path is wrong or missing stuff, you might end up with a build that’s not what you expected.
Also, debugging might help to add more detailed logs to your GitLab CI/CD pipeline. More info could give you clues on where things are getting tangled.
From my angle, you seem pretty close to cracking this. Sometimes, breaking down each step and looking at them one by one can help pinpoint the problem.
Thanks for the reply. So I’ve made a very simple test Docker image that essentially just downloads an alpine:3.14 image and sets me as the MAINTAINER. This builds multi-arch fine on my mac with latest Docker Desktop. The full log from the build on a shared linux gitlab runner. The docker buildx build … command I’m using for the GitLab pipeline has --push. So it directly pushes an image to the gitlab registry… but its simply empty. Not sure why this is so easy on Docker Desktop and so hard in GitLab pipeline. Does this log have any extra info that is helpful?
[0KRunning with gitlab-runner 16.9.1 (782c6ecb)e[0;m
e[0K on blue-3.saas-linux-large-amd64.runners-manager.gitlab.com/default VxfMyqp3, system ID: s_a57dbf17d1f2e[0;m
e[0K feature flags: FF_USE_IMPROVED_URL_MASKING:truee[0;m
section_start:1710184698:resolve_secrets
e[0Ke[0Ke[36;1mResolving secretse[0;me[0;m
section_end:1710184698:resolve_secrets
e[0Ksection_start:1710184698:prepare_executor
e[0Ke[0Ke[36;1mPreparing the "docker+machine" executore[0;me[0;m
e[0KUsing Docker executor with image docker:25.0.3 ...e[0;m
e[0KStarting service docker:25.0.3-dind ...e[0;m
e[0KPulling docker image docker:25.0.3-dind ...e[0;m
e[0KUsing docker image sha256:e5fbe8997fd9ff8f2894874c12a4fbc5fc1bb42d08a7db8433bbe09066562a2a for docker:25.0.3-dind with digest docker@sha256:0d70c541ee98e66b8f7ece8c0e9f7910732466e337a9087c2ac2868ef0775092 ...e[0;m
e[0KWaiting for services to be up and running (timeout 30 seconds)...e[0;m
e[0KPulling docker image docker:25.0.3 ...e[0;m
e[0KUsing docker image sha256:e5fbe8997fd9ff8f2894874c12a4fbc5fc1bb42d08a7db8433bbe09066562a2a for docker:25.0.3 with digest docker@sha256:0d70c541ee98e66b8f7ece8c0e9f7910732466e337a9087c2ac2868ef0775092 ...e[0;m
section_end:1710184727:prepare_executor
e[0Ksection_start:1710184727:prepare_script
e[0Ke[0Ke[36;1mPreparing environmente[0;me[0;m
Running on runner-vxfmyqp3-project-40608643-concurrent-0 via runner-vxfmyqp3-s-l-l-amd64-1710184654-d93e88b0...
section_end:1710184727:prepare_script
e[0Ksection_start:1710184727:get_sources
e[0Ke[0Ke[36;1mGetting source from Git repositorye[0;me[0;m
e[32;1mFetching changes with git depth set to 20...e[0;m
Initialized empty Git repository in /builds/txcorp/chemistream/chemistream-hpc-images/.git/
e[32;1mCreated fresh repository.e[0;m
e[32;1mChecking out f70d20a9 as detached HEAD (ref is refs/merge-requests/38/head)...e[0;m
e[32;1mSkipping Git submodules setupe[0;m
e[32;1m$ git remote set-url origin "${CI_REPOSITORY_URL}"e[0;m
section_end:1710184739:get_sources
e[0Ksection_start:1710184739:step_script
e[0Ke[0Ke[36;1mExecuting "step_script" stage of the job scripte[0;me[0;m
e[0KUsing docker image sha256:e5fbe8997fd9ff8f2894874c12a4fbc5fc1bb42d08a7db8433bbe09066562a2a for docker:25.0.3 with digest docker@sha256:0d70c541ee98e66b8f7ece8c0e9f7910732466e337a9087c2ac2868ef0775092 ...e[0;m
e[32;1m$ export CS_REPO="registry.gitlab.com/txcorp/chemistream/chemistream-hpc-images"e[0;m
e[32;1m$ echo "CS_REPO=$CS_REPO"e[0;m
CS_REPO=registry.gitlab.com/txcorp/chemistream/chemistream-hpc-images
e[32;1m$ echo "CS_IMAGES_VERSION=$CS_IMAGES_VERSION"e[0;m
CS_IMAGES_VERSION=latest
e[32;1m$ docker infoe[0;m
Client:
Version: 25.0.3
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.1
Path: /usr/local/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.24.6
Path: /usr/local/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 25.0.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
Kernel Version: 5.4.109+
Operating System: Alpine Linux v3.19 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.37GiB
Name: feaa4907b966
ID: 9e3cdf19-640b-472c-8527-f1532498c71d
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
Access to the remote API is equivalent to root access on the host. Refer
to the 'Docker daemon attack surface' section in the documentation for
more information: https://docs.docker.com/go/attack-surface/
e[32;1m$ apk updatee[0;m
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
v3.19.1-217-g0e9c38378f6 [https://dl-cdn.alpinelinux.org/alpine/v3.19/main]
v3.19.1-219-g2e1eeb11dbd [https://dl-cdn.alpinelinux.org/alpine/v3.19/community]
OK: 22987 distinct packages available
e[32;1m$ apk add gite[0;m
OK: 42 MiB in 71 packages
e[32;1m$ echo ""e[0;m
e[32;1m$ echo ""e[0;m
e[32;1m$ echo "Buildx create for multi-platform builds"e[0;m
Buildx create for multi-platform builds
e[32;1m$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes --install alle[0;m
Unable to find image 'multiarch/qemu-user-static:latest' locally
latest: Pulling from multiarch/qemu-user-static
205dae5015e7: Pulling fs layer
816739e52091: Pulling fs layer
30abb83a18eb: Pulling fs layer
0657daef200b: Pulling fs layer
30c9c93f40b9: Pulling fs layer
0657daef200b: Waiting
30c9c93f40b9: Waiting
816739e52091: Verifying Checksum
816739e52091: Download complete
30abb83a18eb: Verifying Checksum
30abb83a18eb: Download complete
205dae5015e7: Verifying Checksum
205dae5015e7: Download complete
205dae5015e7: Pull complete
0657daef200b: Verifying Checksum
0657daef200b: Download complete
816739e52091: Pull complete
30abb83a18eb: Pull complete
0657daef200b: Pull complete
30c9c93f40b9: Verifying Checksum
30c9c93f40b9: Download complete
30c9c93f40b9: Pull complete
Digest: sha256:fe60359c92e86a43cc87b3d906006245f77bfc0565676b80004cc666e4feb9f0
Status: Downloaded newer image for multiarch/qemu-user-static:latest
getopt: unrecognized option '--install'
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Setting /usr/bin/qemu-sparc-static as binfmt interpreter for sparc
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-sparc64-static as binfmt interpreter for sparc64
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
Setting /usr/bin/qemu-hexagon-static as binfmt interpreter for hexagon
e[32;1m$ docker buildx create --platform linux/amd64,linux/arm64 --driver=docker-container --name=mybuilder --use --bootstrape[0;m
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 1.9s done
#1 creating container buildx_buildkit_mybuilder0
#1 creating container buildx_buildkit_mybuilder0 1.6s done
#1 DONE 3.5s
mybuilder
e[32;1m$ echo ""e[0;m
e[32;1m$ echo "Logging onto GitLab registry"e[0;m
Logging onto GitLab registry
e[32;1m$ docker login registry.gitlab.com -u "$GL_USERNAME" -p "$GL_TOKEN"e[0;m
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
e[32;1m$ docker buildx lse[0;m
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
mybuilder * docker-container
mybuilder0 tcp://docker:2375 running v0.12.5 linux/amd64*, linux/arm64*, linux/amd64/v2, linux/amd64/v3, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default docker
default default running v0.12.5 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6, linux/ppc64, linux/mips64le, linux/mips64
e[32;1m$ docker context liste[0;m
NAME DESCRIPTION DOCKER ENDPOINT ERROR
default * Current DOCKER_HOST based configuration tcp://docker:2375
Warning: DOCKER_HOST environment variable overrides the active context. To use a context, either set the global --context flag, or unset DOCKER_HOST environment variable.
e[32;1m$ docker --versione[0;m
Docker version 25.0.3, build 4debf41
e[32;1m$ cd $CI_PROJECT_DIRe[0;m
e[32;1m$ cd $CI_PROJECT_DIR/csbasee[0;m
e[32;1m$ sed -i "s+<tag_name>+$CS_IMAGES_VERSION+" test.dkre[0;m
e[32;1m$ docker buildx build --push --platform=linux/amd64,linux/arm64 -t "$CS_REPO/test2:$CS_IMAGES_VERSION" -f test.dkr --builder=mybuilder .e[0;m
#0 building with "mybuilder" instance using docker-container driver
#1 [internal] load build definition from test.dkr
#1 transferring dockerfile: 882B done
#1 DONE 0.0s
#2 [linux/amd64 internal] load metadata for docker.io/library/alpine:3.14
#2 DONE 0.9s
#3 [linux/arm64 internal] load metadata for docker.io/library/alpine:3.14
#3 DONE 0.9s
#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s
#5 [linux/amd64 1/2] FROM docker.io/library/alpine:3.14@sha256:0f2d5c38dd7a4f4f733e688e3a6733cb5ab1ac6e3cb4603a5dd564e5bfb80eed
#5 resolve docker.io/library/alpine:3.14@sha256:0f2d5c38dd7a4f4f733e688e3a6733cb5ab1ac6e3cb4603a5dd564e5bfb80eed 0.0s done
#5 DONE 0.1s
#6 [linux/arm64 1/2] FROM docker.io/library/alpine:3.14@sha256:0f2d5c38dd7a4f4f733e688e3a6733cb5ab1ac6e3cb4603a5dd564e5bfb80eed
#6 resolve docker.io/library/alpine:3.14@sha256:0f2d5c38dd7a4f4f733e688e3a6733cb5ab1ac6e3cb4603a5dd564e5bfb80eed 0.0s done
#6 sha256:422ed46b1a92579f7c475c0c19fade6880a8d98f23a2b4ccfb77c265d4f72dfc 2.73MB / 2.73MB 0.1s done
#6 extracting sha256:422ed46b1a92579f7c475c0c19fade6880a8d98f23a2b4ccfb77c265d4f72dfc 0.1s done
#6 DONE 0.3s
#5 [linux/amd64 1/2] FROM docker.io/library/alpine:3.14@sha256:0f2d5c38dd7a4f4f733e688e3a6733cb5ab1ac6e3cb4603a5dd564e5bfb80eed
#5 sha256:f7dab3ab2d6ec29aa28769bec35331fb485b5837501b1e8556413d8b5a79c9c8 2.83MB / 2.83MB 0.1s done
#5 extracting sha256:f7dab3ab2d6ec29aa28769bec35331fb485b5837501b1e8556413d8b5a79c9c8 0.1s done
#5 DONE 0.3s
#7 [linux/arm64 2/2] WORKDIR /home/hpcuser
#7 DONE 0.1s
#8 [linux/amd64 2/2] WORKDIR /home/hpcuser
#8 DONE 0.1s
#9 exporting to image
#9 exporting layers 0.1s done
#9 exporting manifest sha256:ec04d524b8d7639e3e5ecb6d9b22d559ca9bcdbad9f6edb4b825682280bf2346 0.0s done
#9 exporting config sha256:a369e285ba92cea82849c4b85ea387127ca523200c35d033d2c817cc5bb82377 0.0s done
#9 exporting attestation manifest sha256:4dfabe46cf8ed408a59fa0e95e9068772e38db77eacb24b8aff74b15c90afb46 0.0s done
#9 exporting manifest sha256:e376e409c64bad4f211ba504223948754930910be67651df657896bb0e784b64 0.0s done
#9 exporting config sha256:ced7250629481f42797b4cfa3981bda54be708966c9b5c06c9e9320035115a34 0.0s done
#9 exporting attestation manifest sha256:e0a47ac2bb0429710d17098f348efddb60702747511746b34b41512386d612b0 0.0s done
#9 exporting manifest list sha256:5df3dacae70ec0e6e26b2195687845a6b6681a7f0247a713ca6b9239b342cc6c
#9 ...
#10 [auth] txcorp/chemistream/chemistream-hpc-images/test2:pull,push token for registry.gitlab.com
#10 DONE 0.0s
#9 exporting to image
#9 exporting manifest list sha256:5df3dacae70ec0e6e26b2195687845a6b6681a7f0247a713ca6b9239b342cc6c 0.0s done
#9 pushing layers
#9 pushing layers 3.8s done
#9 pushing manifest for registry.gitlab.com/txcorp/chemistream/chemistream-hpc-images/test2:latest@sha256:5df3dacae70ec0e6e26b2195687845a6b6681a7f0247a713ca6b9239b342cc6c
#9 pushing manifest for registry.gitlab.com/txcorp/chemistream/chemistream-hpc-images/test2:latest@sha256:5df3dacae70ec0e6e26b2195687845a6b6681a7f0247a713ca6b9239b342cc6c 0.5s done
#9 DONE 4.6s
e[32;1m$ docker imagese[0;m
REPOSITORY TAG IMAGE ID CREATED SIZE
moby/buildkit buildx-stable-1 480495983c47 5 weeks ago 172MB
multiarch/qemu-user-static latest 3539aaa87393 13 months ago 305MB
section_end:1710184757:step_script
e[0Ksection_start:1710184757:upload_artifacts_on_success
e[0Ke[0Ke[36;1mUploading artifacts for successful jobe[0;me[0;m
e[32;1mUploading artifacts...e[0;m
/builds/txcorp/chemistream/chemistream-hpc-images: found 1269 matching artifact files and directoriese[0;m
.git: excluded 1 files e[0;m
.git/**/*: excluded 19 files e[0;m
e[0;33mWARNING: Upload request redirected e[0;m e[0;33mlocatione[0;m=https://gitlab.com/api/v4/jobs/6368170292/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day e[0;33mnew-urle[0;m=https://gitlab.com
e[0;33mWARNING: Retrying... e[0;m e[0;33mcontexte[0;m=artifacts-uploader e[0;33merrore[0;m=request redirected
Uploading artifacts as "archive" to coordinator... 201 Createde[0;m ide[0;m=6368170292 responseStatuse[0;m=201 Created tokene[0;m=glcbt-65
section_end:1710184776:upload_artifacts_on_success
e[0Ksection_start:1710184776:cleanup_file_variables
e[0Ke[0Ke[36;1mCleaning up project directory and file based variablese[0;me[0;m
section_end:1710184777:cleanup_file_variables
e[0Ke[32;1mJob succeedede[0;m
I tried a few more things and looked at error logs and didnt find anything helpful. So I reluctantly decided to give up and return to this issue when gitlab docker enviornment on linux shared runners is as good as mac Docker Desktop.
So, I decided to build arm64 images on my mac (which has been working all along) and explicitly push them to the gitlab registry and tag them so I knew which ones to pull… relying on ‘caller logic’ to decide which image to pull. So I would (from my local mac Docker Desktop):
build arm64 image
tag arm64 image with gitlab registry name
push to gitlab registry
However this image appeared in my gitlab registry with 0 size!!! Moreover, I then removed my local docker arm64 image and pulled from my gitlab registry onto my local machine and it shows up with size 4.93GB as it should. So somehow, I think that the gitlab registry is not calculating the size correctly.
Does this sound like something that is possible? Now I’m beginning to think I was creating the images in my gitlab ci-cd pipelines correctly all along… but it wasnt showing with the correct size and I interpreted this as indicating it had built in error.
Hi there! I know this is a few weeks old, and hopefully you found some workaround in the meantime.
I wanted to throw my solution into the hat in case it helps someone – I spent many hours figuring out how to do multiplatform builds on shared gitlab runners. Was in the process of opening a ticket for a broken DinD shared runner and found this thread.
Thanks for the help. We are reworking our entire build design for our compute images to upgrade from Centos7 which is approaching EOL. I’ll attack this problem again once this reworking is done. We can close this thread for now. Thanks again for everyone’s time. Cheers,
Please help fill in this template with all the details to help others help you more efficiently. Use formatting blocks for code, config, logs and ensure to remove sensitive data.