Same Code Failing Across Different Repositories

jwilliams16 · February 24, 2022, 1:47am

Same Code Fails In Gitlab CI When In Different Repositories

I have a repository with the same exact code, everything except git commit history. In a legacy repository it fails in CI build steps. When it is in a new repository. that same exact code is passing.
Both of these projects are in the same gitlab project folder and should be inheriting the same CICD variables found in the repositories settings

gitlab.com/linkto/repo/-/settings/ci_cd

 info [bazel] success Already up-to-date.
 info [bazel] $ node scripts/kbn patch_native_modules
 info [bazel] INFO: Analyzed target //packages:build (1910 packages loaded, 8454 targets configured).
 info [bazel] INFO: Found 1 target...
 info [bazel] ERROR: /kibana/packages/elastic-datemath/BUILD.bazel:61:11: Copying files failed: missing input file '//packages/elastic-datemath:README.md'
 info [bazel] Target //packages:build failed to build
 info [bazel] Use --verbose_failures to see the command lines of failed build steps.
 info [bazel] ERROR: /kibana/packages/elastic-datemath/BUILD.bazel:61:11 Copying files failed: 1 input file(s) do not exist
 info [bazel] INFO: Elapsed time: 13.816s, Critical Path: 0.45s
 info [bazel] INFO: 93 processes: 29 internal, 64 linux-sandbox.
 info [bazel]

In the working case that same step is passing

info [bazel] $ node scripts/kbn patch_native_modules
 info [bazel] INFO: Analyzed target //packages:build (1911 packages loaded, 8454 targets configured).
 info [bazel] INFO: Found 1 target...
 info [bazel] INFO: From Assembling npm types package packages/kbn-securitysolution-utils/npm_module_types:
 info [bazel] Analysis will use the bundled TypeScript version 4.5.3
 info [bazel] INFO: From Assembling npm types package packages/kbn-securitysolution-list-constants/npm_module_types:
 info [bazel] Analysis will use the bundled TypeScript version 4.5.3
 info [bazel] INFO: From Assembling npm types package packages/kbn-i18n/npm_module_types:
 info [bazel] Analysis will use the bundled TypeScript version 4.5.3
 info [bazel] INFO: From Assembling npm types package packages/kbn-config-schema/npm_module_types:
 info [bazel] Analysis will use the bundled TypeScript version 4.5.3
 info [bazel] INFO: From Assembling npm types package packages/kbn-analytics/npm_module_types:
 info [bazel] Analysis will use the bundled TypeScript version 4.5.3
 info [bazel] INFO: From Assembling npm types package packages/kbn-securitysolution-hook-utils/npm_module_types:
 info [bazel] Analysis will use the bundled TypeScript version 4.5.3
 info [bazel] INFO: From Assembling npm types package packages/kbn-ui-theme/npm_module_types:
 info [bazel] Analysis will use the bundled TypeScript version 4.5.3
 info [bazel] INFO: From Assembling npm types package packages/kbn-apm-utils/npm_module_types:
 info [bazel] Analysis will use the bundled TypeScript version 4.5.3
 info [bazel] INFO: From Assembling npm types package packages/elastic-datemath/npm_module_types:
 info [bazel] Analysis will use the bundled TypeScript version 4.5.3
....

Sorry if this is a bit vague, I’ll see if I can get an open source demonstration of this working, but are there any idea’s out there why the gitlab CI is not idompotent here? I would expect with the same cicd settings and same code they should behave the same regardless of commit history. Thanks

sdunnin · February 24, 2022, 2:02pm

Is there something different or unique about the failing repository that would affect the git checkout? Sorry, nothing tangible, just throwing out ideas.

snim2 · February 24, 2022, 2:13pm

And are both repos using the same runners?

jwilliams16 · February 24, 2022, 3:40pm

No that’s a fair question. I have checked that the committed content is the same by pulling from remote in separate folders and essentially overlaying the entire contents of the repo to see if there are differences arise and there are none. So I’m very confident that they are the exact same

jwilliams16 · February 24, 2022, 3:45pm

Both repo’s are hosted in Saas gitlab cloud land and should be using the same kind of runners. I’ve checed the settings CI/CD and nothing popped out at me as a difference between the two.

Is there some way I could get a yaml or json of the different settings between the repos? Would be a lot easier to spot the difference than crawling the GUI

dnsmichi · February 25, 2022, 9:25pm

Can you share the .gitlab-ci.yml and the scripts involved? The missing package sounds like a caching issue, or an intermittent network download failure. Maybe bazel can be turned into debug logging to see more as well.

jwilliams16 · February 28, 2022, 3:33pm

@dnsmichi the gitlab file is pretty simple

image: debian:latest

variables:
  NVM_VER: "0.39.1"

build:
  stage: build
  before_script:
    - apt-get update
    - apt-get -qq -y install wget #install wget
    - wget -qO- https://raw.githubusercontent.com/creationix/nvm/v${NVM_VER}/install.sh | bash #install nvm
    - export NVM_DIR="$HOME/.nvm" && . "$NVM_DIR/nvm.sh" --no-use #load nvm
    - eval "[ -f .nvmrc ] && nvm install || nvm install stable" #install node, use .nvmrc file exists, else latest node
  script:
    - npm install -g yarn
    - npm install -g @bazel/bazelisk
    - yarn
    - yarn kbn bootstrap --oss --prefer-offline --verbose

    # Build Kibana distributable skipping unneeded packages, is working locally
    - yarn build --skip-os-packages --no-debug --skip-docker-ubi 
    - ls ./build/default

We realized that the error message is missing the README.md, and I noticed on the remote that README.md was actually in lowercase, so would be missing in the CI build. This is weird because locally it was the all uppercase version.

I’m switching git config core.ignoreCase setting to false and taking another crack at this and hopefully everything is properly aligned.