Gitlab CI job is failing with a "fatal: reference is not a tree" error

I have a gitlab ci job that keeps failing with a “fatal: reference is not a tree” error. Google and IRC have turned up nothing.

My .gitlab-ci.yml file:

  - build
  - test
  - deploy

  stage: deploy
    - master
    - bash scripts/deploy_master
    - tagname

My scripts/deploy_master file:

localuser is the user that owns the directory the project lives in.

ssh -t -t -o 'StrictHostKeyChecking no' -o 'PasswordAuthentication no' -o 'PubkeyAuthentication yes' -i '/home/gitlab-runner/.ssh/localuser.key' localuser@localhost <<EOF
  cd /home/localuser/path/to/project
  git fetch origin
  git reset --hard origin/master
  git submodule update --init

My runner is a shell runner running on the server I’m deploying code to.

The output I get:

gitlab-ci-multi-runner 0.6.0 (4c26bc5)
Using Shell executor...
Running on hostname...

Fetching changes...
Checking out dbed0c03 as master...
fatal: reference is not a tree: <ssha hash>

ERROR: Build failed with: exit status 1

All my google searches for “fatal: reference is not a tree” come up with articles saying I haven’t pushed the referenced commit to the submodules project. Which is not the case here. Plus, the reference in the error is the reference of the projects commit, not the submodules.

Note that if I run the commands in the deploy script manually, everything works fine.

Any thoughts?

Server info

My gitlab server is using the Omnibus package. 8.0.3. OS is Ubuntu 14.04 64bit.

The server the runner is on, and I’m trying to deploy code to, is running Ubuntu 12.04, and the runner is installed from the repository via apt.

That error happens if you rebase (squash), or amend a commit.

ssh into your gitlab runner and cd to the build directory and run a git status . You probably have a floating head.

Workaround is to discard local changes on the git repo on the runner. Also don’t run tests on branches where users have squashed or rebased commits.

I don’t think I’ve ever rebased or amended this repo. I certainly have not done so recently.

From the build repo:

root@hostname:/home/gitlab-runner/../project# sudo -u gitlab-runner git status
# Not currently on any branch.
nothing to commit (working directory clean)

I just tried:

root@hostname:/home/gitlab-runner/../project# sudo -u gitlab-runner git reset --hard
HEAD is now at 6c5f029 commit message here.

Then I hit the retry button on the latest commit in CI, and got the same error as last time.

I just tried removing the /home/gitlab-runner/builds/project directory. Then pushed a new commit. And the build succeeded.

Any idea what actually happened?

Ok, now I’m even more confused. I just pushed another change and got the “reference is not a tree” error again. Deleting the build in gitlab-runners directory and hitting the retry button fixed it.

Any ideas?

Edit to avoid spamming replies…

I tried switching the job to a runner running on the gitlab server. That actually worked. Two commits in a row were built successfully.

I already mentioned my the OS version differences. But could different versions of git have something to do with it?

The Gitlab server has git 1.9.1

The other server has git

I have the same problem.

fatal: reference is not a tree:

I’m getting this fatal: reference is not a tree also on all builds that are configured to update using git fetch (which is the default setting for Gitlab CI).

Once I change it to git clone, my builds no longer fail.

It’s probably existing gitlab error with wrongly done fetch. Confirm clone as workaround.

Downgrading to gitlab-runner 0.5.0 fixed this issue for me.