Src refspec master does not match any

werther99 · March 16, 2019, 2:53pm

Dear GitLab community,

out of sudden my deployment jobs are failing with message “src refspec master does not match any”.
I did some research and I have absolutely no idea why.
However, when I compare current deployment job log with previous one I see this difference:

Current (Not working):
Running on runner-72989761-project-4417089-concurrent-0 via runner-72989761-srm-1552747221-b30389b8…
Initialized empty Git repository in /builds/xxx
Fetching changes…
Created fresh repository.
From https://gitlab.com/xxx

[new branch] master -> origin/master
Checking out 3e494a44 as master…

Previous:
Running on runner-ed2dce3a-project-4417089-concurrent-0 via runner-ed2dce3a-srm-1533052716-d10a913c…
Cloning repository…
Cloning into '/builds/xxx
Checking out 14e42d69 as master…

Why it is suddenly initializing an empty git repository instead of just cloning?
Any idea will be appreciated.

Thank you.

son.dang · March 18, 2019, 1:55am

I have the same issue, it’s just happened currently when was deploying React to Heroku

werther99 · March 18, 2019, 7:58am

I have noticed that deployments are not actually failing out of sudden but it happens after Gitlab Runner was updated to ver. 11.9+

For some reason the latest version of runner is always initializing an empty git repository instead of just cloning.

Any idea how to solve this?

ArroTech · March 19, 2019, 7:53am

Hi, we are having the exact same problem as described by werther99, which started a few days ago. We’d much appreciate any help in resolving this issue.

rarewave · March 23, 2019, 11:29pm

Hi, I have had the same problem over the past week.

I have been using ssh / git deployments with gitlab runners for the past couple of years without any issues.

Basically a push is being made to a bare git folder on my servers. Once the push has been made to the server, it triggers a hook to deploy to a public repository on the server.

Exactly the same errors as everyone else.

steveazz · March 25, 2019, 8:04am

Hello! Anyone who is having this problem would you mind if you post a bit more detail on the environment and the job you are running?

More specifically these parts:

GitLab version
Job specification from .gitlab-ci.yml
Is git strategy defined?
What is the executor you are using?
What is the runner configuration?

We changed how we clone the project to support a new feature in the long run details can be found in #3983 and the MR in !1203

shinya · March 25, 2019, 9:20am

Giving more contexts on the https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1203 change.

The new behavior on the fetching refs instead of cloning repo happens under the following condition,

GitLab Runner 11.9+
GitLab CE/EE (Rails) 11.9+

Basically, GitLab CE/EE sends refspecs parameter to GitLab Runner https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/presenters/ci/build_runner_presenter.rb#L34, this parameter is to used in GitLab Runners for fetching branch/tag refs from remote repository. This change was introduced because we wanted GitLab Rails side to leverage respecs in order for https://gitlab.com/gitlab-org/gitlab-ee/issues/7380 though, there should not be a big difference between git clone $URL or mkdir $REPO_DIR && git remote add origin $URL && git fetch +refs/heads/branch_name:refs/remotes/origin/branch_name. In fact, the new behavior has already run on our development project https://gitlab.com/gitlab-org/gitlab-ce/pipelines and has no issues so far.

Here is an example job log, when GitLab-Runner initializes a repository directory and fetches refs instead of git clone commands.

Running with gitlab-runner 11.10.0~beta.1249.ga9b92ea5 (a9b92ea5)
  on shinya-MS-7A34 LXxsZ3hC
Using Shell executor...
Running on shinya-MS-7A34...
Initialized empty Git repository in /home/shinya/workspace/thin-gdk/builds/LXxsZ3hC/0/root/job-run-test/.git/
Fetching changes...
Created fresh repository.
From http://localhost:8181/root/job-run-test
 * [new branch]      master     -> origin/master
Checking out 3a894e28 as master...

Skipping Git submodules setup
$ echo 'a'
a
Job succeeded

One thing that I’m seeing a similar situation in those reports is that everyone has encountered issues during deployments. So it might happen under a complicated situation. I’d appreciate if you’d help us to investigate this problem. Thanks.

werther99 · March 25, 2019, 7:50pm

Hi steveazz,

before I give my answers, I have to tell you that I am a basic Gitlab user. All my settings are default except setting 1 variable (SSH_PRIVATE_KEY) and writing my own .gitlab-ci.yml file.

GitLab version

11.9.0-ee

Job specification from .gitlab-ci.yml

deployment:
stage: deploy

environment:

    name: production

    url: http://XXX

only:

    - master
script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'

    - eval $(ssh-agent -s)

    - ssh-add <(echo "$SSH_PRIVATE_KEY")

    - mkdir -p ~/.ssh

    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

    - git remote add production ssh://XXX.git
    
    - git push production master

Is git strategy defined?

No

What is the executor you are using?

Default one: Using Docker executor with image ruby:2.5

What is the runner configuration?

Shared runner: gitlab-runner 11.9.0-rc2

Thank you for your help.

pjbrown11 · March 27, 2019, 1:42pm

This isn’t overly different from werther99, but if it helps, here is mine. Like the others, this was working the last time I tried a few weeks ago. Let me know if you need any other information. Thanks!

GitLab version

11.9.1-ee

Job specification from .gitlab-ci.yml

    deploy to production:
      stage: deploy
      script:
        git remote add heroku https://heroku:$HEROKU_API_KEY@git.heroku.com/<project>.git && git push -f heroku master
      environment:
        name: production
        url: https://<project>.herokuapp.com
      only:
      - master

Is git strategy defined?

No

What is the executor you are using?

(output below)

What is the runner configuration?

(output below)

Running with gitlab-runner 11.9.0-rc2 (227934c0)
  on docker-auto-scale ed2dce3a
Using Docker executor with image ruby:2.5 ...
Pulling docker image ruby:2.5 ...
Using docker image sha256:97f68e8fccc290fadf7634c95797892c54bdfd7bd0a0c2e8e4ea876b75802914 for ruby:2.5 ...
Running on runner-ed2dce3a-project-9649949-concurrent-0 via runner-ed2dce3a-srm-1553654931-389f31f2...
Initialized empty Git repository in /builds/<org-name>/<project>/.git/
Fetching changes...
Created fresh repository.
From https://gitlab.com/<org-name>/<project>
 * [new branch]      development -> origin/development
 * [new branch]      master      -> origin/master
Checking out 7d7831ab as master...

Skipping Git submodules setup
$ git remote add heroku https://heroku:$HEROKU_API_KEY@git.heroku.com/<project>.git && git push -f heroku master
error: src refspec master does not match any.
error: failed to push some refs to 'https://heroku:<scrubbed-key>@git.heroku.com/<project>.git'
ERROR: Job failed: exit code 1

monoburn · March 28, 2019, 8:47am

Our problem is LFS-related; might belong in another thread…

The new way of fetching/cloning introduced in runner 11.9 causes problem for a repo using LFS. We have specified:
variables:
GIT_STRATEGY: clone
GIT_LFS_SKIP_SMUDGE: 1
in .gitlab-ci.yml to prevent pulling LFS files. After upgrading to runner 11.9 (and gitlab 11.9.1 shortly after) our CI jobs starts cloning/fetching/pulling:
DEPRECATION: this GitLab server doesn’t support refspecs, gitlab-runner 12.0 will no longer work with this version of GitLab
Cloning repository…
Cloning into ‘/builder/b6861495/0/apps/vcpipe-bundle’…
Checking out 25b52be9 as 1815-subpanels…
Downloading LFS objects: 0% (0/1), 0 B | 0 B/s

This takes several hours (+200GB of LFS files) and eventually stopped with a no-space-left error.

Is there another mechanism to avoid pulling LFS files?

steveazz · March 28, 2019, 9:18am

@monoburn we have also done some LFS work but do you mind if you open a separate thread? You can always open an issue over at https://gitlab.com/gitlab-org/gitlab-runner/issues/new?issuable_template=Bug if you prefer

The error message you see below:

DEPRECATION: this GitLab server doesn’t support refspecs, gitlab-runner 12.0 will no longer work with this version of GitLab

This was before you upgraded to GitLab 11.9.0/1 or after as well?

monoburn · March 28, 2019, 9:49am

My comment on LFS issue has been made into issue https://gitlab.com/gitlab-org/gitlab-runner/issues/4095.

steveazz · March 28, 2019, 1:05pm

Ok, I see that is happening here, since we are using refspec to clone/fetch the repository we checkout a specific commit and not checking out a specific branch, and when the script does git push master the branch is nowhere to be found so git doesn’t know what to push.

This can clearly seen between the two jobs below:

I have created an issue https://gitlab.com/gitlab-org/gitlab-runner/issues/4097

Current workaround would be to do a git checkout master as part of the script

werther99 · March 28, 2019, 3:01pm

That worked out for me!!!
@steveazz you are a hero of the day

Thank you very much for your help!

pjbrown11 · March 28, 2019, 5:06pm

Me, too! Thank you, @steveazz.

steveazz · April 2, 2019, 3:51pm

Hello folks! We have a better workaround which I would like anyone who has time to test out.

deploy:
  stage: deploy
  script:
  - git remote add heroku https://heroku:$HEROKU_API_KEY@git.heroku.com/<project>.git
  - git push -f heroku HEAD:master

This is a bit more efficient at this point.

nahtnam · April 10, 2019, 8:59pm

Yep, I can confirm adding HEAD: before master seems to fix the issue.

MansuriNiyamat · August 21, 2020, 5:42am

I am also facing same issue. by the way Thanks @steveazz it fixed it for me