Build fails when using cached gems with a git bundler dependency

I have a gem in my Gemfile defined like this:

gem 'omniauth-google-oauth2', git: 'https://github.com/ajsharp/omniauth-google-oauth2', ref: 'json-fix'

This seems to cause a problem with my gitlab CI configuration, as I started getting the follow error message:

Fetching https://github.com/ajsharp/omniauth-google-oauth2
warning: no common commits
fatal: '/builds/vin-vin/vinvin-api/vendor/ruby/2.5.0/cache/bundler/git/omniauth-google-oauth2-cd359dbab59c19a83c2c9977094938f94fa9dbf1' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Retrying `git fetch --force --quiet --tags "/builds/vin-vin/vinvin-api/vendor/ruby/2.5.0/cache/bundler/git/omniauth-google-oauth2-cd359dbab59c19a83c2c9977094938f94fa9dbf1"` due to error (2/4): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags "/builds/vin-vin/vinvin-api/vendor/ruby/2.5.0/cache/bundler/git/omniauth-google-oauth2-cd359dbab59c19a83c2c9977094938f94fa9dbf1"` in directory /builds/vin-vin/vinvin-api/vendor/ruby/2.5.0/bundler/gems/omniauth-google-oauth2-b4416f57e81d has failed.
If this error persists you could try removing the cache directory '/builds/vin-vin/vinvin-api/vendor/ruby/2.5.0/cache/bundler/git/omniauth-google-oauth2-cd359dbab59c19a83c2c9977094938f94fa9dbf1'
fatal: '/builds/vin-vin/vinvin-api/vendor/ruby/2.5.0/cache/bundler/git/omniauth-google-oauth2-cd359dbab59c19a83c2c9977094938f94fa9dbf1' does not appear to be a git repository
fatal: Could not read from remote repository.

If I clear the runner cache and the run tests again, everything passes.

Here’s the relevant portion of my .gitlab-ci.yml:

image: ruby:2.5.5

cache:
  key: $CI_COMMIT_REF_SLUG
  untracked: true
  paths:
    - vendor/ruby

services:
  - mongo:3.6.6
  - redis:4.0.8

before_script:
  - gem install bundler --version=2.0.1
  - bundle install -j $(nproc) --path vendor

stages:
  - test
  - deploy

test:
  stage: test
  script:
    - bundle exec rspec spec