Error creating new projects via Gitlab UI

Problem to solve

Can’t create new projects on self-hosted Gitlab.

I changed the automated installation of Gitlab on the Gitaly server, just enforced the user git to have a specific UID.
Now everything works fine and as expected, including with repositories that were already there, I just can’t create new repositories via the Gitlab UI:

Logs indicate the following:

Gitaly:

{"correlation_id":"XXXX","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"unary","grpc.method":"RepositoryExists","grpc.request.deadline":"XXXX","grpc.request.fullMethod":"/gitaly.RepositoryService/RepositoryExists","grpc.request.glProjectPath":"","grpc.request.glRepository":"","grpc.request.payload_bytes":96,"grpc.request.repoPath":"@hashed/xxxx",..., "msg":"finished unary call with code OK"}
{"correlation_id":"XXXXX","error":"rpc error: code = NotFound desc = GetRepoPath: not a git repository: \"/datadrive/gitaly/repositories/myuser/test.git\"","grpc.code":"NotFound","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"unary","grpc.method":"FindDefaultBranchName","grpc.request.deadline":"xxx","grpc.request.fullMethod":"/gitaly.RefService/FindDefaultBranchName","grpc.request.glProjectPath":"myuser/test","grpc.request.glRepository":"project-","grpc.request.payload_bytes":82,"grpc.request.repoPath":"myuser/test.git"

Web node:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/projects" for xxxxx at xxxxxxxx
Processing by ProjectsController#create as HTML
  Parameters: {"authenticity_token"=>"[FILTERED]", "project"=>{"ci_cd_only"=>"false", "name"=>"test", "namespace_id"=>"xxxx", "path"=>"test", "description"=>"[FILTERED]", "visibility_level"=>"10", "initialize_with_readme"=>"1"}}

==> /var/log/gitlab/gitlab-rails/application.log <==
xxxxx: Unable to save project. Error: uncaught throw :abort

==> /var/log/gitlab/gitlab-rails/application_json.log <==
{"severity":"ERROR","time":"xxxxxx","correlation_id":"XXXXX","message":"Unable to save project. Error: uncaught throw :abort"}

Steps to reproduce

  • Access the admin page
  • Create blank project (or from template) with Internal visibility and a simple “test” string for name and slug

Configuration

The solution runs webnodes, sidekiq and Gitaly on different virtual machines. Gitaly stores the repositories in hashed storage and is accessible from any other component

Versions

Please add an x whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Dedicated

Versions

  • GitLab 14.9.5

Could be a bug in this version. GitLab 14 is 3 years old and not supported anymore. Current version is 17.7 tomorrow.

Suggest planning an upgrade: GitLab Upgrade Path Resources

1 Like

Hey, thank you for the reply :slight_smile:

Unfortunately, upgrading is not a solution right now.

Either way, before changing the UID of the git user, everything worked fine on this version, so it doesn’t appear to be a bug of the version

Can you revert the UID change? What was that for?

Already did, just on the Gitaly machine though, but It didn’t solve the issue.

The UID change is to prevent Gitlab from being unable to reach repository data when migrating the OS of the machine (since it can change the UID of users).

I would provide more info but sadly the logs I referred are really everything I find

Hmmm searching in the 14.9.5 source code (Gitaly) for the grpc method RepositoryExists could provide some insights how the functionality works, and see why the error “not a git repository” occurs.

A quick search also brings up some closed issues in RepositoryExists · Search · GitLab - Insightful but not problem related I think. CreateRepository cannot use atomic two-phase voting (#3779) · Issues · GitLab.org / gitaly · GitLab

Another search brought me into internal/gitaly/repoutil/create.go · e9b91447b381c42612796150c7f793fe5a607325 · GitLab.org / gitaly · GitLab and idea to select the 14.9.5 git tag. Problem: File did not exist back then.

But we can use the tag tree to search further already exists · Search · GitLab

Navigating the source path leads to internal/gitaly/service/repository/repository.go · v14.9.5 · GitLab.org / gitaly · GitLab which reflects the error message behavior - it will attempt to check if the path exists, and throw an error if not a Git repository.

IsGitRepository is implemented in internal/gitaly/storage/locator.go · v14.9.5 · GitLab.org / gitaly · GitLab

It checks objects, refs, HEAD files in the Git repo. The old Gitaly version still supports NFS.

Question: How does the content look like of /datadrive/gitaly/repositories/myuser/test.git

ls -lahR

Maybe it is a stale left-over directory without content, or it lacks permission.

My guess is - permission problem, because of the UID changes which are reflected on the filesystem, but also affect the running daemon. Which leads me to the question - did you restart Gitaly/GitLab after reverting the UID change?

Thank you for your help!

I have restarted and reconfigured Gitlab after the changes but with no effect.

Regarding the content of the folder, there is no folder with that structure, but there is one with hashed structure (@hashed/3b/…), which is present in the logs above, I just deleted it. The content is the following:

When looking at “config.git”, it shows at the end of the file:

[gitlab]
	fullpath = (some random group name)/(some random project name)

And those group and project names do not correspond to the repo I’m creating, and I also can’t find them via Gitlab UI, so maybe it’s using a hash from a project that was deleted/archived?

So I found the issue.

It happens that I only restored the repository data from a backup I had, and not the Database. Thus, the amount of projects (and correspondent IDs) on my Gitaly machine were not matching the IDs on the database.

Example:

Backup → 1000 repositories
Machine before restore → 1010 repositories

Since I did not restore the database to a previous state, after restoring the repositories from backup and trying to create a new project via the UI, this would get the project ID 1001, which was marked occupied on the Database, hence the hashed path pointing to some project that I couldn’t find in my Gitlab UI (because it was created after I generated the backup).

The solution was to attempt to create projects until the ID was incremented enough to match the DB (and then it worked), or, in a cleaner way, restore the database to the date when I generated the backup.

Thank you @dnsmichi for your insights!

1 Like