8.15 broke all(?) my machine accounts

I have etckeeper set up on our machines at work, and etckeeper is configured to push to our Gitlab CE instance. This has been working for a few years.

I upgraded to 8.15 (and now to 8.15.2, which changed the error message), and now pushes error out:

# git push metrics
GitLab: The project you were looking for could not be found.
fatal: The remote end hung up unexpectedly

If I go in to the admin area and impersonate the machine’s user (and do absolutely nothing else—just impersonate, then immediately stop impersonating) it fixes it. But there are 30 to 40 of them…

So, looking for what might be causing this—and a less tedious way to fix it.

(Some extra details: these are all local accounts that were created with the API. They are confirmed. We also have LDAP accounts, these do not have any LDAP identities associated.)

EDIT 1:

Comparing a user (in the database users table) before and after impersonating:

-sign_in_count                 | 0
-current_sign_in_at            | 
-last_sign_in_at               | 
-current_sign_in_ip            | 
-last_sign_in_ip               | 
+sign_in_count                 | 1
+current_sign_in_at            | 2016-12-31 18:00:27.743992
+last_sign_in_at               | 2016-12-31 18:00:27.743992
+current_sign_in_ip            | 172.16.15.10
+last_sign_in_ip               | 172.16.15.10
@@ -14 +14 @@
-updated_at                    | 2016-12-31 18:00:11.721347
+updated_at                    | 2016-12-31 18:00:27.745426
@@ -62 +62 @@
-authorized_projects_populated | 
+authorized_projects_populated | t

Only thing that looks at all interesting is authorized_projects_populated

EDIT 2

Indeed appears there is a project_authorizations table that is only populated on web login. There must be some other way to populate this; did some migration get skipped or something?

Could you provide the results of the following checks?

  • For installations with omnibus-gitlab package run and paste the output of:
    sudo gitlab-rake gitlab:check SANITIZE=true
  • For installations from source run and paste the output of:
    sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
  • For installations with omnibus-gitlab package run and paste the output of:
    sudo gitlab-rake gitlab:env:info
  • For installations from source run and paste the output of:
    sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
root@git:~# sudo gitlab-rake gitlab:check SANITIZE=true 
Checking GitLab Shell ...

GitLab Shell version >= 4.1.1 ? ... OK (4.1.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
5/2 ... ok
5/3 ... ok
5/4 ... ok
5/5 ... ok
6/6 ... ok
6/7 ... ok
6/8 ... ok
6/9 ... ok
6/10 ... ok
6/11 ... ok
6/12 ... ok
6/13 ... ok
6/14 ... ok
6/15 ... ok
6/16 ... ok
6/17 ... ok
6/18 ... ok
6/19 ... ok
6/20 ... ok
6/21 ... ok
6/22 ... ok
6/23 ... ok
6/24 ... ok
6/25 ... ok
6/26 ... ok
6/27 ... ok
6/28 ... ok
6/29 ... ok
6/30 ... ok
6/31 ... ok
6/32 ... ok
6/33 ... ok
6/34 ... ok
6/35 ... ok
6/36 ... ok
6/37 ... ok
4/38 ... ok
49/39 ... ok
49/40 ... ok
5/41 ... ok
4/42 ... ok
5/43 ... ok
6/44 ... ok
6/45 ... ok
6/46 ... ok
6/47 ... ok
4/50 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
«SNIP—redacted»

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... no
  Try fixing it:
  sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
  sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} \;
  sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} \;
  For more information see:
  doc/install/installation.md in section "GitLab"
  Please fix the error above and rerun the checks.
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ... 
5/2 ... yes
5/3 ... yes
5/4 ... yes
5/5 ... yes
6/6 ... yes
6/7 ... yes
6/8 ... yes
6/9 ... yes
6/10 ... yes
6/11 ... yes
6/12 ... yes
6/13 ... yes
6/14 ... yes
6/15 ... yes
6/16 ... yes
6/17 ... yes
6/18 ... yes
6/19 ... yes
6/20 ... yes
6/21 ... yes
6/22 ... yes
6/23 ... yes
6/24 ... yes
6/25 ... yes
6/26 ... yes
6/27 ... yes
6/28 ... yes
6/29 ... yes
6/30 ... yes
6/31 ... yes
6/32 ... yes
6/33 ... yes
6/34 ... yes
6/35 ... yes
6/36 ... yes
6/37 ... yes
4/38 ... yes
49/39 ... yes
49/40 ... yes
5/41 ... yes
4/42 ... yes
5/43 ... yes
6/44 ... yes
6/45 ... yes
6/46 ... yes
6/47 ... yes
4/50 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.8.4)
Active users: 41

Checking GitLab ... Finished

and

root@git:~# sudo gitlab-rake gitlab:env:info

System information
System:         Debian 8.6
Current User:   git
Using RVM:      no
Ruby Version:   2.3.3p222
Gem Version:    2.6.6
Bundler Version:1.13.7
Rake Version:   10.5.0
Sidekiq Version:4.2.7

GitLab information
Version:        8.15.2
Revision:       790035f
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://git.metrics.net
HTTP Clone URL: https://git.metrics.net/some-group/some-project.git
SSH Clone URL:  git@git.metrics.net:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no

GitLab Shell
Version:        4.1.1
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:            /opt/gitlab/embedded/bin/git

I then ran the fix commands printed by check, re-running check now no longer complains about the uploads directory. The machine accounts (that I haven’t manually fixed by impersonating them) remain broken.

It broke on gitlab.com, too. Here is an account used for pushes I have registered on gitlab.com:

root@Gyopi:/etc/apache2/sites-enabled# etckeeper  commit 'streaming share'
[master a9f33bc] streaming share
 1 file changed, 8 insertions(+)
git push using:  git@gitlab.com:amvorg/sysadmin-apache-conf.git gyopi
GitLab: The project you were looking for could not be found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

… and so I logged into the account on gitlab.com, and now it works after logging in.

I filed an issue for this: https://gitlab.com/gitlab-org/gitlab-ce/issues/26495

I believe we’re having this issue as well with our LDAP users (external users, GitLab EE 8.15.2) that never login.
Our workaround was to remove and re-add them to the project.

Could you check whether you also have this line in your logs? (Omnibus Installation: gitlab-rails/production.log in Omnibus) If you have the same problem, I’ll add it to your issue.

Started GET "/<namespace>/<repo>.git/info/refs?service=git-upload-pack" for 10.10.10.10 at 2017-01-08 02:33:06 +0000
Processing by Projects::GitHttpController#info_refs as HTML
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"<namespace>", "project_id"=>"<repo>.git"}
Completed 404 Not Found in 99ms (Views: 0.3ms | ActiveRecord: 6.7ms)

I just upgraded our EE instance to 8.15.5, and had one of our generic ldap users got its userid changed by adding a 1 to it. So instead of userid, it was userid1 and broke some of our workflows - the fix was to just rename the user back to userid.

I was fairly shocked so looked for issues out here first - looking at logs now.