Troubleshooting guide wiki


#1

This is a wikified topic, that means any registered member can edit it. We have migrated from GitHub as this will give us more flexibility to add relevant issues posted in the Troubleshooting category.

You have encountered a problem that is not mentioned here and you have the solution? Leave a reply under this thread and discuss it with other members to add it to the list.

If you cannot find your issue here, use this forum’s search to see if it is already resolved. If not, create a new topic in the Troubleshooting category.

Purpose

This guide should help locate and resolve issues you might have with Gitlab.
If you don’t know where to start you should begin with the General section below.

Help refining this guide

Any help is welcome. To contribute to this wiki:

  1. Add the problem description and solution under the relevant section.
  2. Remember to add three dashes --- between errors for better distinction.

Sections

General

GitLab can help you checking if it was set up properly and locating potential sources of problems.
The easiest way is by running the self diagnosis command (note that you must be in the gitlab directory when the command is run):

# for GitLab 3.1 and earlier
sudo -u gitlab -H bundle exec rake gitlab:app:status RAILS_ENV=production

# for GitLab 4.0 and later (uses gitlab user)
sudo -u gitlab -H bundle exec rake gitlab:check RAILS_ENV=production

# for GitLab 5.0 and later (uses git user)
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production

If you are all green you should have eliminated most of the obvious error sources.
You should also check the sections below for your issue and follow the steps to fix it.

If you still need help visit our Support Forum or consult the installation guide.

Forgot / reset admin password

If you forget the admin password (for admin@local.host user), you can easily change it using the rails console.

On the GitLab server:

# Change to gitlab directory. Normally, /home/git/gitlab.
cd /home/git/gitlab

sudo -u git -H bundle exec rails console production

If you are running omnibus-gitlab you should run the following

sudo gitlab-rails console

The console may take several minutes to start up. Be patient. Once you get an irb prompt (you must type the same password in both password and password_confirmation method or validations will not pass and the password will not be updated:

user = User.find_by(email: 'admin@local.host')
user.password = 'secret_pass'
user.password_confirmation = 'secret_pass'
user.save

That’s it! You should be able to login as the admin user with the new password.

Could not read from remote repository

Error: clone from ssh doesn’t work

The SSH path for my project doesn't work because it is missing the repositories directory.

git@git.myserver.com:mygroup/proj1.git

should be

git@git.myserver.com:repositories/mygroup/proj1.git

Problem: Described in https://github.com/gitlabhq/gitlabhq/issues/3686. The public key for the git user already existed in ~/.ssh/authorized_keys before you setup gitlab, the key for gitlab should begin with:

command="/home/git/gitlab/apps/gitlab/gitlab-shell/bin/gitlab-shell key-2",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty

Solution: Remove the keys which do not start with the above command from ~/.ssh/authorized_keys


authorized_keys file is out of sync

Problem Sometimes the authorized_keys file can get out of sync. Maybe some user added an SSH key and at the time there was a file permission issue that prevented that key from being added to the authorized_keys file.

Solution The following rake task can “resync” all of the keys.

sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production
This will rebuild an authorized_keys file.
You will lose any data stored in authorized_keys file.
Do you want to continue (yes/no)? yes

............................

Error: clone from http doesn’t work

Problem:

git clone http://git.myserver.com:mygroup/proj1.git doesn't work

Solution: update gitlab nginx config: https://github.com/gitlabhq/gitlabhq/issues/3384#issuecomment-15457287

Fail To Load Assets Due To Permission Issues

Error: When accessing GitLab, the page starts to load, and then times out.

In log/unicorn.stderr.log

E, [2014-02-14T22:16:23.902375 #9798] ERROR – : worker=1 PID:10223 timeout (31s > 30s), killing
E, [2014-02-14T22:16:23.959955 #9798] ERROR – : reaped #<Process::Status: pid 10223 SIGKILL (signal 9)> worker=1

Solution: Make sure your assets are precompiled.


When accessing GitLab, the login page is broken, images and CSS files are not correctly loaded.

In Nginx error log

2013/08/23 13:52:19 [error] 31035#0: *1 open() "/home/git/gitlab/public/assets/login-logo-20fd89985ad8929292196a39fa51787c.png" failed (13: Permission denied), client: 10.191.131.75, server: devops.company.com, request: "GET /assets/login-logo-20fd89985ad8929292196a39fa51787c.png HTTP/1.1", host: "devops.company.com", referrer: "http://devops.company.com/users/sign_in"
2013/08/23 13:52:22 [error] 31035#0: *1 open() "/home/git/gitlab/public/assets/application-8d0430122c22e07faecdd974cd21a38b.css" failed (13: Permission denied), client: 10.191.131.75, server: devops.company.com, request: "GET /assets/application-8d0430122c22e07faecdd974cd21a38b.css HTTP/1.1", host: "devops.company.com", referrer: "http://devops.company.com/users/sign_in"
2013/08/23 13:52:22 [error] 31035#0: *10 open() "/home/git/gitlab/public/assets/application-7b88a88b92ad70a827dbedb75e5206b1.js" failed (13: Permission denied), client: 10.191.131.75, server: devops.company.com, request: "GET /assets/application-7b88a88b92ad70a827dbedb75e5206b1.js HTTP/1.1", host: "devops.company.com", referrer: "http://devops.company.com/users/sign_in"

Problem: Permission denied when nginx user was trying to open assets (images, CSS files). The nginx user does NOT have access to the assets directory because it is NOT member of the git group.

NOTE: In the installation guide, adding Nginx user to the git group is NOT explicitly mentioned. However, it is well documented in the Gitlab Recipes.

Solution:

Recommended

add nginx user to git group and give groups read/execute permission on /home/git

sudo usermod -a -G git nginx
sudo chmod g+rx /home/git/

Alternatively, replace the default nginx user with git and group root in /etc/nginx/nginx.conf

#user             nginx;
user              git root;

Then restart Nginx, the issue should be fixed. See issue 4893 for more information.


On first run the homepage return 502 bad gateway (nginx) after 30/60 seconds

Fix First start seems slow so increase the unicorn timeout to something larger, say 200. (Or buy faster hardware)

vim config/unicorn.rb
sudo /etc/init.d/gitlab restart

Nginx won’t start, “duplicate default server”, or starts but shows default page, not Gitlab

solution:

# rm /etc/nginx/sites-enabled/default
# /etc/init.d/nginx restart
[ ok ] Restarting nginx: nginx.

Can’t verify CSRF token authenticity/can’t get past login page/redirected to login page

Problem: Using apache/nginx and SSL, you authenticate but get resent to the login page. Your production.log file has something similar to the following:

Can't verify CSRF token authenticity
Redirected to http://gitlab.yourserver.org/
Completed 302 Found in 109ms (ActiveRecord: 1.8ms)
Started GET "/" for 10.0.1.2 at 2014-01-23 07:47:34 -0500
Processing by DashboardController#show as HTML
Completed 401 Unauthorized in 2ms
Started GET "/users/sign_in" for 10.0.1.2 at 2014-01-23 07:47:34 -0500

Test: in your gitlab.yml, change https to false, restart gitlab. If it now works, then check the solution below.

Info: with 6.5 cookies are now secured when using https. This requires changes to the ProxyPass directives in apache. This is a sign that you have been using and upgrading Gitlab for a while and just found out you needed an upgrade to the apache gitlab-recipes.

Solution for Apache: It’s likely you don’t have RequestHeader set X_FORWARDED_PROTO 'https' in your apache config for gitlab. Update your gitlab to the apache recipe https://gitlab.com/gitlab-org/gitlab-recipes/blob/master/web-server/apache/gitlab-ssl.conf. Change https back to true in gitlab.yml

Solution for Nginx: It’s likely you don’t have proxy_set_header X-Forwarded-Proto https; in your nginx config for gitlab. Update your config as shown in the nginx recipe https://gitlab.com/gitlab-org/gitlab-recipes/blob/master/web-server/nginx/gitlab-ssl. Change https back to true in gitlab.yml

If you’re using a https->http reverse proxy in front of nginx on the gitlab machine then you may need to follow the Apache approach and hardcode X-Forwarded-Proto rather than using $scheme as in the recipe.


Ruby (ruby-2.0.0-p353) compile error

Problem: Ruby won’t compile, readline error, ‘Function’ undeclared

 readline.c: In function ‘Init_readline’:
 readline.c:1977:26: error: ‘Function’ undeclared (first use in this function)
     rl_pre_input_hook = (Function *)readline_pre_input_hook;
                          ^
 readline.c:1977:26: note: each undeclared identifier is reported only once for each function it appears in
 readline.c:1977:36: error: expected expression before ‘)’ token
     rl_pre_input_hook = (Function *)readline_pre_input_hook;
                                    ^
 readline.c: At top level:
 readline.c:634:1: warning: ‘readline_pre_input_hook’ defined but not used [-Wunused-function]
 readline_pre_input_hook(void)

Solution: Edit ext/readline/readline.c in the Ruby source and change the line

    rl_pre_input_hook = (Function *)readline_pre_input_hook;

to this:

    rl_pre_input_hook = (rl_hook_func_t *)readline_pre_input_hook;

SSH

Error:

$ git push origin master
Permission denied (publickey).
fatal: The remote end hung up unexpectedly

In your server logs (/var/log/auth.log):

User git not allowed because account is locked

Problem: On some linux distros the adduser script creates an entry in /etc/shadow starting with git:!:...

Solution: change git:!: to git:*:.

Git

Error: Various strange errors loading commit data. (See: #3614, #3666 and #3979 )

Problem: You are running a git version that is too old (i.e. it does not have all the features gitlab uses). Most likely you are not running on Ubuntu but on something else (like CentOS)

Solution: Update git to version 1.8 or newer.

Gitlab Shell

Error:

$ git push origin master
/usr/local/lib/ruby/1.9.1/net/http.rb:762:in initialize': getaddrinfo: Name or service not known (SocketError)

Problem: /home/git/gitlab-shell/config.yml has wrong gitlab_url. Make sure it has / at the end of url

Solution: Fix gitlab_url


Error: remote: GitLab: You are not allowed to access some of the refs!

$ git push origin master
...
remote: GitLab: You are not allowed to access some of the refs!
To ssh://git@gitlab.example.com/username/project.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@gitlab.example.com/username/project.git'

Problem: repos_path and auth_path in /home/git/gitlab-shell/config.yml have a location which contains a symlinked directory.

Solution: Use the full path eg. /HD50GB/home/git/repositories instead of /home/git/repositories


Error:

$ git push origin master
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
$ ssh git@your_server_here
Server refused to allocate pty
/home/git/gitlab-shell/bin/gitlab-shell:8: undefined method 'require_relative' for main:Object (NoMethodError)

Problem: You are using system-wide RVM.

Solution: Run the following as a user who has RVM set up correctly.:

env | grep -E "^(GEM_HOME|PATH|RUBY_VERSION|MY_RUBY_HOME|GEM_PATH)=" > /var/tmp/tempenv
sudo -u git -H cp /var/tmp/tempenv /home/git/.ssh/environment
echo "PermitUserEnvironment yes" >> /etc/ssh/sshd_config

MySQL

Error: Mysql2::Error: Specified key was too long; max key length is 1000 bytes

Solution: described in https://github.com/gitlabhq/gitlabhq/issues/2412

Error: Mysql2::Error: Got a packet bigger than 'max_allowed_packet' bytes: UPDATEmerge_requestsSETst_diffs= ...

Solution: described in https://github.com/gitlabhq/gitlabhq/issues/1728

Error: Lost connection to MySQL server during query

Solution: upgrade to Ruby 2.0.0+ or disable reaping_frequency in config/database.yml (just leave it blank) according to Denniss.

PostgreSQL

Error: ERROR: operator does not exist: character varying = integer at character 59

Error: ActionView::Template::Error (PG::Error: ERROR: operator does not exist: character varying = integer ...

Problem: comparting integer values (in queries) to a text column (notes.noteable_id)

Solution: described in https://github.com/gitlabhq/gitlabhq/issues/1957#issuecomment-10213437

Note: this has been fixed since GitLab 4.0

Sidekiq

Error: 'ERROR: Error fetching message: ERR unknown command ‘brpop’`

Error: Notification emails are not sent and recently pushed commits aren’t showing up in the home feed

Problem: brpop is only available since redis-server version 2.0.0 which is not shipped with Debian Squeeze and Ubuntu 10.04 LTS for example.

Solution: Install new redis-server via backports, see → https://github.com/gitlabhq/gitlabhq/issues/2675#issuecomment-12504306

On Debian Squeeze:

echo "deb http://backports.debian.org/debian-backports squeeze-backports main" >> /etc/apt/sources.list
apt-get update
apt-get -t squeeze-backports install redis-server

On Ubuntu 10.04 LTS:

sudo add-apt-repository ppa:rwky/redis
sudo apt-get update
sudo apt-get upgrade

If the “add-apt-repository” command is not found install it using:

sudo apt-get install python-software-properties

Be sure to stop / kill all existing redis-server processes after installing the new redis-server (check via ps -C redis-server), restart afterwards.

GitLab

Error: TypeError (can't dump anonymous class Class)

Problem: wrong Ruby version (i.e. < 1.9.3) or your interpreter was not compiled with YAML support

Solution: update your Ruby interpreter (or recompile making sure that it links with libyaml correctly)


Error: Missing setting 'web' in /home/gitlabhq/gitlabhq/config/gitlab.yml (Settingslogic::MissingSetting)

Problem: your configuration file needs to be updated

Solution: see config/gitlab.yml.example for how an up-to-date config file looks


Error: no such file to load -- rb-inotify when running rake (as seen in #1752)

Problem: task run in wrong environment

Solution: add RAILS_ENV=production to the end of the command


Error: no such file to load -- rb-inotify when running rake -T RAILS_ENV=production

Problem: RAILS_ENV is not set during collection of tasks, but apparently needs to be

Solution: add RAILS_ENV=production to the beginning of the command: RAILS_ENV=production bundle exec rake -T (see this comment by randx)


Error: RuntimeError: Satellite doesn't exist

Problem: satellite repos don’t exists

Solution: Run the following command:

# for GitLab 3.1 and earlier
sudo -u gitlab -H bundle exec rake gitlab:app:enable_automerge RAILS_ENV=production

# for GitLab 4.0 and later (uses gitlab user)
sudo -u gitlab -H bundle exec rake gitlab:satellites:create RAILS_ENV=production

# for GitLab 5.0 and later (uses git user)
sudo -u git -H bundle exec rake gitlab:satellites:create RAILS_ENV=production

You may also need to remove the tmp/repo_satellites directory and rerun the rake task.


Error: Merge requests stuck at Checking for ability to automatically merge

Problem: automatic merges are not enabled

Solution: described in https://github.com/gitlabhq/gitlabhq/issues/1104#issuecomment-7056318

You might also try to check your setup: $ sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production
For GitLab 6, try updating git greater than 1.7.0


Error: Merge requests stuck at Checking for ability to automatically merge

Problem: incorrect file system rights inside gitlab satellites folder

Solution: Run the following command:

sudo chmod u+rwx,g=rx,o-rwx /path/to/gitlab-satellites

Error: --broken encoding: unknown in views

Error: ActionView::Template::Error (could not find any magic files!)

Problem: CharlockHolmes was not installed correctly

Solution: described in https://github.com/gitlabhq/gitlabhq/issues/679#issuecomment-5282141 (please note that the required version for CharlockHolmes as of GitLab 3.1 is 0.6.9)


Error: .../gems/charlock_holmes-0.6.8/ext/charlock_holmes/charlock_holmes.so: undefined symbol: ucsdet_open_46

Problem: your ICU library is outdated

Solution: described in https://github.com/gitlabhq/gitlabhq/issues/1587#issuecomment-8999065. Also see https://github.com/gitlabhq/gitlabhq/issues/679#issuecomment-5282141 for reinstalling CharlockHolmes.


(Gentoo-specific)
Error: encoding_detector.c:(.text+0xa0): undefined reference to 'uenum_count'
Problem: default linker flags on gentoo produce a binary with missing symbols
Solution: described in https://github.com/brianmario/charlock_holmes/issues/32


Error: ActionView::Template::Error (Failed to get header.) when navigating to a directory with a README or a showing a file

Error: ActionView::Template::Error (EPIPE) in app/models/tree.rb:5:in 'colorize'

Problem: GitLab can’t find Python

Solution: described in https://github.com/gitlabhq/gitlabhq/issues/2214#issuecomment-11137058


Error: Forbidden when accessing Resque admin page

Problem: incorrect proxy settings in Nginx

Solution: described in https://github.com/gitlabhq/gitlabhq/issues/1158#issuecomment-7390383


Error: commits pushed to Gitlab not showing (as seen in #365)

Problem: gitlab user and group not setup properly

Solution: follow the appropriate steps in the Installation Guide.
It may also be that the git directory containing the repo’s does not have proper permissions. If not, “chmod g+rx” it.


Error: rake aborted! cannot load such file -- omniauth/google/oauth2

Solution: Open the Gemfile and find the line:

gem 'omniauth-google-oauth2'

replace it with:

gem 'omniauth-google-oauth2', :require => "omniauth-google-oauth2"

Save it and run rake again.


Error: rake aborted! cannot load such file -- openssl (or similar)

Problem: Ruby is not properly compiled with OpenSSL support

Solution: Solutions explored in http://www.ruby-forum.com/topic/90083


Error: Page not found (404) on project page

Problem: Namespaces not properly migrated.

Solution: Open your MysqlDB console and execute following statement:

UPDATE projects SET namespace_id = NULL WHERE namespace_id IS NOT NULL AND namespace_id NOT IN (SELECT namespaces.id FROM namespaces);

That should fix projects with non-existing namespaces.


Error: R any gitolite-admin admin_local_host_XXXXXXXX DENIED by fallthru

Problem: Admin user and Gitlab user have same SSH public keys.

Solution: Remove SSH key from admin user or generate new ssh keys for gitlab user and repeat migration to new version (rm /home/gitlab/.ssh/id_rsa* and sudo -u gitlab -H ssh-keygen).


Error: link_to helper failed with routing error after upgrade to GitLab 5

Problem: Some users have no username field filled.

Solution: Fill username field.

Typical way:

RAILS_ENV=production bundle exec rails console
User.where(:username => nil).each { |u| u.username = u.email.sub(/@.+/, ''); u.save! }

Error: File browsing does not show the last commit of file.

Error: File browsing is not working at all.

Error: “Loading commit data…” never end (looping forever).

Problem: Gitlab needs the option follow of the git-lab command which is not available in older versions of git.

Solution: Update git for at least 1.7.10.4.


Backup (Huge Commits -> abort)

When doing a backup with repos having huge commits, it can abort due to a timeout from git.

Error:

Grit::Git::GitTimeout
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/gitlab-grit-2.6.2/lib/grit/git.rb:362:in `rescue in native' 
...

Solution: Increase Git Timeout in gitlab.yml.

git:
  timeout: 60

Gravatar ssl not working

When https is set to true in the configuration file, you must clear redis cache for the ssl to work. To do so, enter the following commands:

cd /home/git/gitlab/
sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production

and restart your GitLab application. See this issue for more info.


Troubleshooting GitLab EE / Google omniauth SSO -- no Google logo on GitLab login screen
Id_rsa.pub key access is only temporary
#2

We should probably remove all the old sections that refer to versions under 6.0.


#3

Problem: Projects miss repositories and show “No repository” after NFS outage

We have our repositories on an NFS share, which went down. Even after re-mounting the share and rebooting the GitLab machine, about two thirds of repositories seemed to be missing in the web GUI event if they were indeed present and accessible in the mount point.

sudo gitlab-rake gitlab:check showed the affected repositories as “repository is empty”.

Solution Clear the Redis cache:

sudo gitlab-rake cache:clear