This is a wikified topic, that means any registered member can edit it. We have migrated from GitHub as this will give us more flexibility to add relevant issues posted in the Troubleshooting category.
You have encountered a problem that is not mentioned here and you have the solution? Leave a reply under this thread and discuss it with other members to add it to the list.
If you cannot find your issue here, use this forum’s search to see if it is already resolved. If not, create a new topic in the Troubleshooting category.
Purpose
This guide should help locate and resolve issues you might have with Gitlab.
If you don’t know where to start you should begin with the General section below.
Help refining this guide
Any help is welcome. To contribute to this wiki:
- Add the problem description and solution under the relevant section.
- Remember to add three dashes
---between errors for better distinction.
Sections
General
GitLab can help you checking if it was set up properly and locating potential sources of problems.
The easiest way is by running the self diagnosis command (note that you must be in the gitlab directory when the command is run):
# for GitLab 3.1 and earlier
sudo -u gitlab -H bundle exec rake gitlab:app:status RAILS_ENV=production
# for GitLab 4.0 and later (uses gitlab user)
sudo -u gitlab -H bundle exec rake gitlab:check RAILS_ENV=production
# for GitLab 5.0 and later (uses git user)
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production
If you are all green you should have eliminated most of the obvious error sources.
You should also check the sections below for your issue and follow the steps to fix it.
If you still need help visit our Support Forum or consult the installation guide.
Forgot / reset admin password
If you forget the admin password (for admin@local.host user), you can easily change it using the rails console.
On the GitLab server:
# Change to gitlab directory. Normally, /home/git/gitlab.
cd /home/git/gitlab
sudo -u git -H bundle exec rails console production
If you are running omnibus-gitlab you should run the following
sudo gitlab-rails console
The console may take several minutes to start up. Be patient. Once you get an irb prompt (you must type the same password in both password and password_confirmation method or validations will not pass and the password will not be updated:
user = User.find_by(email: 'admin@local.host')
user.password = 'secret_pass'
user.password_confirmation = 'secret_pass'
user.save
That’s it! You should be able to login as the admin user with the new password.
Could not read from remote repository
Error: clone from ssh doesn’t work
The SSH path for my project doesn't work because it is missing the repositories directory.
git@git.myserver.com:mygroup/proj1.git
should be
git@git.myserver.com:repositories/mygroup/proj1.git
Problem: Described in https://github.com/gitlabhq/gitlabhq/issues/3686. The public key for the git user already existed in ~/.ssh/authorized_keys before you setup gitlab, the key for gitlab should begin with:
command="/home/git/gitlab/apps/gitlab/gitlab-shell/bin/gitlab-shell key-2",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty
Solution: Remove the keys which do not start with the above command from ~/.ssh/authorized_keys
authorized_keys file is out of sync
Problem Sometimes the authorized_keys file can get out of sync. Maybe some user added an SSH key and at the time there was a file permission issue that prevented that key from being added to the authorized_keys file.
Solution The following rake task can “resync” all of the keys.
sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production
This will rebuild an authorized_keys file.
You will lose any data stored in authorized_keys file.
Do you want to continue (yes/no)? yes
............................
Error: clone from http doesn’t work
Problem:
git clone http://git.myserver.com:mygroup/proj1.git doesn't work
Solution: update gitlab nginx config: https://github.com/gitlabhq/gitlabhq/issues/3384#issuecomment-15457287
Fail To Load Assets Due To Permission Issues
Error: When accessing GitLab, the page starts to load, and then times out.
In log/unicorn.stderr.log
E, [2014-02-14T22:16:23.902375 #9798] ERROR – : worker=1 PID:10223 timeout (31s > 30s), killing
E, [2014-02-14T22:16:23.959955 #9798] ERROR – : reaped #<Process::Status: pid 10223 SIGKILL (signal 9)> worker=1
Solution: Make sure your assets are precompiled.
When accessing GitLab, the login page is broken, images and CSS files are not correctly loaded.
In Nginx error log
2013/08/23 13:52:19 [error] 31035#0: *1 open() "/home/git/gitlab/public/assets/login-logo-20fd89985ad8929292196a39fa51787c.png" failed (13: Permission denied), client: 10.191.131.75, server: devops.company.com, request: "GET /assets/login-logo-20fd89985ad8929292196a39fa51787c.png HTTP/1.1", host: "devops.company.com", referrer: "http://devops.company.com/users/sign_in"
2013/08/23 13:52:22 [error] 31035#0: *1 open() "/home/git/gitlab/public/assets/application-8d0430122c22e07faecdd974cd21a38b.css" failed (13: Permission denied), client: 10.191.131.75, server: devops.company.com, request: "GET /assets/application-8d0430122c22e07faecdd974cd21a38b.css HTTP/1.1", host: "devops.company.com", referrer: "http://devops.company.com/users/sign_in"
2013/08/23 13:52:22 [error] 31035#0: *10 open() "/home/git/gitlab/public/assets/application-7b88a88b92ad70a827dbedb75e5206b1.js" failed (13: Permission denied), client: 10.191.131.75, server: devops.company.com, request: "GET /assets/application-7b88a88b92ad70a827dbedb75e5206b1.js HTTP/1.1", host: "devops.company.com", referrer: "http://devops.company.com/users/sign_in"
Problem: Permission denied when nginx user was trying to open assets (images, CSS files). The nginx user does NOT have access to the assets directory because it is NOT member of the git group.
NOTE: In the installation guide, adding Nginx user to the git group is NOT explicitly mentioned. However, it is well documented in the Gitlab Recipes.
Solution:
Recommended
add nginx user to git group and give groups read/execute permission on /home/git
sudo usermod -a -G git nginx
sudo chmod g+rx /home/git/
Alternatively, replace the default nginx user with git and group root in /etc/nginx/nginx.conf
#user nginx;
user git root;
Then restart Nginx, the issue should be fixed. See issue 4893 for more information.
On first run the homepage return 502 bad gateway (nginx) after 30/60 seconds
Fix First start seems slow so increase the unicorn timeout to something larger, say 200. (Or buy faster hardware)
vim config/unicorn.rb
sudo /etc/init.d/gitlab restart
Nginx won’t start, “duplicate default server”, or starts but shows default page, not Gitlab
solution:
# rm /etc/nginx/sites-enabled/default
# /etc/init.d/nginx restart
[ ok ] Restarting nginx: nginx.
Can’t verify CSRF token authenticity/can’t get past login page/redirected to login page
Problem: Using apache/nginx and SSL, you authenticate but get resent to the login page. Your production.log file has something similar to the following:
Can't verify CSRF token authenticity
Redirected to http://gitlab.yourserver.org/
Completed 302 Found in 109ms (ActiveRecord: 1.8ms)
Started GET "/" for 10.0.1.2 at 2014-01-23 07:47:34 -0500
Processing by DashboardController#show as HTML
Completed 401 Unauthorized in 2ms
Started GET "/users/sign_in" for 10.0.1.2 at 2014-01-23 07:47:34 -0500
Test: in your gitlab.yml, change https to false, restart gitlab. If it now works, then check the solution below.
Info: with 6.5 cookies are now secured when using https. This requires changes to the ProxyPass directives in apache. This is a sign that you have been using and upgrading Gitlab for a while and just found out you needed an upgrade to the apache gitlab-recipes.
Solution for Apache: It’s likely you don’t have RequestHeader set X_FORWARDED_PROTO 'https' in your apache config for gitlab. Update your gitlab to the apache recipe Files · master · GitLab.org / GitLab recipes · GitLab. Change https back to true in gitlab.yml
Solution for Nginx: It’s likely you don’t have proxy_set_header X-Forwarded-Proto https; in your nginx config for gitlab. Update your config as shown in the nginx recipe Files · master · GitLab.org / GitLab recipes · GitLab. Change https back to true in gitlab.yml
If you’re using a https->http reverse proxy in front of nginx on the gitlab machine then you may need to follow the Apache approach and hardcode X-Forwarded-Proto rather than using $scheme as in the recipe.
Ruby (ruby-2.0.0-p353) compile error
Problem: Ruby won’t compile, readline error, ‘Function’ undeclared
readline.c: In function ‘Init_readline’:
readline.c:1977:26: error: ‘Function’ undeclared (first use in this function)
rl_pre_input_hook = (Function *)readline_pre_input_hook;
^
readline.c:1977:26: note: each undeclared identifier is reported only once for each function it appears in
readline.c:1977:36: error: expected expression before ‘)’ token
rl_pre_input_hook = (Function *)readline_pre_input_hook;
^
readline.c: At top level:
readline.c:634:1: warning: ‘readline_pre_input_hook’ defined but not used [-Wunused-function]
readline_pre_input_hook(void)
Solution: Edit ext/readline/readline.c in the Ruby source and change the line
rl_pre_input_hook = (Function *)readline_pre_input_hook;
to this:
rl_pre_input_hook = (rl_hook_func_t *)readline_pre_input_hook;
SSH
Error:
$ git push origin master
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
In your server logs (/var/log/auth.log):
User git not allowed because account is locked
Problem: On some linux distros the adduser script creates an entry in /etc/shadow starting with git:!:...
Solution: change git:!: to git:*:.
Git
Error: Various strange errors loading commit data. (See: #3614, #3666 and #3979 )
Problem: You are running a git version that is too old (i.e. it does not have all the features gitlab uses). Most likely you are not running on Ubuntu but on something else (like CentOS)
Solution: Update git to version 1.8 or newer.
Gitlab Shell
Error:
$ git push origin master
/usr/local/lib/ruby/1.9.1/net/http.rb:762:in initialize': getaddrinfo: Name or service not known (SocketError)
Problem: /home/git/gitlab-shell/config.yml has wrong gitlab_url. Make sure it has / at the end of url
Solution: Fix gitlab_url
Error: remote: GitLab: You are not allowed to access some of the refs!
$ git push origin master
...
remote: GitLab: You are not allowed to access some of the refs!
To ssh://git@gitlab.example.com/username/project.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@gitlab.example.com/username/project.git'
Problem: repos_path and auth_path in /home/git/gitlab-shell/config.yml have a location which contains a symlinked directory.
Solution: Use the full path eg. /HD50GB/home/git/repositories instead of /home/git/repositories
Error:
$ git push origin master
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
$ ssh git@your_server_here
Server refused to allocate pty
/home/git/gitlab-shell/bin/gitlab-shell:8: undefined method 'require_relative' for main:Object (NoMethodError)
Problem: You are using system-wide RVM.
Solution: Run the following as a user who has RVM set up correctly.:
env | grep -E "^(GEM_HOME|PATH|RUBY_VERSION|MY_RUBY_HOME|GEM_PATH)=" > /var/tmp/tempenv
sudo -u git -H cp /var/tmp/tempenv /home/git/.ssh/environment
echo "PermitUserEnvironment yes" >> /etc/ssh/sshd_config
MySQL
Error: Mysql2::Error: Specified key was too long; max key length is 1000 bytes
Solution: described in https://github.com/gitlabhq/gitlabhq/issues/2412
Error: Mysql2::Error: Got a packet bigger than 'max_allowed_packet' bytes: UPDATE merge_requestsSETst_diffs = ...
Solution: described in https://github.com/gitlabhq/gitlabhq/issues/1728
Error: Lost connection to MySQL server during query
Solution: upgrade to Ruby 2.0.0+ or disable reaping_frequency in config/database.yml (just leave it blank) according to Denniss.
PostgreSQL
Error: ERROR: operator does not exist: character varying = integer at character 59
Error: ActionView::Template::Error (PG::Error: ERROR: operator does not exist: character varying = integer ...
Problem: comparting integer values (in queries) to a text column (notes.noteable_id)
Solution: described in https://github.com/gitlabhq/gitlabhq/issues/1957#issuecomment-10213437
Note: this has been fixed since GitLab 4.0
Sidekiq
Error: 'ERROR: Error fetching message: ERR unknown command ‘brpop’`
Error: Notification emails are not sent and recently pushed commits aren’t showing up in the home feed
Problem: brpop is only available since redis-server version 2.0.0 which is not shipped with Debian Squeeze and Ubuntu 10.04 LTS for example.
Solution: Install new redis-server via backports, see → https://github.com/gitlabhq/gitlabhq/issues/2675#issuecomment-12504306
On Debian Squeeze:
echo "deb http://backports.debian.org/debian-backports squeeze-backports main" >> /etc/apt/sources.list
apt-get update
apt-get -t squeeze-backports install redis-server
On Ubuntu 10.04 LTS:
sudo add-apt-repository ppa:rwky/redis
sudo apt-get update
sudo apt-get upgrade
If the “add-apt-repository” command is not found install it using:
sudo apt-get install python-software-properties
Be sure to stop / kill all existing redis-server processes after installing the new redis-server (check via ps -C redis-server), restart afterwards.
GitLab
Error: TypeError (can't dump anonymous class Class)
Problem: wrong Ruby version (i.e. < 1.9.3) or your interpreter was not compiled with YAML support
Solution: update your Ruby interpreter (or recompile making sure that it links with libyaml correctly)
Error: Missing setting 'web' in /home/gitlabhq/gitlabhq/config/gitlab.yml (Settingslogic::MissingSetting)
Problem: your configuration file needs to be updated
Solution: see config/gitlab.yml.example for how an up-to-date config file looks
Error: no such file to load -- rb-inotify when running rake (as seen in #1752)
Problem: task run in wrong environment
Solution: add RAILS_ENV=production to the end of the command
Error: no such file to load -- rb-inotify when running rake -T RAILS_ENV=production
Problem: RAILS_ENV is not set during collection of tasks, but apparently needs to be
Solution: add RAILS_ENV=production to the beginning of the command: RAILS_ENV=production bundle exec rake -T (see this comment by randx)
Error: RuntimeError: Satellite doesn't exist
Problem: satellite repos don’t exists
Solution: Run the following command:
# for GitLab 3.1 and earlier
sudo -u gitlab -H bundle exec rake gitlab:app:enable_automerge RAILS_ENV=production
# for GitLab 4.0 and later (uses gitlab user)
sudo -u gitlab -H bundle exec rake gitlab:satellites:create RAILS_ENV=production
# for GitLab 5.0 and later (uses git user)
sudo -u git -H bundle exec rake gitlab:satellites:create RAILS_ENV=production
You may also need to remove the tmp/repo_satellites directory and rerun the rake task.
Error: Merge requests stuck at Checking for ability to automatically merge
Problem: automatic merges are not enabled
Solution: described in https://github.com/gitlabhq/gitlabhq/issues/1104#issuecomment-7056318
You might also try to check your setup: $ sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production
For GitLab 6, try updating git greater than 1.7.0
Error: Merge requests stuck at Checking for ability to automatically merge
Problem: incorrect file system rights inside gitlab satellites folder
Solution: Run the following command:
sudo chmod u+rwx,g=rx,o-rwx /path/to/gitlab-satellites
Error: --broken encoding: unknown in views
Error: ActionView::Template::Error (could not find any magic files!)
Problem: CharlockHolmes was not installed correctly
Solution: described in https://github.com/gitlabhq/gitlabhq/issues/679#issuecomment-5282141 (please note that the required version for CharlockHolmes as of GitLab 3.1 is 0.6.9)
Error: .../gems/charlock_holmes-0.6.8/ext/charlock_holmes/charlock_holmes.so: undefined symbol: ucsdet_open_46
Problem: your ICU library is outdated
Solution: described in https://github.com/gitlabhq/gitlabhq/issues/1587#issuecomment-8999065. Also see https://github.com/gitlabhq/gitlabhq/issues/679#issuecomment-5282141 for reinstalling CharlockHolmes.
(Gentoo-specific)
Error: encoding_detector.c:(.text+0xa0): undefined reference to 'uenum_count'
Problem: default linker flags on gentoo produce a binary with missing symbols
Solution: described in Build fails to link `-licuuc -lz` on gentoo · Issue #32 · brianmario/charlock_holmes · GitHub
Error: ActionView::Template::Error (Failed to get header.) when navigating to a directory with a README or a showing a file
Error: ActionView::Template::Error (EPIPE) in app/models/tree.rb:5:in 'colorize'
Problem: GitLab can’t find Python
Solution: described in https://github.com/gitlabhq/gitlabhq/issues/2214#issuecomment-11137058
Error: Forbidden when accessing Resque admin page
Problem: incorrect proxy settings in Nginx
Solution: described in https://github.com/gitlabhq/gitlabhq/issues/1158#issuecomment-7390383
Error: commits pushed to Gitlab not showing (as seen in #365)
Problem: gitlab user and group not setup properly
Solution: follow the appropriate steps in the Installation Guide.
It may also be that the git directory containing the repo’s does not have proper permissions. If not, “chmod g+rx” it.
Error: rake aborted! cannot load such file -- omniauth/google/oauth2
Solution: Open the Gemfile and find the line:
gem 'omniauth-google-oauth2'
replace it with:
gem 'omniauth-google-oauth2', :require => "omniauth-google-oauth2"
Save it and run rake again.
Error: rake aborted! cannot load such file -- openssl (or similar)
Problem: Ruby is not properly compiled with OpenSSL support
Solution: Solutions explored in Openssl error - ubuntu - Ruby - Ruby-Forum
Error: Page not found (404) on project page
Problem: Namespaces not properly migrated.
Solution: Open your MysqlDB console and execute following statement:
UPDATE projects SET namespace_id = NULL WHERE namespace_id IS NOT NULL AND namespace_id NOT IN (SELECT namespaces.id FROM namespaces);
That should fix projects with non-existing namespaces.
Error: R any gitolite-admin admin_local_host_XXXXXXXX DENIED by fallthru
Problem: Admin user and Gitlab user have same SSH public keys.
Solution: Remove SSH key from admin user or generate new ssh keys for gitlab user and repeat migration to new version (rm /home/gitlab/.ssh/id_rsa* and sudo -u gitlab -H ssh-keygen).
Error: link_to helper failed with routing error after upgrade to GitLab 5
Problem: Some users have no username field filled.
Solution: Fill username field.
Typical way:
RAILS_ENV=production bundle exec rails console
User.where(:username => nil).each { |u| u.username = u.email.sub(/@.+/, ''); u.save! }
Error: File browsing does not show the last commit of file.
Error: File browsing is not working at all.
Error: “Loading commit data…” never end (looping forever).
Problem: Gitlab needs the option follow of the git-lab command which is not available in older versions of git.
Solution: Update git for at least 1.7.10.4.
Backup (Huge Commits → abort)
When doing a backup with repos having huge commits, it can abort due to a timeout from git.
Error:
Grit::Git::GitTimeout
/home/git/gitlab/vendor/bundle/ruby/1.9.1/gems/gitlab-grit-2.6.2/lib/grit/git.rb:362:in `rescue in native'
...
Solution: Increase Git Timeout in gitlab.yml.
git:
timeout: 60
Gravatar ssl not working
When https is set to true in the configuration file, you must clear redis cache for the ssl to work. To do so, enter the following commands:
cd /home/git/gitlab/
sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production
and restart your GitLab application. See this issue for more info.
