Summary
Gitlab is running on a fresh Debian Jessie custom server (Openmediavault) install. It’s an omnibus (apt-get package) install.
It’s working for hours/days, with no users (only me). Then this amount of time, the web is giving me an error 500, and with a ps-ef |grep redis I can see that it is defunct, but I have no clue why. Nothing in the logs, just something in the messages log that seems like a crash :
Sep 6 22:15:53 nitromediavault kernel: [2112111.700220] PGD 86cc2067 PUD 7a4a6067 PMD 0
Sep 6 22:15:53 nitromediavault kernel: [2112111.703656] Oops: 0002 [#6] SMP
Sep 6 22:15:53 nitromediavault kernel: [2112111.726060] CPU: 1 PID: 18449 Comm: redis-server Tainted: G D 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-2+deb8u3
Sep 6 22:15:53 nitromediavault kernel: [2112111.733812] task: ffff8801184ecd20 ti: ffff880003358000 task.ti: ffff880003358000
Sep 6 22:15:53 nitromediavault kernel: [2112111.737693] RIP: 0010:[<ffffffff8120da20>] [<ffffffff8120da20>] do_task_stat+0x700/0xaa0
Sep 6 22:15:53 nitromediavault kernel: [2112111.741610] RSP: 0018:ffff88000335bd68 EFLAGS: 00010283
Sep 6 22:15:53 nitromediavault kernel: [2112111.745522] RAX: 0000000000000000 RBX: ffff880118be2b80 RCX: 0000000000000007
Sep 6 22:15:53 nitromediavault kernel: [2112111.749461] RDX: 0000000001c41000 RSI: 0000000000000000 RDI: ffff880115914105
Sep 6 22:15:53 nitromediavault kernel: [2112111.753407] RBP: ffff8801184ecd20 R08: 000000000000002b R09: 000000000000ffff
Sep 6 22:15:53 nitromediavault kernel: [2112111.757272] R10: 0000000000000000 R11: ffff88000335bc1e R12: ffff880100750480
Sep 6 22:15:53 nitromediavault kernel: [2112111.761056] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000007b5e
Sep 6 22:15:53 nitromediavault kernel: [2112111.764841] FS: 00007f555c894780(0000) GS:ffff88011ed00000(0000) knlGS:0000000000000000
Sep 6 22:15:53 nitromediavault kernel: [2112111.768652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 6 22:15:53 nitromediavault kernel: [2112111.772472] CR2: 0000000000000000 CR3: 0000000005f00000 CR4: 00000000000007e0
Sep 6 22:15:53 nitromediavault kernel: [2112111.776303] Stack:
Sep 6 22:15:53 nitromediavault kernel: [2112111.780114] 0000001400000001 000000000b89cfe5 0000000000013f4c 0000000000000001
Sep 6 22:15:53 nitromediavault kernel: [2112111.783959] 0000000000000000 00007f555bf72add 00007ffc1a1c2600 000000000001f1ea
Sep 6 22:15:53 nitromediavault kernel: [2112111.787778] 0000000000000000 0000000000000131 0000000000000117 0000000000000000
Sep 6 22:15:53 nitromediavault kernel: [2112111.791569] Call Trace:
Sep 6 22:15:53 nitromediavault kernel: [2112111.795304] [<ffffffff81209787>] ? proc_single_show+0x47/0x80
Sep 6 22:15:53 nitromediavault kernel: [2112111.799037] [<ffffffff811c9df2>] ? seq_read+0xe2/0x360
Sep 6 22:15:53 nitromediavault kernel: [2112111.802743] [<ffffffff811a83f3>] ? vfs_read+0x93/0x170
Sep 6 22:15:53 nitromediavault kernel: [2112111.806414] [<ffffffff811a9022>] ? SyS_read+0x42/0xa0
Sep 6 22:15:53 nitromediavault kernel: [2112111.810062] [<ffffffff8151420d>] ? system_call_fast_compare_end+0x10/0x15
Sep 6 22:15:53 nitromediavault kernel: [2112111.813710] Code: be 20 00 00 00 48 89 df e8 2e ce fb ff 49 8b 94 24 f8 00 00 00 be 20 00 00 00 48 89 df e8 19 ce fb ff 49 8b 94 24 00 01 00 00 ae <20> 00 00 00 48 8$
Sep 6 22:15:53 nitromediavault kernel: [2112111.821518] RSP <ffff88000335bd68>
Sep 6 22:15:53 nitromediavault kernel: [2112111.825151] CR2: 0000000000000000
Sep 6 22:15:53 nitromediavault kernel: [2112111.863431] ---[ end trace 0e973e4b00d56b2f ]---
To restart gitlab, I need to do gitlab-ctl graceful-kill redis and then restart it.
Before, I was on Debian Wheezie, and it was working. I then completly reinstall the OS, and restored the backup (but it was from an older version of gitlab, so I had to downgrade, restore, error with redis or something else, fix it, upgrade to gitlab last version).
Results of GitLab application Check
Checking GitLab Shell ...
GitLab Shell version >= 3.4.0 ? ... OK (3.4.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
2/1 ... ok
2/2 ... ok
2/4 ... ok
3/8 ... ok
2/9 ... ok
2/12 ... ok
2/13 ... ok
2/14 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: FAILED: Failed to connect to internal API
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
2/1 ... yes
2/2 ... yes
2/4 ... yes
3/8 ... yes
2/9 ... yes
2/12 ... yes
2/13 ... yes
2/14 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 3
Checking GitLab ... Finished
Results of GitLab environment info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.3.1p112
Gem Version: 2.6.6
Bundler Version:2.3.0
Rake Version: 10.5.0
Sidekiq Version:4.1.4
GitLab information
Version: 8.11.4
Revision: b871b76
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: ****
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 3.4.0
Repository storage paths:
- default: /mnt/vol1/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/git
Last info, here’s my gitlab.conf :
external_url "..."
web_server['external_users'] = ['www-data']
nginx['enable'] = false
git_data_dirs({"default" => "/mnt/vol1/git-data"})