Constant db background saving

Hey,

I have an issue, which may be caused by a lack of knowledge of my side. Therefore, I try to get some help here.
I’m running the official GitLab CE docker image on my home server for only about 20 projects and 20 commits per week. So most of the time the gitlab server is idle. But while gitlab is running i discovered the hdds never go to sleep. After a bit of log reading and searching I found out that gitlab is constantly interacting with the database and doing background savings. I’m not sure if this is a configuration error of mine, or a bug. Here is the log of a freshly started gitlab instance:

gitlab-monitor

    2017-06-12_15:51:13.53153 [2017-06-12 15:51:13] INFO  WEBrick 1.3.1
    2017-06-12_15:51:13.53174 [2017-06-12 15:51:13] INFO  ruby 2.3.3 (2016-11-21) [x86_64-linux]
    2017-06-12_15:51:13.54111 == Sinatra (v1.4.8) has taken the stage on 9168 for development with backup from WEBrick
    2017-06-12_15:51:13.54265 [2017-06-12 15:51:13] INFO  WEBrick::HTTPServer#start: pid=382 port=9168
    2017-06-12_15:51:15.90839 127.0.0.1 - - [12/Jun/2017:15:51:15 UTC] "GET /sidekiq HTTP/1.1" 200 3379
    2017-06-12_15:51:15.90885 - -> /sidekiq
    2017-06-12_15:51:17.56631 127.0.0.1 - - [12/Jun/2017:15:51:17 UTC] "GET /database HTTP/1.1" 200 38101
    2017-06-12_15:51:17.56820 - -> /database
    2017-06-12_15:51:19.64085 127.0.0.1 - - [12/Jun/2017:15:51:19 UTC] "GET /process HTTP/1.1" 200 104
    2017-06-12_15:51:19.64218 - -> /process
    2017-06-12_15:51:30.41199 127.0.0.1 - - [12/Jun/2017:15:51:30 UTC] "GET /sidekiq HTTP/1.1" 200 3379
    2017-06-12_15:51:30.41210 - -> /sidekiq
    2017-06-12_15:51:32.44824 127.0.0.1 - - [12/Jun/2017:15:51:32 UTC] "GET /database HTTP/1.1" 200 38101
    2017-06-12_15:51:32.45026 - -> /database
    2017-06-12_15:51:34.63189 127.0.0.1 - - [12/Jun/2017:15:51:34 UTC] "GET /process HTTP/1.1" 200 104
    2017-06-12_15:51:34.63197 - -> /process
    2017-06-12_15:51:45.43610 127.0.0.1 - - [12/Jun/2017:15:51:45 UTC] "GET /sidekiq HTTP/1.1" 200 3379
    2017-06-12_15:51:45.43627 - -> /sidekiq
    2017-06-12_15:51:47.38024 127.0.0.1 - - [12/Jun/2017:15:51:47 UTC] "GET /database HTTP/1.1" 200 38101
    2017-06-12_15:51:47.38031 - -> /database
    2017-06-12_15:51:49.61235 127.0.0.1 - - [12/Jun/2017:15:51:49 UTC] "GET /process HTTP/1.1" 200 1509
    2017-06-12_15:51:49.61252 - -> /process
    2017-06-12_15:52:00.44816 127.0.0.1 - - [12/Jun/2017:15:52:00 UTC] "GET /sidekiq HTTP/1.1" 200 3379
    2017-06-12_15:52:00.44831 - -> /sidekiq
    2017-06-12_15:52:02.41018 127.0.0.1 - - [12/Jun/2017:15:52:02 UTC] "GET /database HTTP/1.1" 200 38101
    2017-06-12_15:52:02.41282 - -> /database
    2017-06-12_15:52:04.61428 127.0.0.1 - - [12/Jun/2017:15:52:04 UTC] "GET /process HTTP/1.1" 200 1511
    2017-06-12_15:52:04.61441 - -> /process
    2017-06-12_15:52:13.35792 == Sinatra has ended his set (crowd applauds)
    2017-06-12_15:52:13.35949 [2017-06-12 15:52:13] INFO  going to shutdown ...
    2017-06-12_15:52:13.35959 [2017-06-12 15:52:13] INFO  WEBrick::HTTPServer#start done.

gitlab-workorse

2017-06-12_15:51:12.40351 2017/06/12 15:51:12 Starting gitlab-workhorse v2.0.0-
20170607.215504
2017-06-12_15:51:12.40490 2017/06/12 15:51:12 keywatcher: starting process loop
2017-06-12_15:51:12.40496 2017/06/12 15:51:12 redis: dialing "unix", "/var/opt/gitlab/redis/redis.socket"
2017-06-12_15:51:12.40905 2017/06/12 15:51:12 error: keywatcher: dial unix /var/opt/gitlab/redis/redis.socket: connect: no such file or directory
2017-06-12_15:51:12.50972 2017/06/12 15:51:12 redis: dialing "unix", "/var/opt/gitlab/redis/redis.socket"
2017-06-12_15:51:20.98478 2017/06/12 15:51:20 error: GET "/snippets/4/notes": badgateway: failed after 0s: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused
2017-06-12_15:51:20.98488 2017/06/12 15:51:20 ErrorPage: serving predefined error page: 502
2017-06-12_15:51:20.98490 192.168.179.10 @ - - [2017-06-12 15:51:20.98076292 +0000 UTC] "GET /snippets/4/notes HTTP/1.1" 502 2925 "http://192.168.179.10/snippets/4" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.114 Safari/537.36 Vivaldi/1.9.818.50" 0.003775
2017-06-12_15:51:24.11556 2017/06/12 15:51:24 error: GET "/dashboard/projects": badgateway: failed after 0s: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused
2017-06-12_15:51:24.11565 2017/06/12 15:51:24 ErrorPage: serving predefined error page: 502
2017-06-12_15:51:24.11566 192.168.179.10 @ - - [2017-06-12 15:51:24.114670962 +0000 UTC] "GET /dashboard/projects HTTP/1.1" 502 2925 "http://192.168.179.10/snippets/4" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.114 Safari/537.36 Vivaldi/1.9.818.50" 0.000778
2017-06-12_15:51:24.26698 2017/06/12 15:51:24 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/favicon.ico" ("") for GET "/favicon.ico"
2017-06-12_15:51:24.27445 192.168.179.10 @ - - [2017-06-12 15:51:24.266324486 +0000 UTC] "GET /favicon.ico HTTP/1.1" 200 5430 "http://192.168.179.10/dashboard/projects" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.114 Safari/537.36 Vivaldi/1.9.818.50" 0.007875
2017-06-12_15:51:24.38050 2017/06/12 15:51:24 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/favicon.ico" ("") for GET "/favicon.ico"
2017-06-12_15:51:24.38059 192.168.179.10 @ - - [2017-06-12 15:51:24.380114746 +0000 UTC] "GET /favicon.ico HTTP/1.1" 304 0 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.114 Safari/537.36 Vivaldi/1.9.818.50" 0.000243
2017-06-12_15:51:33.77613 2017/06/12 15:51:33 error: GET "/dashboard/projects": badgateway: failed after 0s: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused
2017-06-12_15:51:33.77624 2017/06/12 15:51:33 ErrorPage: serving predefined error page: 502
2017-06-12_15:51:33.77626 192.168.179.10 @ - - [2017-06-12 15:51:33.775274004 +0000 UTC] "GET /dashboard/projects HTTP/1.1" 502 2925 "http://192.168.179.10/snippets/4" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.114 Safari/537.36 Vivaldi/1.9.818.50" 0.000746
2017-06-12_15:51:33.94187 2017/06/12 15:51:33 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/favicon.ico" ("") for GET "/favicon.ico"
2017-06-12_15:51:33.94195 192.168.179.10 @ - - [2017-06-12 15:51:33.941536143 +0000 UTC] "GET /favicon.ico HTTP/1.1" 200 5430 "http://192.168.179.10/dashboard/projects" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.114 Safari/537.36 Vivaldi/1.9.818.50" 0.000233
2017-06-12_15:51:47.82723 192.168.179.10 @ - - [2017-06-12 15:51:45.818039295 +0000 UTC] "GET /dashboard/projects HTTP/1.1" 200 26832 "http://192.168.179.10/snippets/4" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.114 Safari/537.36 Vivaldi/1.9.818.50" 2.008927
2017-06-12_15:51:48.45380 2017/06/12 15:51:48 Send file "/opt/gitlab/embedded/service/gitlab-rails/public/uploads/system/project/avatar/1/Player_walking1.png" for GET "/uploads/system/project/avatar/1/Player_walking1.png"
2017-06-12_15:51:48.45393 192.168.179.10 @ - - [2017-06-12 15:51:48.181653752 +0000 UTC] "GET /uploads/system/project/avatar/1/Player_walking1.png HTTP/1.1" 304 0 "http://192.168.179.10/dashboard/projects" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.114 Safari/537.36 Vivaldi/1.9.818.50" 0.272011
2017-06-12_15:51:48.75889 2017/06/12 15:51:48 Send file "/opt/gitlab/embedded/service/gitlab-rails/public/uploads/system/user/avatar/2/avatar.png" for GET "/uploads/system/user/avatar/2/avatar.png"
2017-06-12_15:51:48.75916 192.168.179.10 @ - - [2017-06-12 15:51:47.848403741 +0000 UTC] "GET /uploads/system/user/avatar/2/avatar.png HTTP/1.1" 304 0 "http://192.168.179.10/dashboard/projects" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.114 Safari/537.36 Vivaldi/1.9.818.50" 0.910670
2017-06-12_15:51:48.93197 2017/06/12 15:51:48 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/assets/favicon-075eba76312e8421991a0c1f89a89ee81678bcde72319dd3e8047e2a47cd3a42.ico" ("gzip") for GET "/assets/favicon-075eba76312e8421991a0c1f89a89ee81678bcde72319dd3e8047e2a47cd3a42.ico"
2017-06-12_15:51:48.93238 192.168.179.10 @ - - [2017-06-12 15:51:48.931534921 +0000 UTC] "GET /assets/favicon-075eba76312e8421991a0c1f89a89ee81678bcde72319dd3e8047e2a47cd3a42.ico HTTP/1.1" 200 1384 "http://192.168.179.10/dashboard/projects" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.114 Safari/537.36 Vivaldi/1.9.818.50" 0.000502
2017-06-12_15:51:57.96896 localhost @ - - [2017-06-12 15:51:57.335290468 +0000 UTC] "GET /help HTTP/1.1" 200 31627 "" "curl/7.53.0" 0.633542

postgresql

2017-06-12_15:51:12.68288 LOG:  database system was shut down at 2017-06-12 15:50:12 GMT
2017-06-12_15:51:12.71282 LOG:  MultiXact member wraparound protections are now enabled
2017-06-12_15:51:12.71383 LOG:  database system is ready to accept connections
2017-06-12_15:51:12.71746 LOG:  autovacuum launcher started
2017-06-12_15:51:17.39773 ERROR:  column projects.mirror does not exist at character 57
2017-06-12_15:51:17.39781 STATEMENT:  SELECT   ci_builds.runner_id,   ci_runners.is_shared,   projects.mirror,   projects.pending_delete,   projects.mirror_trigger_builds,   COUNT(*) AS count   FROM ci_builds   JOIN ci_runners     ON ci_runners.id = ci_builds.runner_id   JOIN projects     ON projects.id = ci_builds.project_id   WHERE ci_builds.type = 'Ci::Build'     AND ci_builds.status = 'running'   GROUP BY     ci_builds.runner_id,     projects.mirror,     projects.pending_delete,     projects.mirror_trigger_builds,     ci_runners.is_shared
2017-06-12_15:51:32.32141 ERROR:  column projects.mirror does not exist at character 57
2017-06-12_15:51:32.32154 STATEMENT:  SELECT   ci_builds.runner_id,   ci_runners.is_shared,   projects.mirror,   projects.pending_delete,   projects.mirror_trigger_builds,   COUNT(*) AS count   FROM ci_builds   JOIN ci_runners     ON ci_runners.id = ci_builds.runner_id   JOIN projects     ON projects.id = ci_builds.project_id   WHERE ci_builds.type = 'Ci::Build'     AND ci_builds.status = 'running'   GROUP BY     ci_builds.runner_id,     projects.mirror,     projects.pending_delete,     projects.mirror_trigger_builds,     ci_runners.is_shared
2017-06-12_15:51:47.29492 ERROR:  column projects.mirror does not exist at character 57
2017-06-12_15:51:47.29499 STATEMENT:  SELECT   ci_builds.runner_id,   ci_runners.is_shared,   projects.mirror,   projects.pending_delete,   projects.mirror_trigger_builds,   COUNT(*) AS count   FROM ci_builds   JOIN ci_runners     ON ci_runners.id = ci_builds.runner_id   JOIN projects     ON projects.id = ci_builds.project_id   WHERE ci_builds.type = 'Ci::Build'     AND ci_builds.status = 'running'   GROUP BY     ci_builds.runner_id,     projects.mirror,     projects.pending_delete,     projects.mirror_trigger_builds,     ci_runners.is_shared
2017-06-12_15:52:02.30474 ERROR:  column projects.mirror does not exist at character 57
2017-06-12_15:52:02.30488 STATEMENT:  SELECT   ci_builds.runner_id,   ci_runners.is_shared,   projects.mirror,   projects.pending_delete,   projects.mirror_trigger_builds,   COUNT(*) AS count   FROM ci_builds   JOIN ci_runners     ON ci_runners.id = ci_builds.runner_id   JOIN projects     ON projects.id = ci_builds.project_id   WHERE ci_builds.type = 'Ci::Build'     AND ci_builds.status = 'running'   GROUP BY     ci_builds.runner_id,     projects.mirror,     projects.pending_delete,     projects.mirror_trigger_builds,     ci_runners.is_shared
2017-06-12_15:52:15.91179 received TERM from runit, sending INT instead to force quit connections
2017-06-12_15:52:15.91541 LOG:  received fast shutdown request
2017-06-12_15:52:15.91550 LOG:  aborting any active transactions
2017-06-12_15:52:15.91967 FATAL:  terminating connection due to administrator command
2017-06-12_15:52:15.91976 FATAL:  terminating connection due to administrator command
2017-06-12_15:52:15.91977 LOG:  autovacuum launcher shutting down
2017-06-12_15:52:15.92259 LOG:  shutting down
2017-06-12_15:52:16.16658 LOG:  database system is shut down

redis

2017-06-12_15:51:12.47602                 _._                                                  
2017-06-12_15:51:12.47621            _.-``__ ''-._                                             
2017-06-12_15:51:12.47622       _.-``    `.  `_.  ''-._           Redis 3.2.5 (00000000/0) 64 bit
2017-06-12_15:51:12.47623   .-`` .-```.  ```\/    _.,_ ''-._                                   
2017-06-12_15:51:12.47624  (    '      ,       .-`  | `,    )     Running in standalone mode
2017-06-12_15:51:12.47625  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 0
2017-06-12_15:51:12.47627  |    `-._   `._    /     _.-'    |     PID: 418
2017-06-12_15:51:12.47628   `-._    `-._  `-./  _.-'    _.-'                                   
2017-06-12_15:51:12.47629  |`-._`-._    `-.__.-'    _.-'_.-'|                                  
2017-06-12_15:51:12.47630  |    `-._`-._        _.-'_.-'    |           http://redis.io        
2017-06-12_15:51:12.47631   `-._    `-._`-.__.-'_.-'    _.-'                                   
2017-06-12_15:51:12.47632  |`-._`-._    `-.__.-'    _.-'_.-'|                                  
2017-06-12_15:51:12.47634  |    `-._`-._        _.-'_.-'    |                                  
2017-06-12_15:51:12.47635   `-._    `-._`-.__.-'_.-'    _.-'                                   
2017-06-12_15:51:12.47636       `-._    `-.__.-'    _.-'                                       
2017-06-12_15:51:12.47637           `-._        _.-'                                           
2017-06-12_15:51:12.47639               `-.__.-'                                               
2017-06-12_15:51:12.47641 
2017-06-12_15:51:12.48201 418:M 12 Jun 15:51:12.481 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2017-06-12_15:51:12.48242 418:M 12 Jun 15:51:12.482 # Server started, Redis version 3.2.5
2017-06-12_15:51:12.48248 418:M 12 Jun 15:51:12.482 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2017-06-12_15:51:12.48259 418:M 12 Jun 15:51:12.482 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2017-06-12_15:51:12.57011 418:M 12 Jun 15:51:12.570 * DB loaded from disk: 0.087 seconds
2017-06-12_15:51:12.58410 418:M 12 Jun 15:51:12.584 * The server is now ready to accept connections at /var/opt/gitlab/redis/redis.socket
2017-06-12_15:52:13.09368 418:M 12 Jun 15:52:13.093 * 10000 changes in 60 seconds. Saving...
2017-06-12_15:52:13.09673 418:M 12 Jun 15:52:13.096 * Background saving started by pid 654
2017-06-12_15:52:13.18952 654:C 12 Jun 15:52:13.189 * DB saved on disk
2017-06-12_15:52:13.19078 654:C 12 Jun 15:52:13.190 * RDB: 6 MB of memory used by copy-on-write
2017-06-12_15:52:13.19771 418:M 12 Jun 15:52:13.197 * Background saving terminated with success
2017-06-12_15:52:16.76027 418:signal-handler (1497282736) Received SIGTERM scheduling shutdown...
2017-06-12_15:52:16.81768 418:M 12 Jun 15:52:16.817 # User requested shutdown...
2017-06-12_15:52:16.81782 418:M 12 Jun 15:52:16.817 * Saving the final RDB snapshot before exiting.
2017-06-12_15:52:16.92811 418:M 12 Jun 15:52:16.927 * DB saved on disk
2017-06-12_15:52:16.92829 418:M 12 Jun 15:52:16.928 * Removing the pid file.
2017-06-12_15:52:16.92835 418:M 12 Jun 15:52:16.928 * Removing the unix socket file.
2017-06-12_15:52:16.92838 418:M 12 Jun 15:52:16.928 # Redis is now ready to exit, bye bye...

Thanks everybody :slight_smile: