Hello,
Today I was using gitlab correctly, and suddenly the server has rebooted, checking the logs it seems that now Redis is not starting so I can’t access gitlab or push any code to git.
This is the content of the logs:
/var/log/gitlab/gitlab-rails/production.log
Started GET "/-/metrics" for 127.0.0.1 at 2018-05-24 12:52:07 +0200
Processing by MetricsController#index as HTML
Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms)
Redis::CannotConnectError (Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ECONNREFUSED)):
lib/gitlab/middleware/multipart.rb:95:in `call'
lib/gitlab/request_profiler/middleware.rb:14:in `call'
lib/gitlab/middleware/go.rb:17:in `call'
lib/gitlab/etag_caching/middleware.rb:11:in `call'
lib/gitlab/middleware/read_only/controller.rb:28:in `call'
lib/gitlab/middleware/read_only.rb:16:in `call'
lib/gitlab/request_context.rb:18:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
lib/gitlab/middleware/release_env.rb:10:in `call'
/var/log/gitlab/redis/current
2018-05-24_10:52:59.96730 _._
2018-05-24_10:52:59.96732 _.-``__ ''-._
2018-05-24_10:52:59.96732 _.-`` `. `_. ''-._ Redis 3.2.11 (00000000/0) 64 bit
2018-05-24_10:52:59.96732 .-`` .-```. ```\/ _.,_ ''-._
2018-05-24_10:52:59.96734 ( ' , .-` | `, ) Running in standalone mode
2018-05-24_10:52:59.96735 |`-._`-...-` __...-.``-._|'` _.-'| Port: 0
2018-05-24_10:52:59.96735 | `-._ `._ / _.-' | PID: 12106
2018-05-24_10:52:59.96735 `-._ `-._ `-./ _.-' _.-'
2018-05-24_10:52:59.96736 |`-._`-._ `-.__.-' _.-'_.-'|
2018-05-24_10:52:59.96736 | `-._`-._ _.-'_.-' | http://redis.io
2018-05-24_10:52:59.96736 `-._ `-._`-.__.-'_.-' _.-'
2018-05-24_10:52:59.96737 |`-._`-._ `-.__.-' _.-'_.-'|
2018-05-24_10:52:59.96737 | `-._`-._ _.-'_.-' |
2018-05-24_10:52:59.96737 `-._ `-._`-.__.-'_.-' _.-'
2018-05-24_10:52:59.96738 `-._ `-.__.-' _.-'
2018-05-24_10:52:59.96738 `-._ _.-'
2018-05-24_10:52:59.96739 `-.__.-'
2018-05-24_10:52:59.96740
2018-05-24_10:52:59.96748 12106:M 24 May 12:52:59.967 # Server started, Redis version 3.2.11
2018-05-24_10:52:59.96752 12106:M 24 May 12:52:59.967 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2018-05-24_10:52:59.96757 12106:M 24 May 12:52:59.967 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2018-05-24_10:52:59.96935
2018-05-24_10:52:59.96936
2018-05-24_10:52:59.96936 === REDIS BUG REPORT START: Cut & paste starting from here ===
2018-05-24_10:52:59.96939 12106:M 24 May 12:52:59.969 # === ASSERTION FAILED OBJECT CONTEXT ===
2018-05-24_10:52:59.96941 12106:M 24 May 12:52:59.969 # Object type: 0
2018-05-24_10:52:59.96943 12106:M 24 May 12:52:59.969 # Object encoding: 0
2018-05-24_10:52:59.96945 12106:M 24 May 12:52:59.969 # Object refcount: 1
2018-05-24_10:52:59.96948 12106:M 24 May 12:52:59.969 # Object raw string len: 0
2018-05-24_10:52:59.96950 12106:M 24 May 12:52:59.969 # Object raw string content: ""
2018-05-24_10:52:59.96953 12106:M 24 May 12:52:59.969 # === ASSERTION FAILED ===
2018-05-24_10:52:59.96955 12106:M 24 May 12:52:59.969 # ==> db.c:161 'retval == DICT_OK' is not true
2018-05-24_10:52:59.96957 12106:M 24 May 12:52:59.969 # (forcing SIGSEGV to print the bug report.)
2018-05-24_10:52:59.96960 12106:M 24 May 12:52:59.969 # Redis 3.2.11 crashed by signal: 11
2018-05-24_10:52:59.96963 12106:M 24 May 12:52:59.969 # Crashed running the instuction at: 0x45d12a
2018-05-24_10:52:59.96965 12106:M 24 May 12:52:59.969 # Accessing address: 0xffffffffffffffff
2018-05-24_10:52:59.96967 12106:M 24 May 12:52:59.969 # Failed assertion: retval == DICT_OK (db.c:161)
2018-05-24_10:52:59.96969
2018-05-24_10:52:59.96969 ------ STACK TRACE ------
2018-05-24_10:52:59.96983 EIP:
2018-05-24_10:52:59.96988 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(_serverAssert+0x6a)[0x45d12a]
2018-05-24_10:52:59.96989
2018-05-24_10:52:59.96990 Backtrace:
2018-05-24_10:52:59.96993 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(logStackTrace+0x29)[0x45edd9]
2018-05-24_10:52:59.96997 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(sigsegvHandler+0xac)[0x45f4dc]
2018-05-24_10:52:59.96999 /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f0c7272d390]
2018-05-24_10:52:59.97002 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(_serverAssert+0x6a)[0x45d12a]
2018-05-24_10:52:59.97006 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(dbAdd+0x7f)[0x43aa6f]
2018-05-24_10:52:59.97010 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(rdbLoad+0x246)[0x443fe6]
2018-05-24_10:52:59.97013 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(loadDataFromDisk+0x34)[0x42b3c4]
2018-05-24_10:52:59.97016 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(main+0x381)[0x41e941]
2018-05-24_10:52:59.97019 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f0c72372830]
2018-05-24_10:52:59.97022 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0[0x41ec42]
2018-05-24_10:52:59.97024
2018-05-24_10:52:59.97024 ------ INFO OUTPUT ------
2018-05-24_10:52:59.97032 # Server
2018-05-24_10:52:59.97033 redis_version:3.2.11
2018-05-24_10:52:59.97033 redis_git_sha1:00000000
2018-05-24_10:52:59.97033 redis_git_dirty:0
2018-05-24_10:52:59.97033 redis_build_id:e819ac60ab48b2b9
2018-05-24_10:52:59.97034 redis_mode:standalone
2018-05-24_10:52:59.97034 os:Linux 4.4.0-127-generic x86_64
2018-05-24_10:52:59.97034 arch_bits:64
2018-05-24_10:52:59.97035 multiplexing_api:epoll
2018-05-24_10:52:59.97035 gcc_version:4.8.4
2018-05-24_10:52:59.97035 process_id:12106
2018-05-24_10:52:59.97035 run_id:510287742acf60ec2f6c0bc6c7f162a1da8efc3b
2018-05-24_10:52:59.97036 tcp_port:0
2018-05-24_10:52:59.97036 uptime_in_seconds:0
2018-05-24_10:52:59.97036 uptime_in_days:0
2018-05-24_10:52:59.97037 hz:10
2018-05-24_10:52:59.97037 lru_clock:432523
2018-05-24_10:52:59.97037 executable:/opt/gitlab/embedded/bin/redis-server
2018-05-24_10:52:59.97038 config_file:/var/opt/gitlab/redis/redis.conf
2018-05-24_10:52:59.97038
2018-05-24_10:52:59.97038 # Clients
2018-05-24_10:52:59.97038 connected_clients:0
2018-05-24_10:52:59.97039 client_longest_output_list:0
2018-05-24_10:52:59.97039 client_biggest_input_buf:0
2018-05-24_10:52:59.97039 blocked_clients:0
2018-05-24_10:52:59.97040
2018-05-24_10:52:59.97040 # Memory
2018-05-24_10:52:59.97040 used_memory:1026720
2018-05-24_10:52:59.97040 used_memory_human:1002.66K
2018-05-24_10:52:59.97041 used_memory_rss:0
2018-05-24_10:52:59.97041 used_memory_rss_human:0B
2018-05-24_10:52:59.97041 used_memory_peak:1026720
2018-05-24_10:52:59.97042 used_memory_peak_human:1002.66K
2018-05-24_10:52:59.97042 total_system_memory:2097258496
2018-05-24_10:52:59.97042 total_system_memory_human:1.95G
2018-05-24_10:52:59.97042 used_memory_lua:37888
2018-05-24_10:52:59.97043 used_memory_lua_human:37.00K
2018-05-24_10:52:59.97043 maxmemory:0
2018-05-24_10:52:59.97043 maxmemory_human:0B
2018-05-24_10:52:59.97044 maxmemory_policy:noeviction
2018-05-24_10:52:59.97044 mem_fragmentation_ratio:0.00
2018-05-24_10:52:59.97044 mem_allocator:jemalloc-4.0.3
2018-05-24_10:52:59.97045
2018-05-24_10:52:59.97045 # Persistence
2018-05-24_10:52:59.97045 loading:1
2018-05-24_10:52:59.97045 rdb_changes_since_last_save:0
2018-05-24_10:52:59.97046 rdb_bgsave_in_progress:0
2018-05-24_10:52:59.97047 rdb_last_save_time:1527159179
2018-05-24_10:52:59.97047 rdb_last_bgsave_status:ok
2018-05-24_10:52:59.97047 rdb_last_bgsave_time_sec:-1
2018-05-24_10:52:59.97048 rdb_current_bgsave_time_sec:-1
2018-05-24_10:52:59.97048 aof_enabled:0
2018-05-24_10:52:59.97048 aof_rewrite_in_progress:0
2018-05-24_10:52:59.97049 aof_rewrite_scheduled:0
2018-05-24_10:52:59.97049 aof_last_rewrite_time_sec:-1
2018-05-24_10:52:59.97049 aof_current_rewrite_time_sec:-1
2018-05-24_10:52:59.97049 aof_last_bgrewrite_status:ok
2018-05-24_10:52:59.97050 aof_last_write_status:ok
2018-05-24_10:52:59.97050 loading_start_time:1527159179
2018-05-24_10:52:59.97050 loading_total_bytes:457327
2018-05-24_10:52:59.97051 loading_loaded_bytes:0
2018-05-24_10:52:59.97051 loading_loaded_perc:0.00
2018-05-24_10:52:59.97051 loading_eta_seconds:1
2018-05-24_10:52:59.97051
2018-05-24_10:52:59.97052 # Stats
2018-05-24_10:52:59.97052 total_connections_received:0
2018-05-24_10:52:59.97052 total_commands_processed:0
2018-05-24_10:52:59.97053 instantaneous_ops_per_sec:0
2018-05-24_10:52:59.97053 total_net_input_bytes:0
2018-05-24_10:52:59.97053 total_net_output_bytes:0
2018-05-24_10:52:59.97054 instantaneous_input_kbps:0.00
2018-05-24_10:52:59.97054 instantaneous_output_kbps:0.00
2018-05-24_10:52:59.97055 rejected_connections:0
2018-05-24_10:52:59.97055 sync_full:0
2018-05-24_10:52:59.97055 sync_partial_ok:0
2018-05-24_10:52:59.97056 sync_partial_err:0
2018-05-24_10:52:59.97056 expired_keys:0
2018-05-24_10:52:59.97056 evicted_keys:0
2018-05-24_10:52:59.97056 keyspace_hits:0
2018-05-24_10:52:59.97057 keyspace_misses:0
2018-05-24_10:52:59.97057 pubsub_channels:0
2018-05-24_10:52:59.97057 pubsub_patterns:0
2018-05-24_10:52:59.97058 latest_fork_usec:0
2018-05-24_10:52:59.97058 migrate_cached_sockets:0
2018-05-24_10:52:59.97058
2018-05-24_10:52:59.97059 # Replication
2018-05-24_10:52:59.97059 role:master
2018-05-24_10:52:59.97059 connected_slaves:0
2018-05-24_10:52:59.97059 master_repl_offset:0
2018-05-24_10:52:59.97060 repl_backlog_active:0
2018-05-24_10:52:59.97060 repl_backlog_size:1048576
2018-05-24_10:52:59.97061 repl_backlog_first_byte_offset:0
2018-05-24_10:52:59.97061 repl_backlog_histlen:0
2018-05-24_10:52:59.97062
2018-05-24_10:52:59.97062 # CPU
2018-05-24_10:52:59.97062 used_cpu_sys:0.00
2018-05-24_10:52:59.97063 used_cpu_user:0.00
2018-05-24_10:52:59.97063 used_cpu_sys_children:0.00
2018-05-24_10:52:59.97063 used_cpu_user_children:0.00
2018-05-24_10:52:59.97063
2018-05-24_10:52:59.97064 # Commandstats
2018-05-24_10:52:59.97064
2018-05-24_10:52:59.97064 # Cluster
2018-05-24_10:52:59.97065 cluster_enabled:0
2018-05-24_10:52:59.97065
2018-05-24_10:52:59.97065 # Keyspace
2018-05-24_10:52:59.97065 db0:keys=1091,expires=487,avg_ttl=0
2018-05-24_10:52:59.97066 hash_init_value: 1527254982
2018-05-24_10:52:59.97067
2018-05-24_10:52:59.97068 ------ CLIENT LIST OUTPUT ------
2018-05-24_10:52:59.97070
2018-05-24_10:52:59.97070 ------ REGISTERS ------
2018-05-24_10:52:59.97073 12106:M 24 May 12:52:59.970 #
2018-05-24_10:52:59.97073 RAX:0000000000000000 RBX:00000000000000a1
2018-05-24_10:52:59.97073 RCX:00007f0c724492dd RDX:0000000000000000
2018-05-24_10:52:59.97074 RDI:0000000000000000 RSI:00007f0c72718780
2018-05-24_10:52:59.97074 RBP:00000000004d6f7d RSP:00007fff5800c950
2018-05-24_10:52:59.97074 R8 :00007f0c72718780 R9 :00007f0c7305d780
2018-05-24_10:52:59.97074 R10:000000000000002a R11:0000000000000000
2018-05-24_10:52:59.97075 R12:00000000004d4cfe R13:0000000000000000
2018-05-24_10:52:59.97075 R14:0000000000000000 R15:00007f0c71eb34c0
2018-05-24_10:52:59.97075 RIP:000000000045d12a EFL:0000000000010202
2018-05-24_10:52:59.97076 CSGSFS:0000000000000033
2018-05-24_10:52:59.97078 12106:M 24 May 12:52:59.970 # (00007fff5800c95f) -> 0000000000470ad0
2018-05-24_10:52:59.97080 12106:M 24 May 12:52:59.970 # (00007fff5800c95e) -> 0000000000470ab0
2018-05-24_10:52:59.97082 12106:M 24 May 12:52:59.970 # (00007fff5800c95d) -> 00000163c5647d00
2018-05-24_10:52:59.97084 12106:M 24 May 12:52:59.970 # (00007fff5800c95c) -> 0000000000000001
2018-05-24_10:52:59.97087 12106:M 24 May 12:52:59.970 # (00007fff5800c95b) -> 0000000000000007
2018-05-24_10:52:59.97089 12106:M 24 May 12:52:59.970 # (00007fff5800c95a) -> 0000016391c7cabf
2018-05-24_10:52:59.97091 12106:M 24 May 12:52:59.970 # (00007fff5800c959) -> 00007f0c71e16683
2018-05-24_10:52:59.97093 12106:M 24 May 12:52:59.970 # (00007fff5800c958) -> 0000000000000002
2018-05-24_10:52:59.97095 12106:M 24 May 12:52:59.970 # (00007fff5800c957) -> 0000000000443fe6
2018-05-24_10:52:59.97097 12106:M 24 May 12:52:59.970 # (00007fff5800c956) -> 0000000001da7e60
2018-05-24_10:52:59.97100 12106:M 24 May 12:52:59.970 # (00007fff5800c955) -> 00007f0c71e27800
2018-05-24_10:52:59.97102 12106:M 24 May 12:52:59.971 # (00007fff5800c954) -> ffffffffffffffff
2018-05-24_10:52:59.97104 12106:M 24 May 12:52:59.971 # (00007fff5800c953) -> 000000000043aa6f
2018-05-24_10:52:59.97106 12106:M 24 May 12:52:59.971 # (00007fff5800c952) -> 00007f0c71e27800
2018-05-24_10:52:59.97108 12106:M 24 May 12:52:59.971 # (00007fff5800c951) -> 00007f0c71eb34c0
2018-05-24_10:52:59.97110 12106:M 24 May 12:52:59.971 # (00007fff5800c950) -> 00007f0c71f78618
2018-05-24_10:52:59.97113
2018-05-24_10:52:59.97113 ------ FAST MEMORY TEST ------
2018-05-24_10:52:59.97120 12106:M 24 May 12:52:59.971 # Bio thread for job type #0 terminated
2018-05-24_10:52:59.97126 12106:M 24 May 12:52:59.971 # Bio thread for job type #1 terminated
2018-05-24_10:52:59.97133 *** Preparing to test memory region 718000 (94208 bytes)
2018-05-24_10:52:59.97135 *** Preparing to test memory region 1d9b000 (135168 bytes)
2018-05-24_10:52:59.97137 *** Preparing to test memory region 7f0c709ff000 (10485760 bytes)
2018-05-24_10:52:59.97138 *** Preparing to test memory region 7f0c71400000 (14680064 bytes)
2018-05-24_10:52:59.97141 *** Preparing to test memory region 7f0c72718000 (16384 bytes)
2018-05-24_10:52:59.97143 *** Preparing to test memory region 7f0c72935000 (16384 bytes)
2018-05-24_10:52:59.97146 *** Preparing to test memory region 7f0c7305d000 (20480 bytes)
2018-05-24_10:52:59.97149 *** Preparing to test memory region 7f0c7306d000 (4096 bytes)
2018-05-24_10:52:59.97151 .O.O.O.O.O.O.O.O
2018-05-24_10:53:00.05269 Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
2018-05-24_10:53:00.05276
2018-05-24_10:53:00.05276 ------ DUMPING CODE AROUND EIP ------
2018-05-24_10:53:00.05277 Symbol: _serverAssert (base: 0x45d0c0)
2018-05-24_10:53:00.05278 Module: /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0 (base 0x400000)
2018-05-24_10:53:00.05278 $ xxd -r -p /tmp/dump.hex /tmp/dump.bin
2018-05-24_10:53:00.05278 $ objdump --adjust-vma=0x45d0c0 -D -b binary -m i386:x86-64 /tmp/dump.bin
2018-05-24_10:53:00.05279 ------
2018-05-24_10:53:00.05284 12106:M 24 May 12:53:00.052 # dump of function (hexdump of 234 bytes):
2018-05-24_10:53:00.05286 41548b05f4c72b004989fc554889f585c05389d37505e8a5fdffffbeb6b54d00bf0300000031c0e8247cfcff4d89e089d94889eabecfb54d00bf0300000031c0e80b7cfcffbe60a74d00bf0300000031c04c892590c72b0048892d91c72b00891d93c72b00e8e67bfcffc60425ffffffff785b5d415cc3660f1f84000000000041544989fc55534881eca00000008b1568c72b0064488b042528000000488984249800000031c085d20f842101000031c0be90a74d00bf03000000e8907bfcff418b94249800000031c0beeab54d00bf0300000031ed31dbe8737bfcff418b54240831c0befdb54d00bf
2018-05-24_10:53:00.05305 Function at 0x424d10 is serverLog
2018-05-24_10:53:00.05305
2018-05-24_10:53:00.05305 === REDIS BUG REPORT END. Make sure to include from START to END. ===
2018-05-24_10:53:00.05306
2018-05-24_10:53:00.05306 Please report the crash by opening an issue on github:
2018-05-24_10:53:00.05306
2018-05-24_10:53:00.05307 http://github.com/antirez/redis/issues
2018-05-24_10:53:00.05307
2018-05-24_10:53:00.05307 Suspect RAM error? Use redis-server --test-memory to verify it.
2018-05-24_10:53:00.05307
I’ve updated gitlab and everything but the problem persists. Also I can’t run gitlab-ctl reconfigure as it fails as Redis is not running.
$ gitlab-ctl reconfigure
...
Recipe: gitlab::database_migrations
* bash[migrate gitlab-rails database] action run
[execute] rake aborted!
StandardError: An error has occurred, all later migrations canceled:
Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ECONNREFUSED)
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status.rb:32:in `block in set'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status.rb:31:in `set'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/client_middleware.rb:7:in `call'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/forbid_sidekiq_in_transactions.rb:35:in `block (2 levels) in <module:NoEnqueueingFromTransactions>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:901:in `block in queue_background_migration_jobs_by_range_at_intervals'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:75:in `block in each_batch'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:56:in `step'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:56:in `each_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:895:in `queue_background_migration_jobs_by_range_at_intervals'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20180122154930_schedule_set_confidential_note_events_on_services.rb:15:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
...
Thank you.