hey, i’m having an issue getting objects stored into an s3 bucket. when i upload something to my self-hosted repo, nothing appears in s3.
so i spotted an error:
==> /var/log/gitlab/gitlab-workhorse/current <==
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-07T21:31:17Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-04-07T21:31:17Z"}
this is an error i see in the output when i try to upload a picture to a repo that im managing.
when i check that log file, i see stuff like this:
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-07T19:46:33Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-04-07T19:46:33Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01GXEM4TJHF4R081TXWG1QP8ET","duration_ms":89,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2023-04-07T19:47:26Z","ttfb_ms":88,"uri":"/gitlab/help","user_agent":"curl/8.0.1-DEV","written_bytes":71928}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-07T19:47:34Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-04-07T19:47:34Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01GXEM6P4P3NFG6KK3JQ9P7ARA","duration_ms":87,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2023-04-07T19:48:27Z","ttfb_ms":87,"uri":"/gitlab/help","user_agent":"curl/8.0.1-DEV","written_bytes":71928}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-07T19:48:35Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-04-07T19:48:35Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01GXEM8HQQDSA37NH34Y87Z35W","duration_ms":121,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2023-04-07T19:49:28Z","ttfb_ms":121,"uri":"/gitlab/help","user_agent":"curl/8.0.1-DEV","written_bytes":71928}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-07T19:49:36Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-04-07T19:49:36Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01GXEMAD9BZXVRTQPVD78PW35Q","duration_ms":91,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2023-04-07T19:50:29Z","ttfb_ms":91,"uri":"/gitlab/help","user_agent":"curl/8.0.1-DEV","written_bytes":71928}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-07T19:50:37Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-04-07T19:50:37Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01GXEMC8VGM92RGMDCMP3VTYP3","duration_ms":88,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2023-04-07T19:51:30Z","ttfb_ms":88,"uri":"/gitlab/help","user_agent":"curl/8.0.1-DEV","written_bytes":71928}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-07T19:51:38Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-04-07T19:51:38Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01GXEME3NJT1DQHJ1J893DWSG7","duration_ms":87,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2023-04-07T19:52:30Z","ttfb_ms":87,"uri":"/gitlab/help","user_agent":"curl/8.0.1-DEV","written_bytes":71928}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-07T19:52:39Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-04-07T19:52:39Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01GXEMFZ50G99BNZHP08P46SZG","duration_ms":354,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2023-04-07T19:53:31Z","ttfb_ms":353,"uri":"/gitlab/help","user_agent":"curl/8.0.1-DEV","written_bytes":71928}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-07T19:53:40Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-04-07T19:53:40Z"}
{"level":"info","msg":"shutdown initiated","shutdown_timeout_s":0,"signal":"terminated","time":"2023-04-07T19:54:04Z"}
{"level":"info","msg":"keywatcher: shutting down","time":"2023-04-07T19:54:04Z"}
{"error":"context deadline exceeded","level":"fatal","msg":"shutting down","time":"2023-04-07T19:54:04Z"}
{"build_time":"20230330.174908","level":"info","msg":"Starting","time":"2023-04-07T19:54:04Z","version":"v15.8.5-ee"}
{"address":"localhost:9229","level":"info","msg":"Running metrics server","network":"tcp","time":"2023-04-07T19:54:04Z"}
{"level":"info","msg":"keywatcher: starting process loop","time":"2023-04-07T19:54:04Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-04-07T19:54:04Z"}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"","time":"2023-04-07T19:54:04Z","uri":""}
{"address":"/var/opt/gitlab/gitlab-workhorse/sockets/socket","level":"info","msg":"Running upstream server","network":"unix","time":"2023-04-07T19:54:04Z"}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"","time":"2023-04-07T19:54:14Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"","time":"2023-04-07T19:54:24Z","uri":""}
{"correlation_id":"01GXEMHTNAJ16JK0SWENQ5CXFA","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"","time":"2023-04-07T19:54:32Z","uri":"/gitlab/help"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01GXEMHTNAJ16JK0SWENQ5CXFA","duration_ms":0,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":502,"system":"http","time":"2023-04-07T19:54:32Z","ttfb_ms":0,"uri":"/gitlab/help","user_agent":"curl/8.0.1-DEV","written_bytes":3039}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"","time":"2023-04-07T19:54:34Z","uri":""}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-07T19:55:05Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-04-07T19:55:05Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01GXEMKP599T40Z0YJ7R3JW6HP","duration_ms":479,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2023-04-07T19:55:33Z","ttfb_ms":479,"uri":"/gitlab/help","user_agent":"curl/8.0.1-DEV","written_bytes":71928}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-07T19:56:06Z"}
this is what stands out to me: /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory
so, i check:
# cd /var/opt/gitlab/gitlab-rails/sockets/
# ls -la
total 8
drwxr-x--- 2 git gitlab-www 4096 Apr 7 21:29 .
drwxr-xr-x 9 git root 4096 Apr 7 21:03 ..
srwxrwxrwx 1 git git 0 Apr 7 21:29 gitlab.socket
from chat gpt:
The “s” in “srwxrwxrwx” represents a Unix file permission bit that can
appear in the permissions string of a Unix Domain Socket file. The “s”
indicates that the file has a setuid bit set, which means that the
file will be executed with the privileges of its owner or group,
rather than with the privileges of the user who executes it. In the
case of a Unix Domain Socket file, the setuid bit can be used to
ensure that only the owner or members of a particular group can
communicate with the socket
so… i guess only git can use this socket file?
completely confused about what i can do here, since i have gitlab running in a container - and all these were checked inside the container.
config:
gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['connection'] = {
'provider' => 'AWS',
'region' => 'us-east-1',
'aws_access_key_id' => 'CKKJJJJE88JWM',
'aws_secret_access_key' => 'IVchgkjhgkjhgfffffffffff6DA+56',
'enable_signature_v4_streaming' => 'true',
}
gitlab_rails['object_store']['storage_options'] = {
'server_side_encryption' => 'aws:kms',
'server_side_encryption_kms_key_id' => 'arn:aws:kms:us-east-1:13777777:key/4602-d6d35'
}
gitlab_rails['object_store']['proxy_download'] = false
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'my-bucket/artifacts'
gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'my-bucket/external_diffs'
gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'my-bucket/lfs'
gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'my-bucket/uploads'
gitlab_rails['object_store']['objects']['packages']['bucket'] = 'my-bucket/packages'
gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'my-bucket/dependency_proxy'
gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'my-bucket/terraform_state'
gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'my-bucket/ci_secure_files'
what is the approach to debugging this? thanks for the help.
edit:
not sure if this is related, but i looked at redis logs:
2023-04-07_17:14:20.48359 511:C 07 Apr 2023 17:14:20.483 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2023-04-07_17:14:20.48367 511:C 07 Apr 2023 17:14:20.483 # Redis version=6.2.11, bits=64, commit=720ea82e, modified=1, pid=511, just started
2023-04-07_17:14:20.48367 511:C 07 Apr 2023 17:14:20.483 # Configuration loaded
2023-04-07_17:14:20.48391 511:M 07 Apr 2023 17:14:20.483 * monotonic clock: POSIX clock_gettime
2023-04-07_17:14:20.48424 _._
2023-04-07_17:14:20.48425 _.-``__ ''-._
2023-04-07_17:14:20.48425 _.-`` `. `_. ''-._ Redis 6.2.11 (720ea82e/1) 64 bit
2023-04-07_17:14:20.48425 .-`` .-```. ```\/ _.,_ ''-._
2023-04-07_17:14:20.48425 ( ' , .-` | `, ) Running in standalone mode
2023-04-07_17:14:20.48426 |`-._`-...-` __...-.``-._|'` _.-'| Port: 0
2023-04-07_17:14:20.48426 | `-._ `._ / _.-' | PID: 511
2023-04-07_17:14:20.48426 `-._ `-._ `-./ _.-' _.-'
2023-04-07_17:14:20.48426 |`-._`-._ `-.__.-' _.-'_.-'|
2023-04-07_17:14:20.48427 | `-._`-._ _.-'_.-' | https://redis.io
2023-04-07_17:14:20.48427 `-._ `-._`-.__.-'_.-' _.-'
2023-04-07_17:14:20.48427 |`-._`-._ `-.__.-' _.-'_.-'|
2023-04-07_17:14:20.48427 | `-._`-._ _.-'_.-' |
2023-04-07_17:14:20.48427 `-._ `-._`-.__.-'_.-' _.-'
2023-04-07_17:14:20.48428 `-._ `-.__.-' _.-'
2023-04-07_17:14:20.48428 `-._ _.-'
2023-04-07_17:14:20.48428 `-.__.-'
2023-04-07_17:14:20.48429
2023-04-07_17:14:20.48429 511:M 07 Apr 2023 17:14:20.484 # Server initialized
2023-04-07_17:14:20.48429 511:M 07 Apr 2023 17:14:20.484 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2023-04-07_17:14:20.48446 511:M 07 Apr 2023 17:14:20.484 * The server is now ready to accept connections at /var/opt/gitlab/redis/redis.socket
2023-04-07_17:19:21.07981 511:M 07 Apr 2023 17:19:21.079 * 10 changes in 300 seconds. Saving...
2023-04-07_17:19:21.08012 511:M 07 Apr 2023 17:19:21.080 * Background saving started by pid 1388
2023-04-07_17:19:21.08684 1388:C 07 Apr 2023 17:19:21.086 * DB saved on disk
2023-04-07_17:19:21.08709 1388:C 07 Apr 2023 17:19:21.087 * RDB: 1 MB of memory used by copy-on-write
2023-04-07_17:19:21.18093 511:M 07 Apr 2023 17:19:21.180 * Background saving terminated with success
2023-04-07_17:24:22.05724 511:M 07 Apr 2023 17:24:22.057 * 10 changes in 300 seconds. Saving...
2023-04-07_17:24:22.05759 511:M 07 Apr 2023 17:24:22.057 * Background saving started by pid 1493
edit 2: tried to use the community edition:
==> /var/log/gitlab/gitlab-workhorse/current <==
{"correlation_id":"01GXH4E8F4WEAB8ZHYR8JDW5E8","error":"blob.GetBlob: copy rpc data: rpc error: code = Canceled desc = context canceled","level":"error","method":"GET","msg":"","time":"2023-04-08T19:10:41Z","uri":"/gitlab/gitlab-instance-49242dc9/gitlab_object_store_test/-/raw/main/musk2.jpg"}
{"content_type":"image/jpeg","correlation_id":"01GXH4E8F4WEAB8ZHYR8JDW5E8","duration_ms":68,"host":"10.235.77.28","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"http://10.235.77.28/gitlab/gitlab-instance-49242dc9/gitlab_object_store_test/-/blob/main/musk2.jpg","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2023-04-08T19:10:41Z","ttfb_ms":63,"uri":"/gitlab/gitlab-instance-49242dc9/gitlab_object_store_test/-/raw/main/musk2.jpg","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","written_bytes":401518}
"blob.GetBlob: copy rpc data: rpc error: code = Canceled desc = context canceled"
The error message “blob.GetBlob: copy rpc data: rpc error: code = Canceled desc = context canceled” typically means that a remote procedure call (RPC) was canceled due to a timeout or other issue with the network or system.
In the context of GitLab, this error can occur when GitLab Workhorse, a reverse proxy server used by GitLab, is unable to complete a request for a Git repository due to a timeout or other issue with the Git server or network connection. This can happen if the Git server is overloaded, if there are issues with the network connection between the GitLab server and the Git server, or if there are other issues with the GitLab server itself.
when i go to http://10.235.77.28/gitlab/gitlab-instance-49242dc9/gitlab_object_store_test/-/blob/main/musk2.jpg i am able to see the pictures on my self-hosted gitlab website…