I have a gitlab installation in Azure Kubernetes Service. When I try to upload a profile picture I get a 500 error code with the following log trace. I’ve diagnosed that the object storage is not configured properly and it’s failing to connect to azure blob storage. Is my diagnosis correct?
{"component": "gitlab","subcomponent":"api_json","level":"info","time":"2023-01-27T04:29:09.290Z","severity":"INFO","duration_s":0.00275,"db_duration_s":0.0,"view_duration_s":0.00275,"status":200,"method":"POST","path":"/api/v4/internal/workhorse/authorize_upload","params":[],"host":"gitlab.gtims.gitlabsaas.jaspersolutions.com","remote_ip":"108.31.94.197","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0","route":"/api/:version/internal/workhorse/authorize_upload","redis_calls":6,"redis_allowed_cross_slot_calls":1,"redis_duration_s":0.0027480000000000004,"redis_read_bytes":386,"redis_write_bytes":1271,"redis_cache_calls":1,"redis_cache_duration_s":0.000447,"redis_cache_read_bytes":203,"redis_cache_write_bytes":58,"redis_sessions_calls":4,"redis_sessions_allowed_cross_slot_calls":1,"redis_sessions_duration_s":0.001962,"redis_sessions_read_bytes":183,"redis_sessions_write_bytes":1160,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000339,"redis_shared_state_write_bytes":53,"db_count":2,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":2,"db_main_count":2,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.008,"db_main_duration_s":0.008,"db_main_replica_duration_s":0.0,"cpu_s":0.031787,"mem_objects":9592,"mem_bytes":1100423,"mem_mallocs":2863,"mem_total_bytes":1484103,"pid":40,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01GQRQS2QXQ7JGCTA3R7B3T3ZH","meta.caller_id":"POST /api/:version/internal/workhorse/authorize_upload","meta.remote_ip":"108.31.94.197","meta.feature_category":"not_owned","meta.client_id":"ip/108.31.94.197","content_length":"0","request_urgency":"default","target_duration_s":1}
{"component": "gitlab","subcomponent":"production","level":"info","time":"2023-01-27T04:29:09Z","message":"Started PUT \"/-/profile\" for 108.31.94.197 at 2023-01-27 04:29:09 +0000"}
{"component": "gitlab","subcomponent":"production","level":"info","time":"2023-01-27T04:29:09Z","message":"Processing by ProfilesController#update as JSON"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":" Parameters: {\"authenticity_token\"=>\"[FILTERED]\", \"user\"=>{\"status\"=>{\"emoji\"=>\"\", \"message\"=>\"\", \"availability\"=>\"not_set\"}, \"timezone\"=>\"\", \"name\"=>\"Administrator\", \"id\"=>\"1\", \"pronouns\"=>\"\", \"pronunciation\"=>\"\", \"email\"=>\"gitlabsaasadmin@jaspersolutions.com\", \"validation_password\"=>\"[FILTERED]\", \"public_email\"=>\"\", \"commit_email\"=>\"\", \"skype\"=>\"\", \"linkedin\"=>\"\", \"twitter\"=>\"\", \"website_url\"=>\"\", \"location\"=>\"\", \"job_title\"=>\"\", \"organization\"=>\"\", \"bio\"=>\"\", \"private_profile\"=>\"0\", \"include_private_contributions\"=>\"0\", \"avatar\"=>#<UploadedFile:0x00007ff1777c23b8 @tempfile=#<File:/srv/gitlab/public/uploads/tmp/gitlab-workhorse-upload1388595669>, @size=73121, @upload_duration=0.001514433, @content_type=\"application/octet-stream\", @original_filename=\"avatar.png\", @sha256=\"9402b74205b9486148b53e39464a6eeafb3a64c99437f6f1997cfe27a073a524\", @remote_id=\"\">}}"}
{"component": "gitlab","subcomponent":"production","level":"error","time":"2023-01-27T04:29:09Z","message":"Completed 500 Internal Server Error in 34ms (ActiveRecord: 7.6ms | Elasticsearch: 0.0ms | Allocations: 9179)"}
{"component": "gitlab","subcomponent":"production_json","level":"error","method":"PUT","path":"/-/profile","format":"json","controller":"ProfilesController","action":"update","status":500,"time":"2023-01-27T04:29:09.353Z","params":[{"key":"_method","value":"put"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"user","value":{"status":{"emoji":"","message":"","availability":"not_set"},"timezone":"","name":"Administrator","id":"1","pronouns":"","pronunciation":"","email":"gitlabsaasadmin@jaspersolutions.com","validation_password":"[FILTERED]","public_email":"","commit_email":"","skype":"","linkedin":"","twitter":"","website_url":"","location":"","job_title":"","organization":"","bio":"","private_profile":"0","include_private_contributions":"0","avatar":"#<UploadedFile:0x00007ff1777c23b8>"}}],"correlation_id":"01GQRQS2QXQ7JGCTA3R7B3T3ZH","meta.caller_id":"ProfilesController#update","meta.remote_ip":"108.31.94.197","meta.feature_category":"users","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","remote_ip":"108.31.94.197","user_id":1,"username":"root","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0","queue_duration_s":0.01175,"request_urgency":"low","target_duration_s":5,"redis_calls":8,"redis_allowed_cross_slot_calls":1,"redis_duration_s":0.002558,"redis_read_bytes":690,"redis_write_bytes":1356,"redis_cache_calls":3,"redis_cache_duration_s":0.001173,"redis_cache_read_bytes":507,"redis_cache_write_bytes":147,"redis_sessions_calls":4,"redis_sessions_allowed_cross_slot_calls":1,"redis_sessions_duration_s":0.001062,"redis_sessions_read_bytes":183,"redis_sessions_write_bytes":1156,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000323,"redis_shared_state_write_bytes":53,"db_count":8,"db_write_count":3,"db_cached_count":0,"db_replica_count":0,"db_primary_count":8,"db_main_count":8,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.009,"db_main_duration_s":0.009,"db_main_replica_duration_s":0.0,"cpu_s":0.04528,"mem_objects":14238,"mem_bytes":1365312,"mem_mallocs":3246,"mem_total_bytes":1934832,"pid":40,"worker_id":"puma_0","uploaded_file_upload_duration_s":0.001514433,"uploaded_file_size_bytes":73121,"rate_limiting_gates":[],"exception.class":"NoMethodError","exception.message":"undefined method `create_block_blob' for nil:NilClass","exception.backtrace":["app/uploaders/object_storage.rb:343:in `store!'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:129:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:200:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:118:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","app/services/users/update_service.rb:36:in `execute'","app/controllers/profiles_controller.rb:28:in `block in update'","app/controllers/profiles_controller.rb:27:in `update'","ee/lib/gitlab/ip_address_state.rb:10:in `with'","ee/app/controllers/ee/application_controller.rb:46:in `set_current_ip_address'","app/controllers/application_controller.rb:534:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:525:in `set_session_storage'","lib/gitlab/i18n.rb:107:in `with_locale'","lib/gitlab/i18n.rb:113:in `with_user_locale'","app/controllers/application_controller.rb:513:in `set_locale'","app/controllers/application_controller.rb:507:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:37:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/gitlab/middleware/multipart.rb:178:in `block in call'","lib/gitlab/middleware/multipart.rb:63:in `with_open_files'","lib/gitlab/middleware/multipart.rb:177:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:37:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'","lib/gitlab/middleware/release_env.rb:13:in `call'"],"db_duration_s":0.00765,"view_duration_s":0.0,"duration_s":0.03465}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":" "}
{"component": "gitlab","subcomponent":"production","level":"error","time":"2023-01-27T04:29:09Z","message":"NoMethodError (undefined method `create_block_blob' for nil:NilClass):"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":" "}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"app/uploaders/object_storage.rb:343:in `store!'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/database/load_balancing/load_balancer.rb:129:in `block in read_write'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/database/load_balancing/load_balancer.rb:200:in `retry_with_backoff'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/database/load_balancing/load_balancer.rb:118:in `read_write'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"app/services/users/update_service.rb:36:in `execute'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"app/controllers/profiles_controller.rb:28:in `block in update'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"app/controllers/profiles_controller.rb:27:in `update'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"ee/lib/gitlab/ip_address_state.rb:10:in `with'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"ee/app/controllers/ee/application_controller.rb:46:in `set_current_ip_address'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"app/controllers/application_controller.rb:534:in `set_current_admin'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/session.rb:11:in `with_session'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"app/controllers/application_controller.rb:525:in `set_session_storage'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/i18n.rb:107:in `with_locale'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/i18n.rb:113:in `with_user_locale'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"app/controllers/application_controller.rb:513:in `set_locale'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"app/controllers/application_controller.rb:507:in `set_current_context'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/memory_report.rb:13:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/speedscope.rb:13:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/metrics/web_transaction.rb:46:in `run'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/metrics/rack_middleware.rb:16:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/jira/middleware.rb:19:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/go.rb:20:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/etag_caching/middleware.rb:21:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/database/query_analyzer.rb:37:in `within'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/query_analyzer.rb:11:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/multipart.rb:178:in `block in call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/multipart.rb:63:in `with_open_files'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/multipart.rb:177:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/read_only/controller.rb:50:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/read_only.rb:18:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/basic_health_check.rb:25:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"error","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/request_context.rb:21:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"config/initializers/fix_local_cache_middleware.rb:11:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/compressed_json.rb:37:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2023-01-27T04:29:09Z","message":"lib/gitlab/middleware/release_env.rb:13:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"info","time":"2023-01-27T04:29:10Z","message":"Started GET \"/-/readiness\" for 10.244.1.1 at 2023-01-27 04:29:10 +0000"}
{"component": "gitlab","subcomponent":"production","level":"info","time":"2023-01-27T04:29:10Z","message":"Processing by HealthController#readiness as */*"}
{"component": "gitlab","subcomponent":"production","level":"info","time":"2023-01-27T04:29:10Z","message":"Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 238)"}
Here are the relevant sections of my values.yaml for the helm chart.
global:
## https://docs.gitlab.com/charts/installation/deployment#deploy-the-community-edition
edition: ee
## https://docs.gitlab.com/charts/charts/globals#configure-host-settings
hosts:
domain: redacted
externalIP: redacted
## https://docs.gitlab.com/charts/charts/globals#configure-minio-settings
minio:
enabled: false
## https://docs.gitlab.com/charts/charts/globals#configure-grafana-integration
grafana:
enabled: true
## https://docs.gitlab.com/charts/charts/globals#configure-appconfig-settings
## Rails based portions of this chart share many settings
appConfig:
## https://docs.gitlab.com/charts/charts/globals#lfs-artifacts-uploads-packages-external-mr-diffs-and-dependency-proxy
object_store:
enabled: true
proxy_download: true
storage_options: {}
# server_side_encryption:
# server_side_encryption_kms_key_id
connection:
secret: object-storage
key: connection
This is the content of my connection string in the secret named object-storage
provider: AzureRM
region: usgovvirginia
azure_storage_account_name: <redacted>
azure_storage_access_key: <redacted>
azure_storage_domain: blob.core.usgovcloudapi.net
Has anybody successfully configured azure object storage using the helm chart values.yml file?