Uploaded images don't shows up with custom width

SofianeLasri · January 26, 2022, 1:04pm

Hello, I have a problem with my GitLab installation.
I can’t visualize my uploaded pics such as my profile pic and repositories pics.

I searched a little about that and I found that this doesn’t occure with all width. For example, if I choose a width of 18, the pic will shows up. But that’s not the case if I choose 15.

https://gitlab.sl-projects.com/uploads/-/system/project/avatar/7/favicon-color.png?width=15
https://gitlab.sl-projects.com/uploads/-/system/project/avatar/7/favicon-color.png?width=18

I checked the log with gitlab-ctl tail, but I didn’t find anything interesting.

## THIS IS WHEN I GET 18 WIDTH
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/uploads/-/system/project/avatar/7/favicon-color.png","format":"html","controller":"UploadsController","action":"show","status":200,"time":"2022-01-26T12:50:16.761Z","params":[{"key":"width","value":"18"},{"key":"model","value":"project"},{"key":"mounted_as","value":"avatar"},{"key":"id","value":"7"},{"key":"filename","value":"favicon-color.png"}],"remote_ip":"thisIsMyIp","user_id":2,"username":"SofianeLasri","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36 Edg/97.0.1072.69","correlation_id":"e3677b84-4e29-4867-aa6d-8eb27a0fe499","meta.user":"SofianeLasri","meta.caller_id":"UploadsController#show","meta.remote_ip":"thisIsMyIp","meta.feature_category":"not_owned","meta.client_id":"user/2","redis_calls":2,"redis_duration_s":0.000753,"redis_read_bytes":302,"redis_write_bytes":1463,"redis_cache_calls":1,"redis_cache_duration_s":0.000341,"redis_cache_read_bytes":121,"redis_cache_write_bytes":70,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000412,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":1393,"db_count":4,"db_write_count":0,"db_cached_count":0,"cpu_s":0.0276,"mem_objects":12865,"mem_bytes":1327051,"mem_mallocs":2871,"mem_total_bytes":1841651,"pid":1938615,"db_duration_s":0.00163,"view_duration_s":0.0,"duration_s":0.01308}

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/favicon.ico" for thisIsMyIp at 2022-01-26 13:50:16 +0100

==> /var/log/gitlab/gitlab-exporter/current <==
2022-01-26_12:50:17.22701 127.0.0.1 - - [26/Jan/2022:13:50:17 CET] "GET /sidekiq HTTP/1.1" 200 65685
2022-01-26_12:50:17.22705 - -> /sidekiq


## THIS IS WHEN I GET 15 WIDTH
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/uploads/-/system/project/avatar/7/favicon-color.png?width=15" for thisIsMyIp at 2022-01-26 13:50:17 +0100
Processing by UploadsController#show as HTML
  Parameters: {"width"=>"15", "model"=>"project", "mounted_as"=>"avatar", "id"=>"7", "filename"=>"favicon-color.png"}
Completed 200 OK in 17ms (ActiveRecord: 1.9ms | Elasticsearch: 0.0ms | Allocations: 4557)

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/uploads/-/system/project/avatar/7/favicon-color.png","format":"html","controller":"UploadsController","action":"show","status":200,"time":"2022-01-26T12:50:17.724Z","params":[{"key":"width","value":"15"},{"key":"model","value":"project"},{"key":"mounted_as","value":"avatar"},{"key":"id","value":"7"},{"key":"filename","value":"favicon-color.png"}],"remote_ip":"thisIsMyIp","user_id":2,"username":"SofianeLasri","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36 Edg/97.0.1072.69","correlation_id":"66876957-1b05-4ef9-9654-733a1e8fb423","meta.user":"SofianeLasri","meta.caller_id":"UploadsController#show","meta.remote_ip":"thisIsMyIp","meta.feature_category":"not_owned","meta.client_id":"user/2","redis_calls":1,"redis_duration_s":0.000235,"redis_read_bytes":181,"redis_write_bytes":746,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000235,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":746,"db_count":3,"db_write_count":0,"db_cached_count":0,"cpu_s":0.025518,"mem_objects":9898,"mem_bytes":743168,"mem_mallocs":1828,"mem_total_bytes":1139088,"pid":1938714,"db_duration_s":0.00188,"view_duration_s":0.0,"duration_s":0.01761}

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/favicon.ico" for thisIsMyIp at 2022-01-26 13:50:18 +0100

I installed GitLab with Apache2 throught a proxy. Everything work fine except this bug.
How can I fix that?

Thx in advance.

derekk19 · August 2, 2022, 12:58am

I’m getting the same behaviour. I have avatars for some of my projects and groups (all are png files less than 200Kb). The avatars never show up in the side menu or the project page and breadcrumb trail. One looking further I found (like Sofiane Lasri) that a call to https://…/uploads/-/system/project/avatar/19/Sailboat.png?width=20 gives an “unfound image” icon while https://…/uploads/-/system/project/avatar/19/Sailboat.png?width=25 displays the avatar - but full size (which is 600x600). There didn’t seem to be any pattern with width served an image - generally multiples of 5 did, but not aways!, and the image was always full-size)

I also looked at the output from gitlab-ctl tail and there were only 200 responses. The width=25 request was followed by a Sent file message:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET “/uploads/-/system/project/avatar/19/Sailboat.png?width=25” for 192.168.0.1 at 2022-08-02 12:41:44 +1200
Processing by UploadsController#show as HTML
Parameters: {“width”=>“25”, “model”=>“project”, “mounted_as”=>“avatar”, “id”=>“19”, “filename”=>“Sailboat.png”}
Sent file /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/project/avatar/19/Sailboat.png (0.3ms)
Completed 200 OK in 34ms (ActiveRecord: 4.1ms | Elasticsearch: 0.0ms | Allocations: 5316)

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{“method”:“GET”,“path”:“/uploads/-/system/project/avatar/19/Sailboat.png”,“format”:“html”,“controller”:“UploadsController”,“action”:“show”,“status”:200,“time”:“2022-08-02T00:41:47.936Z”,“params”:[{“key”:“width”,“value”:“25”},{“key”:“model”,“value”:“project”},{“key”:“mounted_as”,“value”:“avatar”},{“key”:“id”,“value”:“19”},{“key”:“filename”,“value”:“Sailboat.png”}],“correlation_id”:“63c3990d-5285-4f85-95e9-ef0e505a1649”,“meta.user”:“derek”,“meta.client_id”:“user/2”,“meta.caller_id”:“UploadsController#show”,“meta.remote_ip”:“192.168.0.1”,“meta.feature_category”:“not_owned”,“remote_ip”:“192.168.0.1”,“user_id”:2,“username”:“derek”,“ua”:“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6 Safari/605.1.15”,“request_urgency”:“default”,“target_duration_s”:1,“redis_calls”:2,“redis_duration_s”:0.000408,“redis_read_bytes”:183,“redis_write_bytes”:1197,“redis_shared_state_calls”:1,“redis_shared_state_duration_s”:0.000149,“redis_shared_state_write_bytes”:53,“redis_sessions_calls”:1,“redis_sessions_duration_s”:0.000259,“redis_sessions_read_bytes”:183,“redis_sessions_write_bytes”:1144,“db_count”:4,“db_write_count”:0,“db_cached_count”:0,“db_replica_count”:0,“db_primary_count”:4,“db_main_count”:4,“db_main_replica_count”:0,“db_replica_cached_count”:0,“db_primary_cached_count”:0,“db_main_cached_count”:0,“db_main_replica_cached_count”:0,“db_replica_wal_count”:0,“db_primary_wal_count”:0,“db_main_wal_count”:0,“db_main_replica_wal_count”:0,“db_replica_wal_cached_count”:0,“db_primary_wal_cached_count”:0,“db_main_wal_cached_count”:0,“db_main_replica_wal_cached_count”:0,“db_replica_duration_s”:0.0,“db_primary_duration_s”:0.005,“db_main_duration_s”:0.005,“db_main_replica_duration_s”:0.0,“cpu_s”:0.026733,“mem_objects”:12909,“mem_bytes”:1127184,“mem_mallocs”:2466,“mem_total_bytes”:1643544,“pid”:3177632,“worker_id”:“puma_1”,“rate_limiting_gates”:,“db_duration_s”:0.0036,“view_duration_s”:0.0,“duration_s”:0.01645}

But the width=20 request just had the 200 status (no Sent file message)
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET “/uploads/-/system/project/avatar/19/Sailboat.png?width=20” for 192.168.0.1 at 2022-08-02 12:44:11 +1200
Processing by UploadsController#show as HTML
Parameters: {“width”=>“20”, “model”=>“project”, “mounted_as”=>“avatar”, “id”=>“19”, “filename”=>“Sailboat.png”}
Completed 200 OK in 35ms (ActiveRecord: 4.1ms | Elasticsearch: 0.0ms | Allocations: 5270)

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{“method”:“GET”,“path”:“/uploads/-/system/project/avatar/19/Sailboat.png”,“format”:“html”,“controller”:“UploadsController”,“action”:“show”,“status”:200,“time”:“2022-08-02T00:44:11.401Z”,“params”:[{“key”:“width”,“value”:“20”},{“key”:“model”,“value”:“project”},{“key”:“mounted_as”,“value”:“avatar”},{“key”:“id”,“value”:“19”},{“key”:“filename”,“value”:“Sailboat.png”}],“correlation_id”:“4e9065f7-5cfa-4854-a78c-89718f8d0a9e”,“meta.user”:“derek”,“meta.client_id”:“user/2”,“meta.caller_id”:“UploadsController#show”,“meta.remote_ip”:“192.168.0.1”,“meta.feature_category”:“not_owned”,“remote_ip”:“192.168.0.1”,“user_id”:2,“username”:“derek”,“ua”:“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6 Safari/605.1.15”,“request_urgency”:“default”,“target_duration_s”:1,“redis_calls”:6,“redis_duration_s”:0.004209,“redis_read_bytes”:914,“redis_write_bytes”:1986,“redis_cache_calls”:4,“redis_cache_duration_s”:0.00287,“redis_cache_read_bytes”:731,“redis_cache_write_bytes”:250,“redis_shared_state_calls”:1,“redis_shared_state_duration_s”:0.000586,“redis_shared_state_write_bytes”:106,“redis_sessions_calls”:1,“redis_sessions_duration_s”:0.000753,“redis_sessions_read_bytes”:183,“redis_sessions_write_bytes”:1630,“db_count”:5,“db_write_count”:0,“db_cached_count”:0,“db_replica_count”:0,“db_primary_count”:5,“db_main_count”:5,“db_main_replica_count”:0,“db_replica_cached_count”:0,“db_primary_cached_count”:0,“db_main_cached_count”:0,“db_main_replica_cached_count”:0,“db_replica_wal_count”:0,“db_primary_wal_count”:0,“db_main_wal_count”:0,“db_main_replica_wal_count”:0,“db_replica_wal_cached_count”:0,“db_primary_wal_cached_count”:0,“db_main_wal_cached_count”:0,“db_main_replica_wal_cached_count”:0,“db_replica_duration_s”:0.0,“db_primary_duration_s”:0.013,“db_main_duration_s”:0.013,“db_main_replica_duration_s”:0.0,“cpu_s”:0.071973,“mem_objects”:16816,“mem_bytes”:1837974,“mem_mallocs”:3875,“mem_total_bytes”:2510614,“pid”:3176785,“worker_id”:“puma_2”,“rate_limiting_gates”:,“db_duration_s”:0.00405,“view_duration_s”:0.0,“duration_s”:0.03551}

I’m running the latest Gitlab on my own Debian server (15.2.1-ee). Everything else works except this. I’m going to look at the code for UploadsController, which looks like the place where it’s decided whether to serve the image file or scale it.

derekk19 · August 4, 2022, 1:00am

I’ve resolved my problem.

The issue was with my Apache site configuration. I was basically redirecting all traffic to port 8080 - the gitlab Rails app, but in fact I should have been redirecting everything to the gitlab workhorse.

I have configured the gitlab workhorse to use tcp port 8181 as its socket - rather than a named socket. There have been notes that apache will not redirect to a named socket (nginx does, but not apache)

I had the following (WRONG) rules in my apache vhost config

  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule .* http://127.0.0.1:8080%{REQUEST_URI} [P,QSA]
  RewriteCond %{REQUEST_URI} ^/uploads/.*
  RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE]

I think the second rule would never operate, because the first would catch everything (I’m not an expert on Apache, so I may be wrong there)

I changed my vhost rewrite rules to the following, which I copied from Files · master · GitLab.org / GitLab recipes · GitLab

  # Don't escape encoded characters in API requests. This makes the IDE work.
  # Don't escape encoded characters in -/refs/*/logs_tree/ or -/tree/, since 12.85,
  # these are pre-encoded, and excluding them causes double-encoding.
  RewriteCond %{REQUEST_URI} ^/api/v\d+/.* [OR]
  RewriteCond %{REQUEST_URI} .*-/branches/.* [OR]
  RewriteCond %{REQUEST_URI} .*-/refs/.*/logs_tree/.* [OR]
  RewriteCond %{REQUEST_URI} .*-/tree/.*
  RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE]

  # Forward all requests to gitlab-workhorse except existing files like error documents
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
  RewriteCond %{REQUEST_URI} ^/uploads/.*
  RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA]

My avatars are now rendered and scaled correctly. An extra log message is now included from gitlab-workhorse:

==> /var/log/gitlab/gitlab-workhorse/current <==
{"correlation_id":"01G9K5BZAVTSY7RQFQDXV8TP51","duration_s":0.022240461,"imageresizer.content_type":"image/png","imageresizer.original_filesize":102755,"imageresizer.status":"success","imageresizer.target_width":"64","level":"info","method":"GET","msg":"success","subsystem":"imageresizer","time":"2022-08-04T12:51:27+12:00","uri":"/uploads/-/system/project/avatar/19/Sailboat.png?width=64","written_bytes":3539}
{"content_type":"image/png","correlation_id":"01G9K5BZAVTSY7RQFQDXV8TP51","duration_ms":78,"host":"...","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:43382","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2022-08-04T12:51:27+12:00","ttfb_ms":78,"uri":"/uploads/-/system/project/avatar/19/Sailboat.png?width=64","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6 Safari/605.1.15","written_bytes":3539}

So it seems the secondary image-sizer requests were never getting sent to gitlab-workhorse (they were most likely being sent to port 8080 and then lost

SofianeLasri · August 4, 2022, 8:46am

Thanks a lot for your help, this fixed my issue! <3