Question about webhooks


I have a homelab on-prem, in which I run a Foreman server (srv-foreman) and a GitLab server (srv-gitlab); both of these are on the same VLAN.

I have a Puppet control repo and a similar setup for an ansible codebase; my goal is to have commits to any branch in either repo trigger a webhook I have listening on port 8170 on srv-foreman. (I wrote the webhook myself; it uses unathenticated/unencrypted HTTP). The webhook triggers r10k, which deploys the control repo, the ansible codebase, and also triggers Foreman to refresh its class and role lists after deployment. (It might be significant that both repos are configured to hit the same endpoint. The ansible codebase is a relatively recent addition.)

I can use the “test” button in both the control repo and ansible codebase projects, which triggers the hook successfully.

But when I do the ordinary commit/push workflow, I cannot find any evidence that the webhook is actually firing. (I’m running tcpdump on port 8170 on srv-foreman, and I don’t see traffic - of course I do see traffic when hitting the hook port with curl, or with the “test” buttons on the projects"). I have both the network range and domain specified in the “allow” lists for sending webhooks on the gitlab server.

All of this worked reliably until I added the ansible repo to the r10k mechanism at the end of March.

While I could use another mechanism to trigger the hook (and have been for a while), I would really like to understand how to run this down and troubleshoot it in gitlab itself. I’ve been running gitlab-ctl tail, and I see messages like this:

{"severity":"INFO","time":"2021-07-02T11:57:41.010Z","class":"PostReceive","args":["project-35","key-6","d055e099fb6ba800ee340eda7ced3bdd8f0eff34 6f5a43e191137a551da1dd3c17c0bb40cde7b18c refs/heads/production\n","{}"],"retry":3,"queue":"post_receive","version":0,"jid":"c1afa553a82e86cedeb9f111","created_at":"2021-07-02T11:57:40.045Z","meta.user":"mjackson","meta.project":"puppet/control-repo","meta.root_namespace":"puppet","meta.caller_id":"POST /api/:version/internal/post_receive","meta.remote_ip":"","meta.feature_category":"source_code_management","meta.client_id":"user/2","correlation_id":"01F9KH9RS481PCCYFA938XV12X","idempotency_key":"resque:gitlab:duplicate:post_receive:636ca78dffaf22414694cec1effc1b39758301554d38babf1ba4ce6418dbde0c","enqueued_at":"2021-07-02T11:57:40.053Z","job_size_bytes":133,"pid":370857,"message":"PostReceive JID-c1afa553a82e86cedeb9f111: done: 0.955267 sec","job_status":"done","scheduling_latency_s":0.001723,"gitaly_calls":8,"gitaly_duration_s":0.056902,"rugged_calls":2,"rugged_duration_s":0.001572,"redis_calls":48,"redis_duration_s":0.033529,"redis_read_bytes":3064,"redis_write_bytes":8276,"redis_cache_calls":31,"redis_cache_duration_s":0.017538,"redis_cache_read_bytes":2775,"redis_cache_write_bytes":3111,"redis_queues_calls":6,"redis_queues_duration_s":0.003735,"redis_queues_read_bytes":234,"redis_queues_write_bytes":4611,"redis_shared_state_calls":11,"redis_shared_state_duration_s":0.012256,"redis_shared_state_read_bytes":55,"redis_shared_state_write_bytes":554,"db_count":30,"db_write_count":5,"db_cached_count":1,"cpu_s":0.528397,"mem_objects":178618,"mem_bytes":42519563,"mem_mallocs":181105,"mem_total_bytes":49664283,"duration_s":0.955267,"completed_at":"2021-07-02T11:57:41.010Z","db_duration_s":0.347668}

But to be honest I’m not entirely sure what to look at here to pursue this further.

I am currently running the most recent gitlab-ce Omnibus package (14.0.2 as of this writing) on CentOS-8 Stream. This problem seems to have been with me for several versions of 13 though; it is definitely not new to 14.

Thanks for any help or advice!