After migration to new server, CI pipelines are not created

Hi!

We moved our Gitlab to a new server (Kubernetes with Helm chart gitlab-omnibus, awesome!)

Everything is mostly ok but CI pipelines are not triggered on git push, only for imported project. We created a new project on the new server and pipelines runs correctly.

  • Git hooks are well installed, we ran gitlab-rake gitlab:shell:setup and even gitlab-rake gitlab:shell:create_hooks.
    .git/hooks is well a symlink to gitlab-shell hooks.
    Example:
# ls -la /gitlab-data/git-data/repositories/group/project.git/hooks
/gitlab-data/git-data/repositories/group/project.git/hooks -> /opt/gitlab/embedded/service/gitlab-shell/hooks
  • We inspected the logs and we didn’t find any errors.

  • In sidekiq logs (/var/log/gitlab/sidekiq/current), we found some differences between the non-working project and the new one that works:

Non-working:

2018-04-25_13:11:30.44169 2018-04-25T13:11:30.441Z 519 TID-ow9zp6elo PostReceive JID-24e5d5c6029c717cefc097a6 INFO: start
2018-04-25_13:11:30.45833 2018-04-25T13:11:30.457Z 519 TID-ow9uotx0o UpdateMergeRequestsWorker JID-5da53c485abcb190df22a0bf INFO: start
2018-04-25_13:11:30.50572 2018-04-25T13:11:30.505Z 519 TID-ow9uotx0o UpdateMergeRequestsWorker JID-5da53c485abcb190df22a0bf INFO: done: 0.048 sec
2018-04-25_13:11:30.52115 2018-04-25T13:11:30.519Z 519 TID-ow9zp6h4s SystemHookPushWorker JID-fae30cd35b9ad8c9d1ec7009 INFO: start
2018-04-25_13:11:30.53525 2018-04-25T13:11:30.533Z 519 TID-ow9zp6h4s SystemHookPushWorker JID-fae30cd35b9ad8c9d1ec7009 INFO: done: 0.013 sec
2018-04-25_13:11:30.54189 2018-04-25T13:11:30.541Z 519 TID-ow9zp6g58 ProjectCacheWorker JID-fce0480bda1af3e90c6ac8d1 INFO: start
2018-04-25_13:11:30.55814 2018-04-25T13:11:30.558Z 519 TID-ow9zp6elo PostReceive JID-24e5d5c6029c717cefc097a6 INFO: done: 0.116 sec
2018-04-25_13:11:30.57835 2018-04-25T13:11:30.577Z 519 TID-ow9zp6g58 ProjectCacheWorker JID-fce0480bda1af3e90c6ac8d1 INFO: done: 0.036 sec

Working project:

2018-04-25_13:12:06.38228 2018-04-25T13:12:06.381Z 519 TID-ow9ud1afg PostReceive JID-f78189777bbc9bda133a787f INFO: start
2018-04-25_13:12:06.40099 2018-04-25T13:12:06.400Z 519 TID-ow9zp6fjk UpdateMergeRequestsWorker JID-7f5cb607dde639575de987a4 INFO: start
2018-04-25_13:12:06.44993 2018-04-25T13:12:06.449Z 519 TID-ow9zp6fjk UpdateMergeRequestsWorker JID-7f5cb607dde639575de987a4 INFO: done: 0.049 sec
2018-04-25_13:12:06.53964 2018-04-25T13:12:06.539Z 519 TID-ow9uotvtw BuildHooksWorker JID-dd235a1dfbdbd581a58734c6 INFO: start
2018-04-25_13:12:06.60436 2018-04-25T13:12:06.603Z 519 TID-ow9zp6h4s BuildQueueWorker JID-e3f85b1addeb9cc5eb0d85d2 INFO: start
2018-04-25_13:12:06.60607 2018-04-25T13:12:06.605Z 519 TID-ow9uotx0o PipelineUpdateWorker JID-4ad68b134221060d92d3ac57 INFO: start
2018-04-25_13:12:06.63891 2018-04-25T13:12:06.610Z 519 TID-ow9zp6voo StageUpdateWorker JID-e1161baf2c51b2644ebe1cfa INFO: start
2018-04-25_13:12:06.64857 2018-04-25T13:12:06.648Z 519 TID-ow9zp6g58 ExpireJobCacheWorker JID-757bc89496016a8dfa2d2be8 INFO: start
2018-04-25_13:12:06.65816 2018-04-25T13:12:06.657Z 519 TID-ow9uotvtw BuildHooksWorker JID-dd235a1dfbdbd581a58734c6 INFO: done: 0.118 sec
2018-04-25_13:12:06.68581 2018-04-25T13:12:06.685Z 519 TID-ow9zp6elo PipelineHooksWorker JID-8aaa79e1d48261adc8d6519a INFO: start
2018-04-25_13:12:06.69576 2018-04-25T13:12:06.695Z 519 TID-ow9uotxcw ExpirePipelineCacheWorker JID-4063b69f15bd638eb7fcf08b INFO: start
2018-04-25_13:12:06.72637 2018-04-25T13:12:06.725Z 519 TID-ow9zp6voo StageUpdateWorker JID-e1161baf2c51b2644ebe1cfa INFO: done: 0.115 sec
2018-04-25_13:12:06.73021 2018-04-25T13:12:06.729Z 519 TID-ow9uotx0o PipelineUpdateWorker JID-4ad68b134221060d92d3ac57 INFO: done: 0.124 sec
2018-04-25_13:12:06.74155 2018-04-25T13:12:06.740Z 519 TID-ow9zp6g58 ExpireJobCacheWorker JID-757bc89496016a8dfa2d2be8 INFO: done: 0.093 sec
2018-04-25_13:12:06.74645 2018-04-25T13:12:06.746Z 519 TID-ow9zp6f90 SystemHookPushWorker JID-ff3d6e2be9156b44f461e28f INFO: start
2018-04-25_13:12:06.78229 2018-04-25T13:12:06.781Z 519 TID-ow9zp6f90 SystemHookPushWorker JID-ff3d6e2be9156b44f461e28f INFO: done: 0.035 sec
2018-04-25_13:12:06.80215 2018-04-25T13:12:06.802Z 519 TID-ow9zp6v7g ProjectCacheWorker JID-2a34c10402710f1b833eb835 INFO: start
2018-04-25_13:12:06.84467 2018-04-25T13:12:06.844Z 519 TID-ow9uotxcw ExpirePipelineCacheWorker JID-4063b69f15bd638eb7fcf08b INFO: done: 0.149 sec
2018-04-25_13:12:06.87359 2018-04-25T13:12:06.873Z 519 TID-ow9zp6elo PipelineHooksWorker JID-8aaa79e1d48261adc8d6519a INFO: done: 0.187 sec
2018-04-25_13:12:06.88320 2018-04-25T13:12:06.882Z 519 TID-ow9ud1afg PostReceive JID-f78189777bbc9bda133a787f INFO: done: 0.501 sec
2018-04-25_13:12:06.89225 2018-04-25T13:12:06.891Z 519 TID-ow9zp6h4s BuildQueueWorker JID-e3f85b1addeb9cc5eb0d85d2 INFO: done: 0.288 sec
2018-04-25_13:12:06.91954 2018-04-25T13:12:06.918Z 519 TID-ow9zp6v7g ProjectCacheWorker JID-2a34c10402710f1b833eb835 INFO: done: 0.117 sec
2018-04-25_13:12:07.01992 2018-04-25T13:12:07.019Z 519 TID-ow9u6yuz0 BuildHooksWorker JID-79a406d3ed0e9db4f134b877 INFO: start
2018-04-25_13:12:07.02384 2018-04-25T13:12:07.023Z 519 TID-ow9zp6dx8 PipelineUpdateWorker JID-75b6a01b8fdd666522306a86 INFO: start
2018-04-25_13:12:07.03139 2018-04-25T13:12:07.030Z 519 TID-ow9uotw64 ExpireJobCacheWorker JID-4807f2b77261ce4e56feeb7d INFO: start
2018-04-25_13:12:07.04019 2018-04-25T13:12:07.039Z 519 TID-ow9uotvns StageUpdateWorker JID-3e0a70f9721c48d3c3372771 INFO: start
2018-04-25_13:12:07.11059 2018-04-25T13:12:07.109Z 519 TID-ow9uotw64 ExpireJobCacheWorker JID-4807f2b77261ce4e56feeb7d INFO: done: 0.079 sec
2018-04-25_13:12:07.11620 2018-04-25T13:12:07.116Z 519 TID-ow9uotwic PipelineHooksWorker JID-6897c1422060a06791bf6fbc INFO: start
2018-04-25_13:12:07.11855 2018-04-25T13:12:07.118Z 519 TID-ow9uotwc8 ExpirePipelineCacheWorker JID-2a3dbece00f32d8e04c83718 INFO: start
2018-04-25_13:12:07.12014 2018-04-25T13:12:07.120Z 519 TID-ow9zp6evo PipelineMetricsWorker JID-6c3786559f3be093820db66b INFO: start
2018-04-25_13:12:07.14665 2018-04-25T13:12:07.145Z 519 TID-ow9uotvns StageUpdateWorker JID-3e0a70f9721c48d3c3372771 INFO: done: 0.106 sec
2018-04-25_13:12:07.15120 2018-04-25T13:12:07.150Z 519 TID-ow9zp6dx8 PipelineUpdateWorker JID-75b6a01b8fdd666522306a86 INFO: done: 0.127 sec
2018-04-25_13:12:07.17112 2018-04-25T13:12:07.170Z 519 TID-ow9zp6evo PipelineMetricsWorker JID-6c3786559f3be093820db66b INFO: done: 0.05 sec
2018-04-25_13:12:07.22799 2018-04-25T13:12:07.227Z 519 TID-ow9u6yuz0 BuildHooksWorker JID-79a406d3ed0e9db4f134b877 INFO: done: 0.208 sec
2018-04-25_13:12:07.24011 2018-04-25T13:12:07.239Z 519 TID-ow9uotwc8 ExpirePipelineCacheWorker JID-2a3dbece00f32d8e04c83718 INFO: done: 0.122 sec
2018-04-25_13:12:07.24773 2018-04-25T13:12:07.247Z 519 TID-ow9uotwic PipelineHooksWorker JID-6897c1422060a06791bf6fbc INFO: done: 0.131 sec

As you can see, logs are different, the non-working push does not produce build related tasks.

  • On the non-working project, we created a new branch for testing that pipeline run. (Branch have a good .gitlab-ci.yml). This new branch is well browsable on the “Repository” interface but it is not present in the select when we try to manually run a job on this branch.

  • If we run manually a pipeline with the interface button “Run pipeline” for other existing branch, it works.

  • gitlab-rake gitlab:check show no error or warning

  • Version: 10.7.0
    Old projects have been imported from 10.6.4 to a 10.6.4 and we have upgraded after the migration.

I continue to investigate but help is welcome :wink:

Thanks!

Okay… I think I understand what’s going on…

The original buggy branch was created before I executed gitlab:shell:create_hooks. So the branch has been stored in git repo but Gitlab didn’t receive the info.
I deleted this branch and I pushed again and… it works !
:kissing_closed_eyes:

Thanks to Gitlab team for this awesome platform :heart: This issue makes me to search in the codebase, it was really interesting.