CI/CD pipelines "engine" source code/repo?

GitLab CI/CD
1

Problem to solve

*Hello, everyone; I’ve been working a lot with GitLab’s CI/CD pipelines, and I’m very curious about how it works under the hood, so I was wondering if:

    • Is the “Pipelines” engine open source?
    • If so, could someone help me find the repo? I’m especially interested in the parser.

I found a few options, but none of them seem to be the actual “engine” (maybe that’s not even how it works), for example, this repo:

An example of what I would like to learn:
How does the code that interprets expand and !reference looks like?

I would really appreciate any help,
Many thanks. :slight_smile:

2

The GitLab CI/CD architecture consists of server and runner components, which live in

  • gitlab-org/gitlab - server, in the Ruby on Rails monolith
  • gitlab-org/gitlab-runner - runner, job executor, Go

There are more components in the GitLab architecture, which might be helpful to understand the bigger picture.

My strategy to figure out the config parser/compiler portions is to use the code search on GitLab.com and see which Ruby files and directories include said terms. From git history/blame, I can navigate to git commits and MRs and learn how the feature was implemented.

which brings up the test spec in spec/lib/gitlab/ci/config_spec.rb · master · GitLab.org / GitLab · GitLab and the indicator that this would live in lib/gitlab/ci/config somewhere.

Another idea is to use the code search help, and refine the query to only search for ruby code. !reference lang:ruby !reference lang:ruby · Search · GitLab leading to lib/gitlab/ci/config/yaml/tags/reference.rb · master · GitLab.org / GitLab · GitLab

It is complex code though, so I would recommend using an IDE and understand the object inheritance. You can also use AI and GitLab Duo to explain code, as part of the community contributions project in GitLab Community / Community Members / Duo Access · GitLab and also chime into the GitLab Discord server to ask for help or pairing. Contribute to GitLab | GitLab | GitLab

Other learning methods

Sometimes, I also navigate from the “tier badge / history” list in the docs into the linked issues to understand how the feature was implemented in linked MRs.

  1. expand - CI/CD YAML syntax reference | GitLab
  2. !reference - Optimize GitLab CI/CD configuration files | GitLab

Another strategy can be the GitLab development guide to learn more. Contribute to development | GitLab

3

Thank you so much, @dnsmichi. I’ll definitely have a look; that’s exactly what I was looking for. :grinning:

1 Like