Gitlab and building our Monorepo

Hello,

We are a customer who have recently moved to a self hosted gitlab instance and we keep all of our code in a monorepo. But there are a few issues we are running into and I am not sure if they are problems that Gitlab can solve.

So our repository structure is as follows:

.gitlab-ci.yml
  /apps
    /app1
      .gitlab-ci.yml
    /app2
      .gitlab-ci.yml
  /shared-libs
    .gitlab-ci.yml

So as you can see, we have a gitlab ci file at the root of the repository - this includes each app and the shared libs gitlab ci file.

The way in which expect my builds to run is as follows:

  • If a change occurs in shared-libs, it is built and then all apps are built.
  • If a change in only app1, only it is built.
  • If a change occurs in both shared-libs and app1, build the libs and then the apps.

This is fairly simple to achieve using the rules:, so each app’s job config has something like:

- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
      changes:
        - ".gitlab-ci.yml"
        - "apps/app1/**/*"
        - "shared-libs/**/*"

and the shared-libs has something like:

- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
      changes:
        - ".gitlab-ci.yml"
        - "shared-libs/**/*"

Ok, this is great, everything works fine! Well, not quite… The problem occurs when a change goes in to shared-libs which fails. Then a subsequent change goes in to one of the apps, that change has no idea that shared-libs had previously failed, and so will run to its conclusion. In our world we need those shared libraries to have previously passed to continue. So we need to either fail the build fairly quickly, or re-run the shared-libs build (so it fails and stops the pipeline)

I have had a look through the documentation, and I can’t seem to see if this would be possible?

We can just run the build for the shared-libs on every change, which would somewhat solve this problem - but it is expensive to do so, so if we can avoid it, I would like to.

I have also looked through the Gitlab API to see if it would be possible to hand roll this behaviour myself, but I would need to be able to get the status of the latest run of a specific job in the default branch, but it seems the API does not provide that - you can only get jobs by ID or Pipelines, not by name.

Any thoughts or help would be appreciated!

Thanks,

1 Like

hello!

unfortunately, i cannot provide an answer to this problem as i am encountering similar issues.

does gitlab have a plan for better handling monorepos?

There are a few issues up about this, such as this one if you comment and upvote the issues that are relevant to you, that will make them more likely to be resolved.

In the situation mentioned in the OP, I would suggest using the package registry. The suggestion would be that when the shared libs are built, they can be uploaded to the repo’s own package registry, then every pipeline can fetch either the latest build on a protected branch (such as main or develop), or use the latest shared lib that was successful in the current branch.

In fact, there’s a docs page specifically about using the registry for monorepos already.

HTH

thank you for your quick reply.

The container registry has been very helpful. I am using 7 different docker containers to help with building the different components of my product.

I also checked out the issue you linked and left a comment on it.

1 Like