CI/CD Minutes For GitLab SaaS Free Tier

We’re thinking through how to support our open source ecosystem better. This CI minute change will most impact those with Free accounts and private projects.

Please see my more detailed answer to some of the questions around open source projects and the CI min announcement in this comment: CI/CD Minutes For Free Tier

1 Like

@nuritzi Thanks for your reply.

But could you clarify? It has been announced that this limit applies to public projects as well, but can your reply be understood to mean that the time limit may not apply or may change for public projects?

This is an important point for some open source projects in deciding whether to migrate to another ci/cd service or keep using gitlab ci/cd. :thinking:

2 Likes

The distinction here about free CI for public project is critically important for Open Source projects like my own, Samba.

I realize that, because we do not pay, we are the product, not the customer. :slight_smile:

However, I think we do bring a value to the platform, and while I’m not asking (but of course would appreciate) that we get unlimited free CI forever, we need to know what the rules will be, and when they will apply from.

Significant financial, planning and infrastructure decisions rest on understanding clearly the plan here. Samba is a massive user of GitLab CI, using (at an estimate, because of the mentioned bug) in the order of 720,000 mins of CI pipeline a month. If this was all to fall to our own runners (which we use for some jobs, and as a backup when we are throttled) or we started paying for CI mins, it would cost us considerably!

Again, we have appreciated the generosity to date but need a clear direction from GitLab.com so we can make our own plans!

Furthermore in general, 400mins would not be enough for even a single Samba CI run (they take 1200min), so I can’t imagine many new contributors being willing to first pay GitLab to contribute to Samba.

It is not possible, so far, to run guest MRs against our CI runners, but this means even the newest contributors need to be brought into our group first-up. This will be an issue more broadly.

Therefore, what I ask is for a new timeframe to be set before which no CI on public projects will be charged, and between now and then a clearly communicated plan for what the charges will be and how it will work for projects like Samba.

It is not practical for us to plan based on the indefinite date a bug gets fixed. :frowning_face:

1 Like

I am wondering if no replies happened in the recent 8 days… Was all discussion over???

@nuritzi Is that really a bug? That has been a specification about CI minutes of GitLab CI since it has launched, as far as I understand. @mycroft @yogi might be some of specialists for these fields. (Unfortunately, except for all GitLab products and some small PoC, all my contributions to OSS are on GitHub\.com :bowing_woman: )

Will track Enforce CI minutes limits for all public projects (#243722) · Issues · GitLab.org / GitLab · GitLab progress! Thanks!

From a viewpoint of a UX/IA designer of Web UI, #243722 might harder than we think…

Thanks, @nuritzi - though can we get some clarity on when you’ll apply the limit to public projects please? The latest comments on Enforce CI minutes limits for all public projects (#243722) · Issues · GitLab.org / GitLab · GitLab seems to indicate the ‘fix’ is just to change a configuration value, so this could easily happen by/at the end of September.

A definitive date for changing the limits for public projects would be good, so that everyone can prepare - or failing that, a clear statement that this won’t apply to public projects at the end of September, and that there’ll be (say) a further announcement with at least 2 months notice before you do apply it to public projects.

@abartlet

I realize that, because we do not pay, we are the product, not the customer. :slight_smile:

Reference to The Social Dilemma? (I just watched it this weekend and thought it was pretty good. Lots to think about).

However, I think we do bring a value to the platform, and while I’m not asking (but of course would appreciate) that we get unlimited free CI forever, we need to know what the rules will be, and when they will apply from.

Yes, I agree. I added a comment about that in the bug issue and I (and others) have already been advocating for that internally. I used your comment as an example, so thanks for adding to this conversation! Having more real use cases to point to is helpful. (So thank you also to everyone else adding comments here about your open source projects!)

cc/ @tnir @joseph.heenan @berrange @supadhyaya

3 Likes

Thank you for the update, I’m following that issue too. I tried to get an idea of how much CI time libvirt is using by temporarily making my fork private, and running the CI jobs. I appeared to consume 450 minutes in a single CI run, and that’s with many jobs skipped as their cache was primed. So even if we were approved for the Open Source Program, I fear we would easily hit the larger 50,000 minute limit with our rate of contribution and current scale of testing (ignoring that we want to add even more CI). I’ve not checked their single-run CI time, but I expect the QEMU project will consume even more per-run and their plans call for more CI jobs too.

I completely understand why GitLab needs to put some limits on usage, as it isn’t sustainable for a business to provide free unlimited compute resources forever. So I’m expecting that at some point we’re going to have to at least part fund it ourselves with custom runners, even if we join the Open Source Program to get the extra CI minutes allowance.

If going down the custom runner route though, we would need to figure out an improved way to deal with the forking workflow wrt custom runners. We can’t expect contributors to setup custom runners, and AFAICT, runners from a project are inaccessible to CI jobs run in forks right now. We need a way for a project maintainer to explicitly trigger CI jobs on a new or updated merge request, such that the jobs run on the main project’s runners, not the fork’s runners. If GitLab’s integrated CI features can’t cope with that, then it looks like we’ll have to consider use of webhooks to trigger external CI instead (eg https://gitlab.com/ayufan/merge-requests-triggers), reporting back results as merge request comments or labels or approvals in some manner. It would be a shame to loose integration with GitLab CI infra though, as that’s one of my favourite features of GitLab.

1 Like

Just to add a link to the issue about inheriting runners by forks: https://gitlab.com/gitlab-org/gitlab/-/issues/15861 (and potential duplicate at https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3526).

1 Like

Thanks for adding additional details @berrange. I’ve noted your example.

If others have similar usage stories, where contributors would be affected by the changes, please add them to this thread and tag me!

@nuritzi

As another example, the GnuTLS project, on which Samba depends appears to do 50 pipelines a month, at around 90mins per pipeline. This puts it just on the cusp of the new limit.

I think it is really important that some analysis is done - the public rationale that most groups don’t use anywhere near 400mins per month is (we know learn) based on private projects only.

Also, for Samba, a significant part of the value proposition to using gitlab.com (compared with our previous patches on mailing list approach in particualr) is easy drive-by contributions. In those, we want new contributions to be able to do as much of the preparation work on their change, including making it pass an almost-full CI, before we first look at it, and before their first manual interaction.

Finally, if the gate is ‘genuine open source’ for additional minutes, please make participation in the GitLab Open Source program really easy, ideally one below the ‘Gold’ offering and instead just providing the CI resources. The current process (which Samba started) is to fill in web forms and then be invited to become a full GitLab customer with manual forms, signatures on formal contracts and an annual renewal. I don’t think this will scale to the number of open source projects that will want this!

1 Like

@abartlet

Finally, if the gate is ‘genuine open source’ for additional minutes, please make participation in the GitLab Open Source program really easy, ideally one below the ‘Gold’ offering and instead just providing the CI resources. The current process (which Samba started) is to fill in web forms and then be invited to become a full GitLab customer with manual forms, signatures on formal contracts and an annual renewal. I don’t think this will scale to the number of open source projects that will want this!

ACK on the easier application process. We’ll first be removing the merge request step of the process. We hope to automate the process more so in the near future.

There isn’t a definite timeline that I can provide for those changes at this time, but those interested can follow this epic for now: Epic: Improve OSS Program application experience for applicants and processing team.

For those wondering about the bug fix timeline… please see this new comment from a member of the Gitlab team about the rollout:

For those who are interested in this issue - I know there is some concern about WHEN we will fix this bug - and in an effort to be as transparent as possible I want to give everyone some re-assurance.

  • We understand that despite this being a bug, we owe you as our users advance notice of it changing as it could impact the future cost of your usage of GitLab.com .
  • We will provide at least 8 weeks notice (on our blog and this issue) before we begin accumulating shared runner minutes for public projects.

(Comment is copy and pasted from: Enforce CI minutes limits for all public projects (#243722) · Issues · GitLab.org / GitLab · GitLab)

If others have open source public projects with high/unknown CI minute usage on the Free tier, please continue to add your use case to this thread. Thank you!

It wasn’t so much about the MR, it was the paperwork of the contract that was a problem.

In particular the fact that the contract renews into a real (for Samba $80,000) invoice in 12 months time unless specific proactive steps are taken by the project, which in turn means we need it signed properly by our ‘fiscal sponsor’ (for Samba that is the Software Freedom Conservancy) and for other projects - the type of project you want to target, for whom GitLab.com might be their primary presence - a single individual has to take on that liability.

This might seem like a different issue, but the change in the CI/DC Minutes is the lever to making these projects need to be full legal/contractual customers, not just website users.

1 Like

ACK. I’ve updated the relevant epic with this input and created a separate issue about it in order to prompt internal discussion.

We do want to have some kind of renewal process though. Some projects no longer qualify for our program because of changes they make and we want to make sure that the program is not abused. If you have specific ideas for how to improve the renewal process, please let us know! You’ve provided great feedback so far. Thank you!

That is useful information, as the need for a contract with financial liability was not clearly described on the Open Source program landing/signup page. That unfortunately makes the program a non-starter from the POV of many of the open source projects I’m involved in. While most of the developers are doing work on behalf of their employer, the projects themselves are not formally represented by any organization. There is no way the individual contributors will be willing to sign a contract for the project which has any chance of turning into a financial liability. It would need to be much more like a traditional website Terms of Service agreement for it to be viable for non-sponsored open source projects.

The desire to have renewal process is understandable, but if not renewed, it needs to just revert back to the free tier, not turn into a paid tier with liability whoever signed up the open source project.

1 Like

@berrange

That is useful information, as the need for a contract with financial liability was not clearly described on the Open Source program landing/signup page.

ACK. I agree that it’s a good idea to add that to the Open Source program application page.

For now I’ve started an issue to remind myself of this feedback. Feel free to follow along if you want to provide additional input about that as I’m still not sure where to add it (maybe as an FAQ for now). I’m trying to remove the MR step asap, so that page will be changing soon anyway.

That unfortunately makes the program a non-starter from the POV of many of the open source projects I’m involved in. While most of the developers are doing work on behalf of their employer, the projects themselves are not formally represented by any organization.

Thanks for the feedback here. Are there any specific projects you can link to? If not, not worries, but having real use cases we can point to always helps!

The desire to have renewal process is understandable, but if not renewed, it needs to just revert back to the free tier, not turn into a paid tier with liability whoever signed up the open source project.

Rest assured, the GitLab for Open Source program does not turn into a paid tier if a member does not renew. Today, if someone does not renew their GitLab for Open Source program membership, they are downgraded to the Free tier. There is even a 14 day grace period after the expiration date. We do not collect payment information for the GitLab for Open Source program, unless you purchase a support add-on package.

Thank you for your response, so I wonder if the $80,000 potential cost for Samba that @abartlet refers to is related to support add-ons or something else ?

FYI, the biggest project I’m working on is libvirt, but also libosinfo & virt-viewer are related. I also work on QEMU who is a big gitlab CI user, but that’s in a slightly different situation as it is a member of the Software Freedom Conservancy non-profit.

@berrange

I wonder if the $80,000 potential cost for Samba that @abartlet refers to is related to support add-ons or something else ?

I took that to mean that Samba would have to sign on as a customer and pay that much if they wanted to keep their Gold-tier features in order to maintain the same workflows.

@abartlet, do you mind clarifying?

For the QEMU project our main concern would be moving from free to billed CI minutes. Our back of the envelope calculations think we are several orders of magnitude over the free minutes and the projects certainly doesn’t want to be presented with a massive bill because we forgot to get the renewal in on time. We do suspect we are one of your bigger CI minute users though but once the accounting is switched on we will have a better idea of how much we actually use.

@stsquad thanks for commenting with your use case example!

Our back of the envelope calculations think we are several orders of magnitude over the free minutes and the projects certainly doesn’t want to be presented with a massive bill because we forgot to get the renewal in on time.

Just to reiterate, we do not start charging people for the top tier and/or CI minutes if you forget to renew your GitLab for Open Source program membership.

Here are the terms around expirations:

What happens when my subscription is about to expire or has expired?

GitLab.com

Once the subscription has expired, the system will revert to the Free tier and users will no longer see paid features, however no data will be lost.

Self-managed GitLab

  • Starting 30 days before the subscription end date, GitLab will display a notice to all administrators informing them of the impending expiration.
  • On the day the subscription expires, nothing changes.
  • 14 days after the subscription has ended, GitLab will display a notice to all users informing them of the expiration, and pushing code and creation of issues and merge requests will be disabled. The system will not be functional anymore.

Source: Licensing and subscription FAQ | GitLab

Not happy about losing 1600 montly CI minutes, but is a sensible decision at face value (and only at face value).

A few thoughts to consider:

  • If “98.5% of free users use 400 CI/CD minutes or less per month”, then how many minutes will be saved from reducing the limit?
    • I expect the graph of “% of users” on the y-axis and “Monthly CI minutes usage” on the x-axis to be negatively-skewed. Considering the “fact” that 98.5% of users are under 400 minutes, then it would be an insignificant change to reduce the limit. (Please post the data, because it can be interpreted in different ways)
  • One of the main reason I initially chose GitLab over GitHub is that GitHub “only” has unlimited action/CI minutes for public projects and 500 MB and 2000 action/CI minutes every month for private projects, but GitLab had (when I signed up) unlimited CI minutes for public projects and 10 GB and 2000 CI minutes every month for private projects.
    • Before, GitLab was clearly better than GitHub in every way
    • Now, GitHub also has Windows, Linux, and MacOS runners. GitLab only has Linux and Beta Windows runners
    • Is it really worth limiting the 1.5% of people who use more than 400 minutes in exchange for fewer people signing up for GitLab?

If I were a new developer choosing between GitHub and GitLab, I’d honestly go for GitHub because of the better features and because it is more popular.

If the 400 monthly CI minutes is also applied to public projects by default, then I really have no reason to choose GitLab (because I am now more intimately aware of the features of GitHub and GitLab by using them both extensively, I would choose GitLab over GitHub, but new users would not have this experience).