How to "bypass" new 200 CI/CD environment variables limit?

Hi everyone,

Introduced in version 15.7, projects can now define a maximum of 200 variables (before, it was unlimited). In our case, we have slightly more than 200, and actually, we could see (or could have seen) ourselves grow to 1000+. Let me explain why, and please let me know if you have any advice on how to bypass that limit, or redesign our project in some way that would let us continue working.

At the core, our project is a web application (with dev and prod environment) that needs some environment variables (Cloud- and Firebase-related variables, Django secrets, …); it probably needs around 25, times 2 since we have 2 environments and the value of the variable is different based on the environment, so 50 in total. Initially, we built this web application for ourselves only, but we’re now offering it to other clients as well, but in a white-label way (i.e., their name, their logo, their domain, etc…). As we planned to offer this solution to dozens of clients with the same code base (except for some custom texts, colors, …), it didn’t make sense to us to create a different project for each new client as it would be very annoying to make fixes to all projects when needed. Hence we’ve decided to use the same project, but create a different environment for each client (actually, 2 environments for each client, dev + prod); deploying in dev would deploy to all dev environments, and deploying in prod would deploy to all prod environments. This means that our total number of variables is now ~2xCxV, where C is our number of clients, and V is our number of variables (it’s approximate as some variables are common to all clients). That way, we end up needing more than 200 variables quite quickly.

Now, is there some kind of design mistake on our side here? I think the benefits of having a single project are greater than having one project per client, but this limit now makes it impossible to work. Is there some way to “bypass” this limit, maybe by storing JSON with a key/value in it for each client instead of a simple string in each variable and reading the “actual” variables from those JSON variables?

I would love to get the insights of others into that. I was considering opening a new issue on GitLab, but if someone has an idea on how to deal with that, I’d be grateful.

Best,
Valentin

Hi,

Well 15.7 hasn’t been released yet, I’m currently on 15.6.2. That said, if a limit is set there isn’t going to be a way to override it, unless Gitlab give an Admin configuration option to allow people to choose to increase the limit or not. Reading the issue leads me to believe that the limit is hard-coded. But until we see the released 15.7 version it’s hard to say.

However, I don’t see how managing a separate project for each client is going to be a problem. If I was in your situation I would do this:

1. Manage and maintain the main project with it’s appropriate branches.
2. Fork main project from the master/main branch or whatever branch name you are using for production and this would become the project for the client.
3. When changes are made to project from point 1, just go to the fork which is the client and update the fork by pulling the changes.

That way you don’t end up with thousands of variables on a single project split across hundreds of branches. Also, by keeping your clients separate, you ensure each client has their own set of variables, that don’t potentially end up getting mixed up between clients. It would look pretty bad if client variables got mixed up. By separating the projects for each client, you ensure this doesn’t happen, and you also fall within the 200 variable limit. And updating is just a case of pulling the changes since the initial fork was made.

You might be able to do something with variables of type “file”. Like, make the value of the variable be lines of key=value pairs, and then have your pipeline job source the file as the first step. I don’t know if that would work for you, but it might be worth looking into.