I just created a project in GitLab and discovered that these days you automatically get a pipeline with auto devops. Which sounds nice on paper, but I wasn’t too happy with this. You see, I’m on a bronze plan so I only have a limited amount of build minutes. And the auto devops pipeline is quite a big one. I created three branches before discovering this (master, develop, and one feature branch) and by this time I already burned through 131 of my 2000 minutes. Especially because two tasks in one of the pipelines were just idling (they were running, but nothing was happening, I had to cancel them).
In my opinion, auto devops should be disabled by default, or at the very least you should be prompted on project creation whether you want to use it or not. Had I not noticed that this pipeline was running, I could have easily burned through all my minutes in a short amount of time.
Okay, so this is interesting. Just created another project, and this one had auto devops disabled by default. Does Gitlab remember this choice? Or does auto devops get enabled automatically when certain conditions are met?
@rdcl We’re currently in the process of incrementally enabling the Auto DevOps feature instance-wide on GitLab.com to see how things go and we’ve been discussing it in this issue if you’d like to take a look. Overall though, it should remember your choice and AutoDevOps will be disabled automatically if any of the pipelines fail initially.
Can you share your GitLab.com username? Feel free to private message it to me if you feel more comfortable that way.
My new project have Auto DevOps turned on by default, then switched off after pipeline failure.
I dont need Auto DevOps anyway, how can I disable it for my future project?
@diepdtn You can manually disable it for any of your projects by unchecking Default to Auto DevOps pipeline in the Settings -> CI / CD -> Auto DevOps section of any project.
I completely agree that it was a terrible decision to enable AutoDevOps by default! We had many projects where it caused a lot of problems, and didn’t disable as it should.
It doesn’t help that the release announcement refers to the documentation saying it tells how to “disable it […] for an entire instance”, which it does not! We have figured out that the setting described in the note about enabling it instance wide, also allowed us to turn it off instance wide, but I still lack a description of how to do so (of it can be done) from gitlab.rb. I soon have to set up another instance and it would be really nice not having to do this after seting it up (gitlab.rb will be generated by our chef setup).
Gitlab, did you toggle Auto DevOps to “on” for preexisting projects? Or is it possible that some database migration script did so unintentionally?
I got an email message today about how “Pipeline has failed for master”, and it made me think someone had hacked into my gitlab repo.
This is a project I have been pushing to for years, and I never intentionally added any pipeline, and I had never heard of this before today.
I wish to echo other users here who are saying this was a bad decision, or at least a botched roll-out of a new feature. Sudden, surprising behavior from a private repo that I have not reconfigured in over a year is a bad user experience.
Hi Grove, GitLab PM here. Sorry to hear this caused you a lot of problems. Would you be able to elaborate on the nature of the problems it caused? We really thought long and hard about doing this and in the end we saw more positives than negatives. Would like to understand more about your situation. Thanks.
Hi @pestophagous. Yes, it was indeed enabled by default as part of our 11.3 release. It was enabled only for projects that did not have a gitlab-ci.yml file present on the repo. We tried to minimize confusion by sending you detailed emails about enabling/disabling but will work to make it clearer.
Unfortunately I don’t know much about what happened. I read the release announcement for 11.3 and thought that it sounded safe, so I upgraded (following the procedure for doing it without downtime) in the afternoon of 24 September, and then I went on vacation, and when I came back 3 October, I understood from the colleagues that autodevops had caused several problems and that they had had to spend a lot of time finding out how to disable it. But as we have a .gitlab-ci.yml file in a lot of our projects (I know that’s hard to detect but it’s a good sign that this “feature” wouldn’t do any good) I guess it mostly concerned the other projects, most of which is just git repositories not containing code (among other things there’s some high level documentation of the department and how things work here), so whatever gitlab tried to do didn’t make sense.
It would please me more if you could address the issue of turning it of automatically when installing a new instance (it won’t benefit on the new instance I have been tasked with setting up).
Thanks, @danielgruesso . I did not receive any emails about this. Prior to the surprising email that “pipeline has failed for master”, the only announcement I have from GitLab via email is on Sept 20 with the subject: “Important Changes to your GitLab.com Account”. That email only pertains to retiring the early adopter plan.
Either the detailed emails about Auto Devops did not get successfully mailed as planned, or perhaps they were filtered out by a spam filter?
In any case, for the sake of your users that get huge volumes of email, I would suggest the following:
For announcing changes like this, (in addition to email), consider using server-side hooks on the git server to echo messages back to the git client.
I just thought of an additional way this hurt. The extra CPU load it caused! Looking at some of jobs that failed, some of them took almost a minute, even though whatever gitlab tried didn’t make sense.
With a lot of forks of the some of the projects (I think some of the projects not containing code might the the most forked) and gitlab having to try this for every fork, it wasted quite a few CPU cycles on this, and as our gitlab server is heavily loaded (that’s one of the reasons why I have to set up a another one, security being another) this wasn’t good.
And by chance I just stumbled upon a fork where autodevops apparently wasn’t disabled after the first failure (we have 4 failed jobs for the same commit with the same pipeline id, but different job ids, over a period of 52 minutes). You can have screenshots of the job list and job outputs, but they are not really interesting, all the failed jobs have “build” in both the “stage” and “name” columns, while the “test” stage jobs were skipped, and except for some git differences in the job outputs they are all alike, ending with:
Checking out 56f382d7 as master…
Skipping Git submodules setup
$ # Auto DevOps variables and functions # collapsed multi-line command
$ setup_docker
$ build
Logging to GitLab Container Registry with CI credentials…
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
Building Heroku-based application using gliderlabs/herokuish docker image…
-----> Unable to select a buildpack
ERROR: Job failed: exit status 1
I maintain the belief that it was a terrible idea to enable this be default, and the belief that only being able to disable it for all projects from the webinterface is terrible lack of configurability.
I’m going to add to what everybody else has already said… Auto Devops shouldn’t be getting enabled without the user requesting it. We had existing projects which auto-deply after a successful merqe request but were all of a sudden not deploying anymore. We realised they were being held up because Auto Devops had been switched on and was blocking the merqe request from completing.
This reflects really poorly on Gitlab. Please strongly consider reversing your decision.
In order to test your beta service, you will force your happy (paying) customers to unknowingly have their private repos practically stolen and processed on some public runner. Let’s see how many will be happy once they found out what you are doing. This is really sad.
HI Daniel. The issue I’m having is that we have hundreds of repos on gitlab.com and on some of the repos these autodevops jobs are now running when we push any commits and all failing. I have found documentation on how to turn this off globally (https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#auto-devops) but these directions do not seem to apply to gitlab.com. If they do, then I’ve got other issues because I have no “admin area” little wrench icon in my toolbar. If we’re going to have to go into the settings of every one of these hundreds of repos, we’re in trouble.
We had existing projects which auto-deply after a successful merqe request but were all of a sudden not deploying anymore. We realised they were being held up because Auto Devops had been switched on and was blocking the merqe request from completing.
@jspurling Interesting. I’d like to understand your usage better. Were you automating deployments using something outside .gitlab-ci.yml? If so, is the problem that we were blocking the MR from being merged at all, because of the failed pipeline?
Thanks for the feedback @grove. What sort of your problems did your team come across?
We try to minimize time spent on projects that are not compatible by automatically disabling Auto DevOps on projects where its pipeline fails the first time.