This job is stuck because the project doesn't have any runners online assigned to it

This question has been asked a lot here and elsewhere. I’ve read through all the threads. I’ve tried a dozen different solutions, hacks and restarted with a new server 3 times. I’ve come to the conclusion that this is one of two things A) run with tags or B) run without tags. I was unable to find any documentation that explains what tags are or how to use them effectively. They seem to not be required anyway so I didn’t worry too much about that. Moving on to B, I set out to configure my runner to run without tags. As it turned out that’s actually default setting.

my .gitlab-cli.yml file

stages:
   - test
testWeb:
   stage: test
   script:
      - cd web
      - vue-cli-service test:unit

Note: not tags: listed.

I’ve installed the runner on an Ubuntu 18.04 VPS on DO following the instructions found in the official docs here.

image

I didn’t run into any issues and the runner appeared under “Runners activated for this project” automatically. I thought, “Bingo! It connected.” But Pipelines didn’t run. See error above.

When things don’t work and RTFM fails, check the logs. Right? Okay. I’m not positive of where the log file is for the runner. I checked /var/log, for anything more relevant to gitlab-runner or gitlab but found nothing I wouldn’t otherwise expect to see on a stock 18.04. The only reference in the docs that I could find says they’re in the general /var/log/syslog. The generic syslog did, however, contain a reference. Okay…

I searched online for anything related to the two errors I found in the logs. From what I was reading, I believed that these errors were related to metrics and should not prevent the pipeline from running. I also found some posts relating this error to the same “tags” issue above with the same two solutions, “use tags” or “don’t use tags”. In an attempt to simplify testing I stuck with the “don’t use tags” approach. Perhaps this was in error, but not being able to dig into what tags do or how they work was proving to be more frustrating than trying to figure out why no tags wasn’t working. A less complex use case should be, theoretically, easier to debug.

The repository I’m using is simply a testing project spun up specifically for the purpose of experimenting, learning and evaluating how pipelines might be able to help our company. It contains nothing more than a default Vue JS project with default tests and it works out of the box.

My two questions are, am I even remotely close to getting this to work? What am I missing here? Is there anywhere that I can get more clarity on what’s wrong other than, “This job is stuck because the project doesn’t have any runners online assigned to it.”

Update:

While I was working on the above write up, retracing my steps to confirm I had followed the instructions correctly and collecting the screenshots - I discovered this (see image below). Completely by chance, I hovered over what appears to be a disabled warning icon. a) it’s grey. b) it doesn’t do anything if you click on it. It seems to suggest that should there be a warning, the icon would become enabled, change colors, and then allow for a click action. If it wasn’t disabled or if it were otherwise important it would have a primary color attached to it. Red, being consistent with the rest of the design language on your site. For example, see the red “pause” and “Remove Runner” buttons to the immediate right.

image

This happy accident uncovered a new clue, “New runner. Has not connected yet”. Searching this error led me to several recommendations including

“…delete the config.toml in my home directory” - source

Which appears to be a an issue mostly related to Windows, but I checked my home directory on my Linux machine anyway. Not there.

I looked into this issue,
sudo chown -R gitlab-runner:gitlab-runner /home/gitlab-runner - source

Which suggests the existence of and therefore need to create a gitlab-runner user. I can see how that would be an excellent approach for a final setup, but A) the documentation makes no reference to creating such a user or applying said permissions and B) I’m just experimenting and plan on tearing down this droplet shortly. Anyway, there is no home folder for said user ergo, not config file in said folder. The logs don’t seem to suggest any issue loading the config file (Which it would if it had, right?). Also, the runner was registered and is clearly reporting in the web interface, as you can see. I think it’s fair to assume that the runner on my server used the api key I gave it during the one-line install step or how else would it show up in the interface? So, if it can connect to register correctly why can’t it also connect-connect correctly without logging an error either time? The error and this solution seems counterindicative to available information, specifically the fact that it did connect. On top of that, the official documentation makes no reference to this user or directory or file.

The documentation I’m referring to is linked to in step 1 on the page where you set this all up. That page has a link that goes to the documentation on manually installing a runner on Linux.

I also found this solution,
“Simple answer: RESTART the server!” - source
Which, tbh, seems like an unusual and unnecessary step for a Linux environment. I tried it anyway. Checking the status of the runner on reboot to find it was running. Alas, it also had no effect on removing the error in the title of this post.

I found this video where I learned the command gitlab-runner --debug run - source

Fascinating tool for debugging, alas I found nothing to indicate an error on first run.

image

However, new tool for debugging! Let’s play! Let’s run the pipeline!! Let’s see what happens!

Here’s where things get bizarre. It worked.

The pipeline worked. It worked in the sense that it failed exactly where I would expect it to.

I don’t even know what to do with this or where to take this next? I’ve already been at this for 2.5 days. I would like to automate our testing, build and deploy cycles, but if this is the solution I’ll stick with the problem.

This started as a technical support question, but I don’t even know what question to ask right now. It’s Friday. I’m tired. I’m frustrated. I’ve got zer0 gas left for what is essentially a product trial.

I hope you have had a nice weekend and have some new fresh hope and gas :). So you got it runner, but only in debug mode. Can you add your config.toml? It seems like the listen address isn’t in there, not really sure how much that matters, but I think it is needed for communication between gitlab and the runner.
Also, do you have your systemd service file (assuming systemd) or otherwise your init file?

Thanks @bartj On the upside, it was a great weekend; a long weekend even.

I’m not 100% on where the correct config.toml file is. I may be posting the wrong one. As I recall it could be in a couple of different places depending on which tutorial one might follow or which method one attempts. As I followed the docs that seem to install via a package manager I assume the file will be in the usual place for my distro (Ubuntu). That being /etc/gitlab-runner/config.toml where I do indeed find such a file.

The contents of that file are (if this is the correct file):

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

I notice there’s no runner configured in this file. I followed the runner registration instructions found here.

For executor on step 6 I selected shell.

These instructions are labelled as, “To register a GitLab Runner under Linux:” So perhaps this is the wrong file? Is there another place I should possibly look? Is there a way to tell which config.toml file is being loaded by gitlab-runner? I seem to recall there being a command to show which one was loaded.

At any rate, I thought I would walk through the register runner steps again. This time there appears to be success at adding the runner. The config.toml file now shows the new runner.

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

    [[runners]]
      name = "description goes here"
      url = "https://gitlab.com/"
      token = "_ABC123ETC"
      executor = "shell"
      [runners.custom_build_dir]
      [runners.cache]
        [runners.cache.s3]
        [runners.cache.gcs]
        [runners.cache.azure]

image

It appears that I can see the one that I thought I had registered the first time around and the new one that I just registered. This new one is green, as expected. I removed the first one and ran the runner.

The runner appears to work, however it doesn’t get as far as the last one did when I ran it in debug mode. I am now seeing an error about shell profile loading.

When I visit the URL in the error message it mentions that, “A common failure is when you have a .bash_logout that tries to clear the console.

I notice I now have a /home/gitlab-runner folder with said .bash_logout file that does indeed try to clear the console.

When I remove those lines in that file the runner works. Or, to be more precise, the runner fails where I would expect it to which is great. The lingering question in my mind that the documentation doesn’t answer on this point is, what are the ramifications of removing those lines? Am I just kicking a problem further down the field? The documentation doesn’t actually mention an appropriate fix, only that this is a problem.

Overall, I hope I have sufficiently documented the experience so others (future me?) might be able to find solutions for all the issues involved. I’m really hoping someone responsible for the documentation can take a hint.

I’m not sure why the first runner wasn’t added correctly and yet registered on the site. Does that need to be a bug report?

I’m not sure why a critical error message was buried so deeply in the design of the page; hover over an apparently inactive 18px icon? Really?

I’m not sure why the documentation is so convoluted and spread across 5 different pages with various deep linked anchors and supplemental reading and support off-site required.

I’m not sure why the CI/CD settings are designed the way they are. I’m not sure why defaults don’t get a working version out of the box. eg. in every example, how-to blog post, and video one of the first steps is always a note to disable shared runners. What are they and why are they enabled by default?

I’m not sure why the terminology is different in the documentation around the gitlab ci coordinator URL, the gitlab-runner register steps and the CI/CD config page.

I’m not sure why the runner logs to syslog with only limited logging. I’m not sure why the --debug flag is not mentioned earlier and more often in the docs because those logs are much more useful and not swamped by general syslog information. I’m not sure why there isn’t a /var/log/gitlab-runner.log or /var/log/gitlab-runner/error.log. I would love it if those existed.

Why so many references to and importance placed on tags? And yet there is very little documentation on them, how to use them effectively, examples or cookbook on that point? They appear to be important and useful, but the documentation doesn’t clarify why or how.

I don’t understand the decision to use both yml and toml formats.

The “Runners Activated for this project” header is misleading when the runners are not actually activated.

I wanted to use a simple Linux machine as opposed to a full Docker container to remove layers of complexity and simplify testing so I could evaluate the tools and experience more directly. I also assumed the docs for such an experience would be simpler. I’m not sure if the documentation for that workflow are any easier to follow, but they seem equally convoluted. I did end up having to read carefully which sections of the docs were related to my use case and which were related to Docker users. I’m not sure why they’re presented in parallel like that, but it was a distraction and complicated the workflow.

I would really like whoever is in charge over at GitLab to really change the value they place on their documentation. Considering this is a B2B API with a paying customer at the end of the experience, the documentation should really be considered part of their sales strategy as this is nothing more than content marketing. Good docs sell.

When your market strategy is paid support, bad docs can seem appealing. I think that’s a short sighted and uninspired monetisation strategy. How many potential customers get stuck and simply continue on to the next provider on that “Top 10 List of …” blog post or video out there?

I’ve been using and recommending GitLab to employers, clients and a dozen personal projects over the years. If it wasn’t for that I wouldn’t have come back today to push on. I would have quietly ditched it and moved on to the next contender on the list. I was excited about GitLab Pipelines, but I can’t recommend them based on this experience. I think there’s plenty of opportunity to improve it. I respect and understand the iterative process so I hope you’ll accept my feedback and that it will help things improve.

If nothing else, please get rid of the warning icon with the error message behind a damn hover thing! GAH! That’s an absolutely terrible thing to do.

Ok, glad you seem to have it working. On the whole strategy and other questions, I’m afraid I don’t have an answer.
We have been using Gitlab with docker runners for quite a while now, without major problems or extreme difficulties setting it up. We do have a paid version, but so far we haven’t had the need to go to support and found most documentation adequate (had some troubles on coupling our own Oauth solution, but that’s all I can remember on missing documentation, also at that moment I had just started with Gitlab as well as Oauth, so it also might have been some newby troubles).
I’m not a Gitlab employee,but I sympathise with people who have a hard time getting stuff working, good luck!

Thank you for help @bartj. I appreciate it. It’s possible that I’ve been spoiled with good docs recently. It’s been a long time since I had to work so hard at parsing the information within. The Stripe documentation totally ruined my expectations I think LOL.

Maybe when this is all said and done I’ll write a lengthy blog post explaining it all for whoever comes next; which could be future me.

Hi @Strixy,

I had a similar problem recently, where the runner did not connect to GitLab SASS. How did you start your runner? Did you use the runner command directly gitlab-ci-multi-runner start or did you use the systemd service sudo service gitlab-runner start (or something?)?

Sarah

Hi @snim2

I was following the documentation for Linux machines which says to use sudo gitlab-runner register only it didn’t fully register for some reason. When I ran gitlab-runner --debug run things worked, but only while debugging. When I ran the register command a second time it registered a second runner and everything worked.

I think I caused the issue originally by not issuing the sudo part of the register command. At least, that’s the best fit explanation I can come up with. It doesn’t seem to explain everything, but it does explain a lot of it. You could try running yours with sudo and/or without sudo to see which, if either, works as expected.

What I’ve discovered is that the difference in how you execute the gitlab-runner command (ie. as a user level program or as an admin (ie. w/ sudo) changes which config.toml file it uses. A user level execution (ie sans sudo) will use the one found in your home folder eg. ~/.gitlab-runner/config.toml which is, I think, expected behaviour. An admin executed command (ie w/ sudo) will use the one in the /etc/gilab-runner/config.toml which is, I think, the expected behaviour.

In order to keep things running smoothly you’ll need to stick to one paradigm or the other; either use sudo consistently (not a great idea for a production environment, imho), use your own user account every time (which is likely not what you want as a permanent solution for production assets either), or run it under a gitlab-runner user all the time and ensure your config.toml file in /home/gitlab-runner/.gitlab-runner/config.toml is the one that’s correctly registered and configured, but I’m not 100% sure on that last point yet. I haven’t moved into that part of the process myself. That’s just where I think the correct toml file would go. I haven’t tested that theory yet.

If you’re on Windows, Mac or in a container I can’t even being to help and I doubt anything above applies. I’m still new here myself.

Hi @Strixy,

yes, you’re exactly right about the difference between running with sudo and not. There’s also a difference between using systemd and just starting the service as the gitlab-runner user.

It’s also worth looking at https://status.gitlab.com as just occasionally you will have an error that is actually an outage at the GitLab end of things.

Hopefully the rest of your CI/CD journey is a bit smoother, it’s honestly not usually this confusing!

Sarah

Thanks @snim2. I’m not sure what happens under the hood of sudo gitlab-runner register on an Ubuntu box (I haven’t seen that explained in any docs, blogs or S/o, but I haven’t really gone looking for that information yet either).