Newly added runner does not show up on GitLab

I have templated my config.toml file so that I can add runners via Ansible. It seems to work, but the runner is not showing up in the list of runners available to the project.

gitlab-runner list does show the new runner.

I’ve tried a couple web searches, but this is a hard one to figure out the correct terms…

I did see this issue. Um, it sounds like you can’t just add a runner to the toml file directly? Is that correct?

Further investigation revealed I can call the gitlab api to register a new runner.

Using the ansible uri module does register a runner, but the runner never connects. I’m left with a grey triangle ! icon and a “New runner. Has not connected yet.” message in the settings of my project.

It also ends up adding multiple new runners to the project every time the play is run.

syslog on the runner host has lots of messages like:

Jun 25 12:15:11 runnerhost.example.org gitlab-runner[1057]: time="2018-06-25T12:15:11-07:00" level=error msg="Runner https://gitlab.example.org/9HX7jXhhzKZi_YMw6kNB is not healthy and will be disabled!"
Jun 25 12:16:11 runnerhost.example.org gitlab-runner[1057]: time="2018-06-25T12:16:11-07:00" level=error msg="Checking for jobs... forbidden" runner=9HX7jXhh
Jun 25 12:16:11 runnerhost.example.org gitlab-runner: time="2018-06-25T12:16:11-07:00" level=error msg="Checking for jobs... forbidden" runner=9HX7jXhh
Jun 25 12:16:11 runnerhost.example.org gitlab-runner: time="2018-06-25T12:16:11-07:00" level=error msg="Runner https://gitlab.example.org/9HX7jXhhzKZi_YMw6kNB is not healthy and will be disabled!"
Jun 25 12:16:11 runnerhost.example.org gitlab-runner[1057]: time="2018-06-25T12:16:11-07:00" level=error msg="Runner https://gitlab.example.org/9HX7jXhhzKZi_YMw6kNB is not healthy and will be disabled!"

How can I figure out why the runner is not healthy?

Any suggestions on how to prevent multiple runners from being added every time the play is run?

For reference:

My ansible tasks:

- name: "Template gitlab-runner's config.toml."
  template:
    src: "gitlab-runner-config.toml.j2"
    dest: "{{ common_gitlabrunner_config_dest | default('/etc/gitlab-runner/config.toml') }}"
    owner: "{{ common_gitlabrunner_config_owner | default('root') }}"
    group: "{{ common_gitlabrunner_config_group | default('root') }}"
    mode: "0600"
  notify: "restart gitlab-runner"
  tags:
  - common
  - common_gitlabrunner

- name: "Post runners to gitlab api."
  uri:
    url: "{{ item.value.url | default('https://git.lanecc.edu/') }}api/v4/runners"
    method: "POST"
    body: 'token={{ item.value.token }}&description={{ item.value.name | urlencode }}&tag_list={{ item.value.tags }}'
    status_code: '200,201'
  with_dict: "{{ common_gitlabrunner_runners }}"
  tags:
  - common
  - common_gitlabrunner

Ansible config:

common_gitlabrunner_config_dest: "/home/specialuser/.gitlab-runner/config.toml"
common_gitlabrunner_config_owner: "specialuser"
common_gitlabrunner_config_group: "users"
common_gitlabrunner_runners:
  mainansiblerunner:
    name: "runnerhost_main_shell"
    token: "9HX7jXhhzKZi_YMw6kNB"
    executor: "shell"
    locked: "yes"
    tags: "tag1,tag2"

I’m running gitlab-runner under a specialuser user, not root or the gitlab-runner user. I’m experimenting with having hosts use gitlab-runner to pull their Ansible configuration and apply it locally, rather than pushing the configuration from another host. Ideally I want gitlab-runner to automatically run the initial configuration of a host as soon as it is done being cloned from a vmware vsphere template.