Error uploading artifacts to coordinator

GitLab CI/CD
1

I’ve been having some fun setting up GitLab and after spending quite a while hacking away at it, I’ve become relatively used to setting it up, now having done that on two machines, the second time around with much more ease than originally…

However, I am faced with a rather large problem, on both machines: My CI pipeline is broken. Somehow, somewhere, my setup is providing a 403 to artifacts once builds are completed, meaning that each and every job that ever technically succeeds will only be doomed to fail…

I upgraded GitLab CE to 10.1.4 minutes ago, as well as GitLab-runner to 10.1.0, the latest packages available to me through apt on the more important of the two machines, running a newer version of Ubuntu than the other - 17.04 zesty on the “beast” compared to 16.10 yakkety on “q2”.
Both gitlab-runner registrations use shell for execution.

The relevant output of the CI job is as follows:

Cloning repository...
Cloning into '/[clonepath]'...
Checking out 8319d586 as master...
Skipping Git submodules setup
mesg: ttyname failed: Inappropriate ioctl for device
mesg: ttyname failed: Inappropriate ioctl for device
mesg: ttyname failed: Inappropriate ioctl for device
$ mvn -B install
[INFO] Scanning for projects...

 ...

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.204 s
[INFO] Finished at: 2017-11-18T05:45:08+01:00
[INFO] Final Memory: 27M/640M
[INFO] ------------------------------------------------------------------------
mesg: ttyname failed: Inappropriate ioctl for device
mesg: ttyname failed: Inappropriate ioctl for device
mesg: ttyname failed: Inappropriate ioctl for device
Uploading artifacts...
target/*.jar: found 1 matching files               
ERROR: Uploading artifacts to coordinator... forbidden  id=35 responseStatus=403 Forbidden status=403 Forbidden token=sP9oHykF
FATAL: permission denied                           
ERROR: Job failed: exit status 1

I run GitLab under an Apache2 Vhost subdomain, mostly for aesthetic and omitting of the port following the host, i.e. 8080 for unicorn, since there are other sites running on Apache.

These are the configured options within my gitlab.rb:

gitlab_rails['trusted_proxies'] = [ '127.0.0.1' ]
gitlab_workhorse['listen_network'] = "tcp"
gitlab_workhorse['listen_addr'] = "127.0.0.1:8181"
nginx['enable'] = false

Setting the values in either of the following options/values as such

web_server['username'] = 'www-data'
web_server['group'] = 'www-data'

produces an error on reconfiguration:

Starting Chef Client, version 12.12.15
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
  - gitlab (0.0.1)
  - registry (0.1.0)
  - package (0.1.0)
  - consul (0.0.0)
  - runit (0.14.2)
Installing Cookbook Gems:
Compiling Cookbooks...
Recipe: gitlab::default
  * directory[/etc/gitlab] action create (up to date)
  Converging 408 resources
  * directory[/etc/gitlab] action create (up to date)
  * directory[Create /var/opt/gitlab] action create (up to date)
  * directory[/opt/gitlab/embedded/etc] action create (up to date)
  * template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date)
Recipe: gitlab::web-server
  * group[Webserver user and group] action create (up to date)
  * user[Webserver user and group] action create

    ================================================================================
    Error executing action `create` on resource 'user[Webserver user and group]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '8'
    ---- Begin output of ["usermod", "-s", "/bin/false", "-d", "/var/opt/gitlab/nginx", "www-data"] ----
    STDOUT:
    STDERR: usermod: user www-data is currently used by process 2656
    ---- End output of ["usermod", "-s", "/bin/false", "-d", "/var/opt/gitlab/nginx", "www-data"] ----
    Ran ["usermod", "-s", "/bin/false", "-d", "/var/opt/gitlab/nginx", "www-data"] returned 8

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/definitions/account.rb

     38:     user params[:name] do
     39:       username username
     40:       shell params[:shell]
     41:       home params[:home]
     42:       uid params[:uid]
     43:       gid params[:ugid]
     44:       system params[:system]
     45:       supports params[:user_supports]
     46:       action params[:action]
     47:     end
     48:   end

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/definitions/account.rb:38:in `block in from_file'

    user("Webserver user and group") do
      params {:action=>nil, :username=>"www-data", :uid=>nil, :ugid=>"www-data", :groupname=>"www-data", :gid=>nil, :shell=>"/bin/false", :home=>"/var/opt/gitlab/nginx", :system=>true, :append_to_group=>true, :group_members=>["www-data"], :user_supports=>{:manage_home=>false}, :manage=>true, :name=>"Webserver user and group"}
      action [:create]
      supports {:manage_home=>false}
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      username "www-data"
      gid 33
      home "/var/opt/gitlab/nginx"
      shell "/bin/false"
      system true
      iterations 27855
      declared_type :user
      cookbook_name "gitlab"
      recipe_name "web-server"
    end

    Platform:
    ---------
    x86_64-linux


Running handlers:
Running handlers complete
Chef Client failed. 0 resources updated in 04 seconds

And as for Apache, here’s the SSL-enabled Vhost:

<IfModule mod_ssl.c>
  <VirtualHost *:443>
    ServerName [host]
    ServerAdmin [email]
    DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public
    ServerSignature Off
    ProxyPreserveHost On
    AllowEncodedSlashes NoDecode
    <Location />
      Order deny,allow
      Allow from all
      Require all granted
      ProxyPassReverse http://127.0.0.1:8181/
      ProxyPassReverse http://[host]/
      RequestHeader set X-Forwarded-Ssl 'on'
    </Location>
    RewriteEngine on
    RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
    RewriteCond %{REQUEST_URI} ^/uploads/.*
    RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE]

    SSLCertificateFile /etc/letsencrypt/live/[host]/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/[host]/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
  </VirtualHost>
</IfModule>

Any idea what’s going on? I haven’t dug the Apache log information yet since it probably won’t be Apache as the request goes straight to gitlab-worker (8181). What logs should I check for that, if necessary?

Thankyou for your time.

2

This isn’t a particularly helpful answer, as the solution has little explanation for its workings.

The configurations I had have remained the same as per above, but the runner I had installed, I removed the config of, rm /etc/gitlab-runner/config.toml, and then proceeded to remove the package from the machine, apt purge gitlab-runner. (gitlab-ci-multi-runner is another package that is available but does not appear to be up to date with GitLab 10 - returns a 404 rather than connecting to the node).

I reinstalled the runner, apt install gitlab-runner, and then registered it - gitlab-runner register. The key thing to note here is that during registration, I used my FQDN, as in https://git.example.com rather than any local address such as http://localhost:8080 or http://localhost:8181 (unicorn, gitlab-workhorse, respectively). And yes, I am running my runners on my local machine. Hazardous, but I have too much trust in my team. That may be our downfall, ignorant systems administration is key to success.