How do you build and run the gitlab CI multi runner on Windows?

How to Use GitLab
1

I see a Makefile that looks pretty Unix-y to me, and I don’t see any batch (.cmd or .bat) file to kick the build off. Am I supposed to cobble together a mingw/cygwin and build using the makefile from there? How do people build and package a go multi platform code project like this and then make it easy to build it from windows as well as from linux?

If this was a C project I’d expect to clone a repo, configure make and install. There is no standard make tool on Windows, and no build notes about the windows build process in the readme.

I have go installed but go install alone does not work because I guess go has some kind of manual dependency fetching bundling/package management stages (using several different optional sets of possible tools, if I have read correctly) that you do before you actually do the main go build step.

I am totally inexperienced in Go, and this is the first Go project I’ve ever checked out and I have to say, it seems to me that it’s like Go projects when you get them are not exactly obvious as to how to use them.

Update: I have installed IntelliJ and the Go plugin and this really helps me figure out why the code won’t build for me. I’m stuck with an error in cache.go:

..\src\gitlab.com\gitlab-org\gitlab-ci-multi-runner\shells\cache.go:63: cannot assign *url.URL to url (type string) in multiple assignment
..\src\gitlab.com\gitlab-org\gitlab-ci-multi-runner\shells\cache.go:97: cannot assign *url.URL to url (type string) in multiple assignment

The source line 63 in cache.go is:
url, err = scl.PresignedGetObject(cache.BucketName, objectName, time.Second*time.Duration(build.Timeout), nil)

I’m guessing some kind of implicit cast that should have happened isn’t happening.

syntax error.png1399×737 60 KB

Warren

2

The reason that there aren’t specific build instructions for Windows is because the Go compiler has built-in cross-platform compilation, so it is likely that no one has needed to do the build on Windows yet.

3

Hi Dude. I am aware that Go has a build command.

But to fetch the dependencies with go-get, it appears a UNIX only Makefile is provided.

So when I say build, let’s just say I mean “run the steps required to do a complete build, including fetching packages.”

So I manually fetched the modules, set up my GO library paths, and I have this compile error when I build, shown above, but I’ll repeat it here:

..\src\gitlab.com\gitlab-org\gitlab-ci-multi-runner\shells\cache.go:63: cannot assign *url.URL to url (type string) in multiple assignment
..\src\gitlab.com\gitlab-org\gitlab-ci-multi-runner\shells\cache.go:97: cannot assign *url.URL to url (type string) in multiple assignment

If anyone knows why this code and its wonderful portable Go Builds Anywhere Even On Windows code doesn’t build for me, that’d be super duper. Thanks.

Otherwise, how is someone going to debug, say, new windows shell integrations if they can’t even build on windows? Go may even have a lovely debugger now which I have installed, and I’d love to build and run this from within an IDE (IntelliJ IDEA Community) and figure out how to debug and build stuff that is for Windows integration purposes, inside the gitlab CI tooling.

Warren

4

Warren,

My original response was aimed at these parts of your original post:

… How do people build and package a go multi platform code project like this and then make it easy to build it from windows as well as from linux?

If this was a C project I’d expect to clone a repo, configure make and install. There is no standard make tool on Windows, and no build notes about the windows build process in the readme.

It took me a little time to find the reason for your error, but it turns out that it’s pretty simple.

You have a bad version of the minio/minio-go dependency. This git commit:

changed the return type of the method from (string, error) to (url.URL, error). That commit was made 10 days ago.

You need to back off that dependency to the previous version.

5

Cool! Thanks for doing the googles for me. I had no idea what that was about.

I am brand new at Go and this is my first time trying to work with Go code. I figured, why not try to build something cool that I need, into this GitLab CI client. What better way to learn right?

Warren

6

I haven’t worked with Go in a while (used it for a class in Fall 2014), but in my opinion it’s a really nice language once you get used to how weird it looks.

The way I heard the story is that Google went to a bunch of programming language inventors and told them “if you knew then what you know now, how would you have designed a major programming language like C?”

7

It’s the most opinionated language I have ever seen. It appears that you should open the file called .godir and make sure that your source is in that directory specified relative to your go root dir, and if you don’t do that you’re gonna have a bad time building.

After I got the above minio change worked around, the next error is:

# gitlab.com/gitlab-org/gitlab-ci-multi-runner/executors/docker
executors\docker\executor_docker.go:151: undefined: Asset

Code:

	data, err := Asset(PrebuiltArchive)
	if err != nil {
		return
	}

Line of code that breaks is an Asset(PrebuiltArchive) one, and from googling I have to type "go generate " but I don’t know what, and this will generate some binary file that the above Asset instruction shoves inline into my binary, somewhat like how Resource File Binaries work in Win32 windows programming.

It seems some people check these binary assets in so that they don’t have to be done manually.

For me it would suffice to disable the docker stuff if I could get the CI agent to build without docker features it would be fine, as I am on Windows here, and docker isn’t much good on Windows.

Update: Looks like I can manually fetch it on windows powershell curl with:
curl -o out/docker/prebuilt.tar.gz https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/master/docker/prebuilt.tar.gz

It seems to me that a windows specific build tooling is required, that configures the source code in whatever way you configure this highly opinionated ball of Go compiler that Docker doesn’t exist on windows and that we can build this gitlab CI agent on Windows without its docker parts.

I don’t see a way in Go to disable a single executor and not build it. I had to actually DELETE the executors/docker sources from my working copy, then hack up the places in main.go and exec.go that reference it, to get the whole deal to build and run.

Favorite thing I learned is that to disable a module on Windows, you add a comment to the top of the file, followed by an empty line. If you don’t have the empty line, it doesn’t work.

There needs to be a way to build on windows, and right now there is not.

Warren

8

Latest GitLab-CI repo fetch from today brings it back into sync with the MinioGO changes.

I still think it ought to be possible to run build.cmd batch file and build, and if I figure out how to make one, I’ll make a pull request for it.

My build.cmd will, in fashionable GO-friendly ways do the following:

  1. Tell you if you haven’t set GOROOT or GOPATH.

  2. Tell you if you haven’t complied with GO’s One True Holy Shining Happy Lucky Folder Structure.

  3. Require you to turn three times and face Hoboken New Jersey while singing a song by Bruce Springsteen. If you do so, it will disable the Docker shell integration feature, so you can build the rest on windows and test with it.

  4. Build the code for you.

9

So here is what I’ve learned, going from 0 to 60 over several months. If any of this seems pedantic, please rest assured that I’m just trying to write the things that I, a windows (and linux) guy with zero prior Go experience, learned, from building this project, which is currently 100% developed by people who run Linux and Macs, from what I can see.

  1. Go is a highly opinionated language, and checking out a whole project like Gitlab CI runner and expecting that you could build it as easily and with as little understanding of Go, as would be possible with a Java or Python project was my first mistake. It’s not. For one thing it thinks you should create a root source folder and structure all the folders inside a certain way, and then git clone into those in a magic and almost-non-optional way. Did that sound confusing? If you’re new to Go, it sure is, and there’s no real ways around it. Learn about GOROOT and GOPATH, and build a few little toy Go apps first.

  2. After learning some Go, then, if you’re going to build a project that currently does not build or pass tests on Windows (as of October 4, 2016), be prepared for some things not to work.

  3. The default make target build all redistributable binaries, and runs tests. Tests do not currently pass on Windows.

  4. I was able to build on Windows, because I downloaded and placed in my path, the MSYS2 64-bit (msys64) tools, a cygwin-like set of bash/gnu/gcc/make binary tools for windows. At a minimum you must have bash, gnu make, curl, and about a dozen other common posix shell binary tools present and working for the Makefile to work. The Makefile is the only current way to build Gitlab-CI-Runner. It might be possible to get this working with cygwin, but I had better luck with MSYS2.

  5. It seems to me that Go lacks (intentionally, as always) any way to conditionally define out features of a gitlab runner. So it’s not currently possible to build a version of Gitlab-CI-Runner that omits the parts that I don’t personally care about yet, like container runners.

What have I still not figured out?

  1. How to debug the gitlab runner on Windows? Should be possible with delve, and maybe vscode or intellij + go plugin.
10

That’s a good summary. The feeling as an outsider coming into Go is a bit like arriving in Jonestown.

The kool-aid is everywhere.

W