Installer fails, remembers faulty configuration, won't forget

I am trying to get a gitlab server working on my local network. I wanted to have it listen on an other port than 80, so when the installer asked for a domain name I entered an IP and port number, like xxx.xxx.xxx.xxx:xxxx . However, the installer did no input sanity checking, failed spectacularly trying to create a directory or file named like this, and now I am having the greatest trouble to get rid of this failed installation. I tried removing packages, purging, manually removing all files, to no avail. Every time I try reinstalling I get this:

Creating runtime directories for gitlab...
Updating file permissions...
Configuring hostname and email...
Registering /etc/gitlab/gitlab.yml via ucf

Creating config file /etc/gitlab/gitlab.yml with new version
Registering /etc/gitlab/gitlab-debian.conf via ucf

Creating config file /etc/gitlab/gitlab-debian.conf with new version

Creating config file /etc/nginx/sites-available/xxx.xxx.xxx.xxx:xxxx with new version
cp: cannot create regular file '/etc/nginx/sites-available/xxx.xxx.xxx.xxx:xxxx': No such file or directory
dpkg: error processing package gitlab (--configure):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu4) ...
Errors were encountered while processing:
 gitlab
E: Sub-process /usr/bin/dpkg returned an error code (1)

How to purge this faulty configuration without having to smash my harddrive with a hammer?