Terraform creation of Gitlab-CE installed with Omnibus in AWS

Infrastructure as Code & Cloud Native
1

Terraform creation of Gitlab-CE installed with Omnibus in AWS

Is anyone else doing this? How are you doing your backup and restore when you need to replace the gitlab instance for some reason? I recently needed to up the disk size of my instance so when I made the change in TF I needed to terminate my currently running instance and then apply the new TF plan which included the new disk size. Once the new instance comes up I have a template file in TF which is parsed into the user data for the instance and it gets a run through and then gitlab runs a reconfigure. I then run a restore of my backup I made before I kick this all off. After that im able to smoketest the instance and verify everything is working. I have run into some issues before with the secrets file being corrupted which caused me to lose CI/CD variables in a few projects. That was a pain.
So how do you do this with terraform? I feel like the way im doing this is pretty hacky but it works and I can typically have a new instance back up and running in about 15 minutes assuming everything goes as planned. Im thinking of using Packer to build an ami of the gitlab instance that gets built when a new release comes out and we can more easily spin up a newly pre-configured instance. The main hiccup I seem to run into is the restore process of secrets and data so thats what im curious to hear how you are dealing with this.

Heres a copy of some of terraform docs with private data redacted or changed. Also im not providing every single thing here like networking and database creation and what not. Ill comment inline if there is something that needs explaining. Also, yes, im aware I could do this in Chef or Ansible but we are specifically moving away from those things as this project would basically be the only thing using chef/ansible.

Would love to hear your thoughts and suggestions.

A few configuration notes:

  • I run the registry to be stored in a S3 bucket.
  • Docker gets installed for pipeline purposes.
  • The database is an external RDS instance in AWS.
  • Backups are run by a cronjob and stored in a S3 bucket.
  • Gitlab Runner runs on an instance by itself and is a pain to setup and get running. It works great once you get it working though. We use a separate instance and template file for it as well.
data "template_file" "gitlab-userdata" {
  template = "${file("${path.module}/gitlab-userdata.tpl")}"

  vars {
    region                          = "${var.region}"
    docker_bucket                   = "${aws_s3_bucket.gitlab_docker_bucket.bucket}"
    access_key                      = "${var.aws_access_key}"
    secret_key                      = "${var.aws_secret_key}"
    db_database                     = "${var.gitlab_postgres_name}"
    db_username                     = "${var.gitlab_postgres_username}"
    db_password                     = "${var.gitlab_postgres_password}"
    db_host                         = "${aws_db_instance.gitlab_postgresql.address}"
    db_port                         = "${var.gitlab_postgres_port}"
    smtp_enable                     = "${var.gitlab_ses_smtp_enable}"
    smtp_address                    = "${var.gitlab_ses_smtp_address}"
    smtp_port                       = "${var.gitlab_ses_smtp_port}"
    smtp_user_name                  = "${var.gitlab_ses_smtp_user_name}"
    smtp_password                   = "${var.gitlab_ses_smtp_password}"
    smtp_domain                     = "${var.gitlab_ses_smtp_domain}"
    smtp_authentication             = "${var.gitlab_ses_smtp_authentication}"
    smtp_enable_starttls_auto       = "${var.gitlab_ses_smtp_enable_starttls_auto}"
    backup_upload_remote_directory  = "${aws_s3_bucket.gitlab_bucket.bucket}"
  }
}

resource "aws_instance" "gitlab_instance" {
  ami                               = "${var.gitlab_ami_id}"
  instance_type                     = "${var.instance_type}"
  user_data                         = "${data.template_file.gitlab-userdata.rendered}"
  iam_instance_profile              = "${aws_iam_instance_profile.gitlab_instance_profile.id}"
  key_name                          = "${aws_key_pair.utility_gitlab.key_name}"
  associate_public_ip_address       = true
  source_dest_check                 = false
  vpc_security_group_ids            = ["${aws_security_group.gitlab_instance.id}","${var.default_security_group_id}"]
  subnet_id                         = "${element(var.public_subnets, 0)}"
  root_block_device {
      volume_size = "100"
      volume_type = "gp2"
  }
  tags {
    Name                            = "${var.project}"
    Owner                           = "${var.owners}"
    Environment                     = "${var.environment}"
  }
}

And here’s my template file I use…

#!/bin/bash -ex
sudo echo "127.0.0.1 $(hostname)" >> /etc/hosts
sudo apt-get update -y
sudo apt-get install -y curl openssh-server ca-certificates apt-transport-https ca-certificates software-properties-common

#Install awscli
sudo apt install -y python-pip
sudo pip install --upgrade pip
sudo pip install awscli

curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash
#The following URL needs to be kept as HTTP
sudo EXTERNAL_URL="http://gitlab.mydomain.net" apt-get install gitlab-ee

echo "#DATABASE" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['db_adapter'] = 'postgresql'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['db_encoding'] = 'unicode'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['db_pool'] = '10'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['db_username'] = 'gitlab'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['db_password'] = '${db_password}'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['db_host'] = '${db_host}'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['db_port'] = '5432'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['db_prepared_statements'] = 'false'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['db_statements_limit'] = '1000'" >> /etc/gitlab/gitlab.rb

echo "#SES Email Settings" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['smtp_enable'] = '${smtp_enable}'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['smtp_address'] = '${smtp_address}'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['smtp_port'] = '${smtp_port}'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['smtp_user_name'] = '${smtp_user_name}'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['smtp_password'] = '${smtp_password}'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['smtp_domain'] = '${smtp_domain}'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['smtp_authentication'] = '${smtp_authentication}'" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['smtp_enable_starttls_auto'] = '${smtp_enable_starttls_auto}'" >> /etc/gitlab/gitlab.rb

echo "#Uploading backups to a remote S3 storage" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['backup_upload_connection'] = {" >> /etc/gitlab/gitlab.rb
echo "  'provider' => 'AWS'," >> /etc/gitlab/gitlab.rb
echo "  'region' => 'us-east-1'," >> /etc/gitlab/gitlab.rb
echo "  'use_iam_profile' => 'true'" >> /etc/gitlab/gitlab.rb
echo "}" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['backup_upload_remote_directory'] = '${backup_upload_remote_directory}'" >> /etc/gitlab/gitlab.rb

# limit backup lifetime to 7 days - 604800 seconds
echo "gitlab_rails['backup_keep_time'] = 604800" >> /etc/gitlab/gitlab.rb

echo "#Adjust unicorn" >> /etc/gitlab/gitlab.rb
echo "unicorn['worker_processes'] = 3" >> /etc/gitlab/gitlab.rb
#echo "unicorn['listen'] = '127.0.0.1'" >> /etc/gitlab/gitlab.rb

#Installing these certs this way seems super hacky :(

echo "#SSL certs" >> /etc/gitlab/gitlab.rb
echo "letsencrypt['enable'] = false" >> /etc/gitlab/gitlab.rb
sudo mkdir -p /etc/gitlab/ssl
sudo chmod 700 /etc/gitlab/ssl

sudo aws s3 cp s3://mydomain-devops/Certs/__mydomain_net/__mydomain_net.key /etc/gitlab/ssl/__mydomain_net.key
sudo cp /etc/gitlab/ssl/__mydomain_net.key /etc/gitlab/ssl/gitlab.mydomain.net.key

sudo aws s3 cp s3://mydomain-devops/Certs/__mydomain_net/__mydomain_net.crt /etc/gitlab/ssl/__mydomain_net.crt
sudo cp /etc/gitlab/ssl/__mydomain_net.crt /etc/gitlab/ssl/gitlab.mydomain.net.crt

sudo aws s3 cp s3://mydomain-devops/Certs/__mydomain_net/__mydomain_net.chained.crt /etc/gitlab/trusted-certs/__mydomain_net.chained.crt
sudo cp /etc/gitlab/trusted-certs/__mydomain_net.chained.crt /etc/gitlab/ssl/__mydomain_net.chained.crt
sudo mv /etc/gitlab/trusted-certs/__mydomain_net.chained.crt /etc/gitlab/trusted-certs/gitlab.mydomain.net.chained.crt

sudo aws s3 cp s3://mydomain-devops/Certs/__mydomain_net/__mydomain_net.ca-bundle /etc/gitlab/ssl/__mydomain_net.ca-bundle
sudo cat /etc/gitlab/ssl/__mydomain_net.chained.crt /etc/gitlab/ssl/__mydomain_net.ca-bundle >> /etc/gitlab/ssl/gitlab.mydomain.net.pem

echo "nginx['ssl_certificate'] = '/etc/gitlab/ssl/gitlab.mydomain.net.pem'" >> /etc/gitlab/gitlab.rb
echo "nginx['ssl_certificate_key'] = '/etc/gitlab/ssl/gitlab.mydomain.net.key'" >> /etc/gitlab/gitlab.rb
echo "nginx['redirect_http_to_https'] = true" >> /etc/gitlab/gitlab.rb

echo "#Container Registry" >> /etc/gitlab/gitlab.rb
echo "registry['enable'] = true" >> /etc/gitlab/gitlab.rb
echo "registry_external_url 'https://gitlab.mydomain.net:6565'" >> /etc/gitlab/gitlab.rb
echo "registry_nginx['ssl_certificate'] = '/etc/gitlab/ssl/gitlab.mydomain.net.pem'" >> /etc/gitlab/gitlab.rb
echo "registry_nginx['ssl_certificate_key'] = '/etc/gitlab/ssl/gitlab.mydomain.net.key'" >> /etc/gitlab/gitlab.rb
echo "registry['storage'] = {" >> /etc/gitlab/gitlab.rb
echo "  's3' => {" >> /etc/gitlab/gitlab.rb
echo "    'accesskey' => '${access_key}'," >> /etc/gitlab/gitlab.rb
echo "    'secretkey' => '${secret_key}'," >> /etc/gitlab/gitlab.rb
echo "    'bucket' => '${docker_bucket}'," >> /etc/gitlab/gitlab.rb
echo "    'region' => '${region}'" >> /etc/gitlab/gitlab.rb
echo "  }" >> /etc/gitlab/gitlab.rb
echo "}" >> /etc/gitlab/gitlab.rb
#To avoid bug with registry healthcheck and S3 we have to put a blank file in the bucket:
touch tempfile
aws s3 cp tempfile s3://${docker_bucket}/

echo "#Monitoring settings" >> /etc/gitlab/gitlab.rb
echo "###! IP whitelist controlling access to monitoring endpoints" >> /etc/gitlab/gitlab.rb
echo "gitlab_rails['monitoring_whitelist'] = ['192.168.0.0/16', '10.10.0.0/16', '100.100.0.0/16', '200.200.0.0/16']" >> /etc/gitlab/gitlab.rb
echo "###! Time between sampling of unicorn socket metrics, in seconds" >> /etc/gitlab/gitlab.rb

echo "Executing gitlab-ctl reconfigure"
sudo gitlab-ctl reconfigure

#Setup Cronjob for DATA and configs backups
sudo echo "30 * * * * /usr/local/bin/aws s3 cp --recursive /etc/gitlab/ s3://mydomain-devops/Gitlab/" >> gitlabcronjobs
sudo echo "0 * * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create" >> gitlabcronjobs
sudo crontab gitlabcronjobs

#Install Docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install -y docker-ce
sudo docker run hello-world
#Docker CE is installed and running. The docker group is created but no users are added to it. You need to use sudo to run Docker commands.
1 Like