WordPress Pipeline

rhand · January 5, 2024, 1:48am

We would like to deploy on push to staging or production and are now working on a pipeline or action. We work with ploi.io for provisioning and management and they have deployment hooks and there even is a Github action to use for it GitHub - Glennmen/ploi-deploy-action: Deploy your application to Ploi with Github actions . We however need a setup for Gitlab. So we made a basic pipeline to deploy latest repo code to our VPS.

stages:
  - deploy

variables:
  VPS_IP_PRODUCTION: "production_vps_ip"
  VPS_IP_STAGING: "staging_vps_ip"
  SSH_PORT: "your_ssh_port"
  SSH_USER: "your_ssh_user"
  SSH_PRIVATE_KEY: "your_ssh_private_key"

deploy_production:
  stage: deploy
  only:
    - master
  script:
    - apt-get update -qy
    - apt-get install -y sshpass

    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > "$HOME/.ssh/id_rsa"
    - chmod 600 "$HOME/.ssh/id_rsa"

    - apt-get install -y rsync

    # Deploy to production VPS
    - rsync -av --exclude={'/wp-content/uploads/','/wp-config.php'} ./ $SSH_USER@$VPS_IP_PRODUCTION:/path/to/production/wordpress/folder/

    # SSH into the production VPS and perform additional deployment steps if needed
    - ssh -o StrictHostKeyChecking=no -p "$SSH_PORT" "$SSH_USER@$VPS_IP_PRODUCTION" 'cd /path/to/production/wordpress/folder/ && chown -R www-data:www-data wp-content/uploads/'

  environment:
    name: production
  only:
    - master

deploy_staging:
  stage: deploy
  only:
    - staging_branch  # Adjust the branch name for your staging environment
  script:
    - apt-get update -qy
    - apt-get install -y sshpass

    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > "$HOME/.ssh/id_rsa"
    - chmod 600 "$HOME/.ssh/id_rsa"

    - apt-get install -y rsync

    # Deploy to staging VPS
    - rsync -av --exclude={'/wp-content/uploads/','/wp-config.php'} ./ $SSH_USER@$VPS_IP_STAGING:/path/to/staging/wordpress/folder/

    # SSH into the staging VPS and perform additional deployment steps if needed
    - ssh -o StrictHostKeyChecking=no -p "$SSH_PORT" "$SSH_USER@$VPS_IP_STAGING" 'cd /path/to/staging/wordpress/folder/ && chown -R www-data:www-data wp-content/uploads/'

  environment:
    name: staging
  except:
    - master

We however wonder if there is no similar way to the Github setup:

name: 'Deploy on push'

on:
  push:
    branches:
      - master

jobs:
  ploi-deploy:
    name: 'Ploi Deploy'
    runs-on: ubuntu-latest

    steps:
      # Checkout the repository to the GitHub Actions runner
      - name: Checkout
        uses: actions/checkout@v2

      # Trigger Ploi deploy webhook
      - name: Deploy
        uses: Glennmen/ploi-deploy-action@v1.2.0
        with:
          webhook_url: ${{ secrets.WEBHOOK_URL }}

to deploy using a web hook .

Perhaps instead of using rsync we should use git directly:

before_script:
  - apt-get update -qq
  - apt-get install -qq git
  - 'which ssh-agent || ( apt-get install -qq openssh-client )'
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  - mkdir -p ~/.ssh
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no" > ~/.ssh/config'
  - ssh-keyscan your-vps-ip >> ~/.ssh/known_hosts
  - chmod 644 ~/.ssh/known_hosts

deploy_live:
  environment:
    name: Live
    url: your-primary-domain
  script:
    - ssh $SSH_USER@$VPS_IP -p $SSH_PORT "cd /path/to/your/wordpress/folder && git checkout main && git pull origin main && exit"
  only:
    - main

deploy_staging:
  environment:
    name: Staging
    url: your-staging-domain
  script:
    - ssh $SSH_USER@$VPS_IP -p $SSH_PORT "cd /path/to/your/wordpress/folder && git checkout staging && git pull origin staging && exit"
  only:
    - staging

as mentioned by Site Management - GitLab CI/CD - Kinsta® Docs . Also tested GitHub - tcallsen/wordpress-gitlab-ci: A sample project that demonstrates how to auto-deploy a WordPress site using GitLab CI/CD. Deployments are handled over SSH. a bit which looks really neat is not set up to run on push

We also wonder if Gitlab has any ready made pipelines for WordPress theme or wp-content folder data on push.

rhand · January 5, 2024, 6:42am

Working on this version now. May remove WordPress addition and stick to only adding of our versioned themes, plugins. Once WordPress is installed I would think we can manage on the server and only need to deploy wp-content changes except for uploads folder. And I add rule to only deploy when pushed to a staging or production branch .

stages:
  - deploy-stage
  - deploy-production

variables:
  WORDPRESS_VERSION: "6.3.2"
  WORDPRESS_SITE_DIR: /var/www/html/blog
  WORDPRESS_THEME_NAME: myBlog
  WORDPRESS_PLUGINS: "ewww-image-optimizer google-sitemap-generator wp-sweep"
  NODE_BIN_PATH: /usr/local/nvm/node/v14.15.3/bin
  COMPOSER_BIN_PATH: /usr/local/bin/composer

deploy-stage:
  stage: deploy-stage
  image: kroniak/ssh-client
  rules:
    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "staging"'
  script:
    - chmod og= $STAGE_ID_RSA

    # prepare destination server - clean destination directory except for /wp-content/uploads
    - ssh -i $STAGE_ID_RSA -o StrictHostKeyChecking=no $STAGE_SERVER_USER@$STAGE_SERVER_IP "
        find $WORDPRESS_SITE_DIR -mindepth 1 ! -regex '^$WORDPRESS_SITE_DIR/wp-content/uploads\(/.*\)?' -delete  || true"
        
    # install wordpress and plugins
    - ssh -i $STAGE_ID_RSA -o StrictHostKeyChecking=no $STAGE_SERVER_USER@$STAGE_SERVER_IP "
        cd $WORDPRESS_SITE_DIR && 
        wp core download --version=$WORDPRESS_VERSION --skip-content &&
        wp core config --dbname=$STAGE_WORDPRESS_DB_NAME --dbuser=$STAGE_WORDPRESS_DB_USER --dbpass='$STAGE_WORDPRESS_DB_PASSWORD' --dbhost=$STAGE_WORDPRESS_DB_HOST &&
        wp plugin install $WORDPRESS_PLUGINS"
    
    # deploy updated theme code to destination server
    - ssh -i $STAGE_ID_RSA -o StrictHostKeyChecking=no $STAGE_SERVER_USER@$STAGE_SERVER_IP "mkdir -p $WORDPRESS_SITE_DIR/wp-content/themes/$WORDPRESS_THEME_NAME"
    - scp -r -i $STAGE_ID_RSA ./wp-content/themes/$WORDPRESS_THEME_NAME $STAGE_SERVER_USER@$STAGE_SERVER_IP:$WORDPRESS_SITE_DIR/wp-content/themes/

    # build Rehub theme & child artifacts on destination server (modify as needed)
    - ssh -i $STAGE_ID_RSA -o StrictHostKeyChecking=no $STAGE_SERVER_USER@$STAGE_SERVER_IP "
        export PATH=$PATH:$NODE_BIN_PATH:$COMPOSER_BIN_PATH &&
        cd $WORDPRESS_SITE_DIR/wp-content/themes/$WORDPRESS_THEME_NAME && 
        composer install &&
        npm install"

    # enable deployed theme and plugins
    - ssh -i $STAGE_ID_RSA -o StrictHostKeyChecking=no $STAGE_SERVER_USER@$STAGE_SERVER_IP "
        cd $WORDPRESS_SITE_DIR && 
        wp theme activate $WORDPRESS_THEME_NAME &&
        wp plugin activate $WORDPRESS_PLUGINS"

    # ensure Wordpress and plugins have permissions on content/themes/plugin dirs
    - ssh -i $STAGE_ID_RSA -o StrictHostKeyChecking=no $STAGE_SERVER_USER@$STAGE_SERVER_IP "
        chgrp -R www-data $WORDPRESS_SITE_DIR || true"

deploy-production:
  stage: deploy-production
  image: kroniak/ssh-client
  rules:
    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "production"'
  script:
    - chmod og= $PRODUCTION_ID_RSA

    # prepare destination server - clean destination directory except for /wp-content/uploads (modify if needed)
    - ssh -i $PRODUCTION_ID_RSA -o StrictHostKeyChecking=no $PRODUCTION_SERVER_USER@$PRODUCTION_SERVER_IP "
        find $WORDPRESS_SITE_DIR -mindepth 1 ! -regex '^$WORDPRESS_SITE_DIR/wp-content/uploads\(/.*\)?' -delete  || true"

    # install wordpress and plugins (modify as needed)
    - ssh -i $PRODUCTION_ID_RSA -o StrictHostKeyChecking=no $PRODUCTION_SERVER_USER@$PRODUCTION_SERVER_IP "
        cd $WORDPRESS_SITE_DIR && 
        wp core download --version=$WORDPRESS_VERSION --skip-content &&
        wp core config --dbname=$PRODUCTION_WORDPRESS_DB_NAME --dbuser=$PRODUCTION_WORDPRESS_DB_USER --dbpass='$PRODUCTION_WORDPRESS_DB_PASSWORD' --dbhost=$PRODUCTION_WORDPRESS_DB_HOST &&
        wp plugin install $WORDPRESS_PLUGINS"
    
    # deploy updated theme code to destination server (modify as needed)
    - ssh -i $PRODUCTION_ID_RSA -o StrictHostKeyChecking=no $PRODUCTION_SERVER_USER@$PRODUCTION_SERVER_IP "mkdir -p $WORDPRESS_SITE_DIR/wp-content/themes/$WORDPRESS_THEME_NAME"
    - scp -r -i $PRODUCTION_ID_RSA ./wp-content/themes/$WORDPRESS_THEME_NAME $PRODUCTION_SERVER_USER@$PRODUCTION_SERVER_IP:$WORDPRESS_SITE_DIR/wp-content/themes/

    # build Rehub theme & child artifacts on destination server (modify as needed)
    - ssh -i $PRODUCTION_ID_RSA -o StrictHostKeyChecking=no $PRODUCTION_SERVER_USER@$PRODUCTION_SERVER_IP "
        export PATH=$PATH:$NODE_BIN_PATH:$COMPOSER_BIN_PATH &&
        cd $WORDPRESS_SITE_DIR/wp-content/themes/$WORDPRESS_THEME_NAME && 
        composer install &&
        npm install"

    # enable deployed theme and plugins (modify as needed)
    - ssh -i $PRODUCTION_ID_RSA -o StrictHostKeyChecking=no $PRODUCTION_SERVER_USER@$PRODUCTION_SERVER_IP "
        cd $WORDPRESS_SITE_DIR && 
        wp theme activate $WORDPRESS_THEME_NAME &&
        wp plugin activate $WORDPRESS_PLUGINS"

    # ensure Wordpress and plugins have permissions on content/themes/plugin dirs (modify as needed)
    - ssh -i $PRODUCTION_ID_RSA -o StrictHostKeyChecking=no $PRODUCTION_SERVER_USER@$PRODUCTION_SERVER_IP "
        chgrp -R www-data $WORDPRESS_SITE_DIR || true"

I also may simplify it a lot more… just want to deploy theme and plugins on push to proper branch really.

rhand · January 5, 2024, 7:51am

Well , I made a more simplified version that only deploy latest

stages:
  - deploy-staging
  - deploy-prod

deploy-staging: 
    image: ubuntu:latest
    stage: deploy-staging
    rules:
        - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "staging"'
    before_script:
        - apt-get -yq update
        - apt-get -yqq install ssh
        - install -m 600 -D /dev/null ~/.ssh/id_rsa
        - echo "$SSH_STAGING_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - ssh-keyscan -p $SSH_PORT $SSH_HOST > ~/.ssh/known_hosts
    script:        
        - ssh $SSH_USER@$SSH_STAGING_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $STAGING_BRANCH && git pull && exit"
    # after_script:
    #     - rm -rf ~/.ssh

deploy-prod: 
    image: ubuntu:latest
    stage: deploy-prod
    rules:
        - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "production"'
    before_script:
        - apt-get -yq update
        - apt-get -yqq install ssh
        - install -m 600 -D /dev/null ~/.ssh/id_rsa
        - echo "$SSH_PROD_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - ssh-keyscan -p $SSH_PORT $SSH_HOST > ~/.ssh/known_hosts
    script:        
        - ssh $SSH_USER@$SSH_PROD_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $PROD_BRANCH && git pull && exit"
    # after_script:
    #     - rm -rf ~/.ssh

Inspiration now is How to setup simple Gitlab CI/CD for your WordPress development – Ranostaj

rhand · January 10, 2024, 6:49am

Could do deployment as separate option and make it manual with variable like so

stages:
  - provision-wordpress-staging
  - deploy-staging
  - deploy-prod

variables:
  WORDPRESS_SITE_DIR: /home/xxx-staging/public
  WORDPRESS_THEME_NAME: xxx-theme
  WORDPRESS_CHILD_THEME_NAME: xxxx-direct
  WORDPRESS_PLUGINS: "admin-columns-for-acf-fields woo-rder-export-lite all-in-one-wp-migration coming-soon wp-sweep wp-contact-form-7 ....."
  NODE_BIN_PATH: /usr/bin/node
  COMPOSER_BIN_PATH: /usr/local/bin/composer

provision-wordpress-staging: 
    image: kroniak/ssh-client
    stage: provision-wordpress-staging
    rules:
        - if: $DEPLOY_WP_TO_STAGING_VAR
        - when: manual
    script:
    - chmod og= $SSH_STAGING_PRIVATE_KEY

    # prepare destination server - clean destination directory except for /wp-content/uploads
    - ssh -i $SSH_STAGING_PRIVATE_KEY -o StrictHostKeyChecking=no $SSH_STAGING_USER@$SSH_STAGING_HOST "
        find $WORDPRESS_SITE_DIR -mindepth 1 ! -regex '^$WORDPRESS_SITE_DIR/wp-content/uploads\(/.*\)?' -delete  || true"
        
    # install wordpress and plugins
    - ssh -i $SSH_STAGING_PRIVATE_KEY -o StrictHostKeyChecking=no $SSH_STAGING_USER@$SSH_STAGING_HOST "
        cd $WORDPRESS_SITE_DIR && 
        wp core download --version=6.4.2 --skip-content &&
        wp core config --dbname=$STAGE_WORDPRESS_DB_NAME --dbuser=$STAGE_WORDPRESS_DB_USER --dbpass='$STAGE_WORDPRESS_DB_PASSWORD' --dbhost=$STAGE_WORDPRESS_DB_HOST &&
        wp plugin install $WORDPRESS_PLUGINS"
    
    # deploy updated theme code to destination server
    - ssh -i $SSH_STAGING_PRIVATE_KEY -o StrictHostKeyChecking=no $SSH_STAGING_USER@$SSH_STAGING_HOST "mkdir -p $WORDPRESS_SITE_DIR/wp-content/themes/$WORDPRESS_THEME_NAME"
    - scp -r -i $SSH_STAGING_PRIVATE_KEY ./wp-content/themes/$WORDPRESS_THEME_NAME $SSH_STAGING_USER@$SSH_STAGING_HOST:$WORDPRESS_SITE_DIR/wp-content/themes/

    # build theme frontend JS/CSS assets on destination server
    - ssh -i $SSH_STAGING_PRIVATE_KEY -o StrictHostKeyChecking=no $SSH_STAGING_USER@$SSH_STAGING_HOST "
        export PATH=$PATH:$NODE_BIN_PATH:$COMPOSER_BIN_PATH &&
        cd $WORDPRESS_SITE_DIR/wp-content/themes/$WORDPRESS_THEME_NAME && 
        composer install &&
        npm install"

    # enable deployed theme and plugins
    - ssh -i $SSH_STAGING_PRIVATE_KEY -o StrictHostKeyChecking=no $SSH_STAGING_USER@$SSH_STAGING_HOST "
        cd $WORDPRESS_SITE_DIR && 
        wp theme activate $WORDPRESS_THEME_NAME &&
        wp plugin activate $WORDPRESS_PLUGINS"

    # ensure WordPress and plugins have permissions on content/themes/plugin dirs
    - ssh -i $SSH_STAGING_PRIVATE_KEY -o StrictHostKeyChecking=no $SSH_STAGING_USER@$SSH_STAGING_HOST "
        chgrp -R www-data $WORDPRESS_SITE_DIR || true"

deploy-staging: 
    image: ubuntu:latest
    stage: deploy-staging
    rules:
        - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "staging"'
    before_script:
        - apt-get -yq update
        - apt-get -yqq install ssh
        - install -m 600 -D /dev/null ~/.ssh/id_rsa
        - echo "$SSH_STAGING_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - ssh-keyscan -p $SSH_PORT $SSH_STAGING_HOST > ~/.ssh/known_hosts
    script:        
        - ssh $SSH_STAGING_USER@$SSH_STAGING_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $STAGING_BRANCH && git pull && exit"
    # after_script:
    #     - rm -rf ~/.ssh

deploy-prod: 
    image: ubuntu:latest
    stage: deploy-prod
    rules:
        - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "production"'
    before_script:
        - apt-get -yq update
        - apt-get -yqq install ssh
        - install -m 600 -D /dev/null ~/.ssh/id_rsa
        - echo "$SSH_PROD_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - ssh-keyscan -p $SSH_PORT $SSH_PROD_HOST > ~/.ssh/known_hosts
    script:        
        - ssh $SSH_PROD_USER@$SSH_PROD_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $PROD_BRANCH && git pull && exit"
    # after_script:
    #     - rm -rf ~/.ssh

but would need to run things two times for child theme and parent theme. Perhaps for WordPress setup with base SFTP is easier… Wonder if others provision their full WordPress app with Gitlab or using other tools.

rhand · January 21, 2024, 4:23am

We have this setup now so simpler:

# GitLab CI Pipeline Script
# Summary of Variables:
# - SSH_STAGING_PRIVATE_KEY: Base64-encoded private key for staging environment
# - SSH_PROD_PRIVATE_KEY: Base64-encoded private key for production environment
# - SSH_STAGING_HOST: Hostname or IP address of the staging server
# - SSH_PROD_HOST: Hostname or IP address of the production server
# - SSH_PORT: SSH port for both staging and production servers
# - SSH_STAGING_USER: SSH username for the staging server
# - SSH_PROD_USER: SSH username for the production server
# - WORK_DIR: Directory on the server where the Git repository is located
# - STAGING_BRANCH: Branch to be deployed on the staging server
# - PROD_BRANCH: Branch to be deployed on the production server

stages:
  - deploy-staging
  - deploy-prod

deploy-staging: 
    image: ubuntu:latest
    stage: deploy-staging
    rules:
        - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "STAGING_BRANCH"'
    before_script:
        # Update and install necessary packages
        - apt-get -yq update
        - apt-get -yqq install ssh

        # Configure SSH key for staging environment
        - install -m 600 -D /dev/null ~/.ssh/id_rsa
        - echo "$SSH_STAGING_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - ssh-keyscan -p $SSH_PORT $SSH_STAGING_HOST > ~/.ssh/known_hosts
    script:        
        # Deploy to staging
        - ssh $SSH_STAGING_USER@$SSH_STAGING_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $STAGING_BRANCH && git pull && exit"
    after_script:
        # Clean up sensitive information from CI/CD environment
        - rm -rf ~/.ssh

deploy-prod: 
    image: ubuntu:latest
    stage: deploy-prod
    rules:
        - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "PROD_BRANCH"'
    before_script:
        # Update and install necessary packages
        - apt-get -yq update
        - apt-get -yqq install ssh

        # Configure SSH key for production environment
        - install -m 600 -D /dev/null ~/.ssh/id_rsa
        - echo "$SSH_PROD_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - ssh-keyscan -p $SSH_PORT $SSH_PROD_HOST > ~/.ssh/known_hosts
    script:        
        # Deploy to production
        - ssh $SSH_PROD_USER@$SSH_PROD_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $PROD_BRANCH && git pull && exit"
    after_script:
        # Clean up sensitive information from CI/CD environment
        - rm -rf ~/.ssh

but we are still thinking about adding PHP Composer so we can install the packages. Ubuntu image in use does not seem to have that so we could need a PHP image instead.

rhand · January 21, 2024, 8:57am

Tweked a bit

# GitLab CI Pipeline Script
# Summary of Variables:
# - SSH_STAGING_PRIVATE_KEY: Base64-encoded private key for staging environment
# - SSH_PROD_PRIVATE_KEY: Base64-encoded private key for production environment
# - SSH_STAGING_HOST: Hostname or IP address of the staging server
# - SSH_PROD_HOST: Hostname or IP address of the production server
# - SSH_PORT: SSH port for both staging and production servers
# - SSH_STAGING_USER: SSH username for the staging server
# - SSH_PROD_USER: SSH username for the production server
# - WORK_DIR: Directory on the server where the Git repository is located
# - STAGING_BRANCH: Branch to be deployed on the staging server
# - PROD_BRANCH: Branch to be deployed on the production server

stages:
  - deploy-staging
  - deploy-prod

deploy-staging: 
    image: ubuntu:latest
    stage: deploy-staging
    rules:
        - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == $STAGING_BRANCH'
        - if: '$CI_COMMIT_BRANCH == $STAGING_BRANCH'
            when: manual
    before_script:
        # Update and install necessary packages
        - apt-get -yq update
        - apt-get -yqq install ssh

        # Configure SSH key for staging environment
        - install -m 600 -D /dev/null ~/.ssh/id_rsa
        - echo "$SSH_STAGING_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - ssh-keyscan -p $SSH_PORT $SSH_STAGING_HOST > ~/.ssh/known_hosts
    script:        
        # Deploy to staging
        - ssh $SSH_STAGING_USER@$SSH_STAGING_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $STAGING_BRANCH && git pull && exit"
    after_script:
        # Clean up sensitive information from CI/CD environment
        - rm -rf ~/.ssh

deploy-prod: 
    image: ubuntu:latest
    stage: deploy-prod
    rules:
        - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == $PROD_BRANCH'
    before_script:
        # Update and install necessary packages
        - apt-get -yq update
        - apt-get -yqq install ssh

        # Configure SSH key for production environment
        - install -m 600 -D /dev/null ~/.ssh/id_rsa
        - echo "$SSH_PROD_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - ssh-keyscan -p $SSH_PORT $SSH_PROD_HOST > ~/.ssh/known_hosts
    script:        
        # Deploy to production
        - ssh $SSH_PROD_USER@$SSH_PROD_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $PROD_BRANCH && git pull && exit"
    after_script:
        # Clean up sensitive information from CI/CD environment
        - rm -rf ~/.ssh

But I am getting

`.gitlab-ci.yml`: (): did not find expected key while parsing a block mapping at line 23 column 11

referring to this line

- if: '$CI_COMMIT_BRANCH == $STAGING_BRANCH'

Any ideas why?

rhand · January 22, 2024, 7:44am

Ran last version

# GitLab CI Pipeline Script
# Summary of Variables:
# - SSH_STAGING_PRIVATE_KEY: Base64-encoded private key for staging environment
# - SSH_PROD_PRIVATE_KEY: Base64-encoded private key for production environment
# - SSH_STAGING_HOST: Hostname or IP address of the staging server
# - SSH_PROD_HOST: Hostname or IP address of the production server
# - SSH_PORT: SSH port for both staging and production servers
# - SSH_STAGING_USER: SSH username for the staging server
# - SSH_PROD_USER: SSH username for the production server
# - WORK_DIR: Directory on the server where the Git repository is located
# - STAGING_BRANCH: Branch to be deployed on the staging server
# - PROD_BRANCH: Branch to be deployed on the production server

stages:
  - deploy-staging
  - deploy-prod

deploy-staging: 
    image: ubuntu:latest
    stage: deploy-staging
    rules:
        - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == $STAGING_BRANCH'
        - if: '$CI_COMMIT_BRANCH == $STAGING_BRANCH'
          when: manual
    before_script:
        # Update and install necessary packages
        - apt-get -yq update
        - apt-get -yqq install ssh

        # Configure SSH key for staging environment
        - install -m 600 -D /dev/null ~/.ssh/id_rsa
        - echo "$SSH_STAGING_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - ssh-keyscan -p $SSH_PORT $SSH_STAGING_HOST > ~/.ssh/known_hosts
    script:        
        # Deploy to staging
        - ssh $SSH_STAGING_USER@$SSH_STAGING_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $STAGING_BRANCH && git pull && exit"
    after_script:
        # Clean up sensitive information from CI/CD environment
        - rm -rf ~/.ssh

# deploy-prod: 
#     image: ubuntu:latest
#     stage: deploy-prod
#     rules:
#         - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == $PROD_BRANCH'
#         - if: '$CI_COMMIT_BRANCH == $PROD_BRANCH'
#           when: manual
#     before_script:
#         # Update and install necessary packages
#         - apt-get -yq update
#         - apt-get -yqq install ssh

#         # Configure SSH key for production environment
#         - install -m 600 -D /dev/null ~/.ssh/id_rsa
#         - echo "$SSH_PROD_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
#         - chmod 600 ~/.ssh/id_rsa
#         - ssh-keyscan -p $SSH_PORT $SSH_PROD_HOST > ~/.ssh/known_hosts
#     script:        
#         # Deploy to production
#         - ssh $SSH_PROD_USER@$SSH_PROD_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $PROD_BRANCH && git pull && exit"
#     after_script:
#         # Clean up sensitive information from CI/CD environment
#         - rm -rf ~/.ssh

and got

$ chmod 600 ~/.ssh/id_rsa
$ ssh-keyscan -p $SSH_PORT $SSH_PROD_HOST > ~/.ssh/known_hosts
usage: ssh-keyscan [-46cDHv] [-f file] [-p port] [-T timeout] [-t type]
		   [host | addrlist namelist]
Running after_script
Running after script...
$ rm -rf ~/.ssh
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1

That second to last line (Cleaning up project directory and file based variables ) is always present in a CI/CD job, pass or fail.

source: python - Gitlab CI/CD fails while "Cleaning up project directory and file based variables" with "ERROR: Job failed: exit code 1" - Stack Overflow

so it must be another issue

rhand · January 23, 2024, 3:55am

Removed the cleaning up of ssh keys part:

# GitLab CI Pipeline Script
# Summary of Variables:
# - SSH_STAGING_PRIVATE_KEY: Base64-encoded private key for staging environment
# - SSH_PROD_PRIVATE_KEY: Base64-encoded private key for production environment
# - SSH_STAGING_HOST: Hostname or IP address of the staging server
# - SSH_PROD_HOST: Hostname or IP address of the production server
# - SSH_PORT: SSH port for both staging and production servers
# - SSH_STAGING_USER: SSH username for the staging server
# - SSH_PROD_USER: SSH username for the production server
# - WORK_DIR: Directory on the server where the Git repository is located
# - STAGING_BRANCH: Branch to be deployed on the staging server
# - PROD_BRANCH: Branch to be deployed on the production server

stages:
  - deploy-staging
  - deploy-prod

deploy-staging: 
    image: ubuntu:latest
    stage: deploy-staging
    rules:
        - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == $STAGING_BRANCH'
        - if: '$CI_COMMIT_BRANCH == $STAGING_BRANCH'
          when: manual
    before_script:
        # Update and install necessary packages
        - apt-get -yq update
        - apt-get -yqq install ssh

        # Configure SSH key for staging environment
        - install -m 600 -D /dev/null ~/.ssh/id_rsa
        - echo "$SSH_STAGING_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - ssh-keyscan -p $SSH_PORT $SSH_STAGING_HOST > ~/.ssh/known_hosts
    script:        
        # Deploy to staging
        - ssh $SSH_STAGING_USER@$SSH_STAGING_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $STAGING_BRANCH && git pull && exit"
    # after_script:
    #     # Clean up sensitive information from CI/CD environment
    #     - rm -rf ~/.ssh

# deploy-prod: 
#     image: ubuntu:latest
#     stage: deploy-prod
#     rules:
#         - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == $PROD_BRANCH'
#         - if: '$CI_COMMIT_BRANCH == $PROD_BRANCH'
#           when: manual
#     before_script:
#         # Update and install necessary packages
#         - apt-get -yq update
#         - apt-get -yqq install ssh

#         # Configure SSH key for production environment
#         - install -m 600 -D /dev/null ~/.ssh/id_rsa
#         - echo "$SSH_PROD_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
#         - chmod 600 ~/.ssh/id_rsa
#         - ssh-keyscan -p $SSH_PORT $SSH_PROD_HOST > ~/.ssh/known_hosts
#     script:        
#         # Deploy to production
#         - ssh $SSH_PROD_USER@$SSH_PROD_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $PROD_BRANCH && git pull && exit"
#     after_script:
#         # Clean up sensitive information from CI/CD environment
#         - rm -rf ~/.ssh

ran production by accident but failed because we have not set that up properly. Then ran deploy-staging and it failed still

e[0KRunning with gitlab-runner 16.6.0~beta.105.gd2263193 (d2263193)e[0;m
e[0K  on blue-3.saas-linux-small-amd64.runners-manager.gitlab.com/default zxwgkjAP, system ID: s_d5d3abbdfd0ae[0;m
e[0K  feature flags: FF_USE_IMPROVED_URL_MASKING:truee[0;m
section_start:1705981896:prepare_executor
e[0Ke[0Ke[36;1mPreparing the "docker+machine" executore[0;me[0;m
e[0KUsing Docker executor with image kroniak/ssh-client ...e[0;m
e[0KPulling docker image kroniak/ssh-client ...e[0;m
e[0KUsing docker image sha256:590358fd69fc516fb1a3929fbb8c2678b9249cf01e44e01edf3b90600678281f for kroniak/ssh-client with digest kroniak/ssh-client@sha256:df1fe1c506eb30c8bf37f080197c326ed99cacba5126fcfa6e8ba82ae71e67fb ...e[0;m
section_end:1705981901:prepare_executor
e[0Ksection_start:1705981901:prepare_script
e[0Ke[0Ke[36;1mPreparing environmente[0;me[0;m
Running on runner-zxwgkjap-project-27991478-concurrent-0 via runner-zxwgkjap-s-l-s-amd64-1705981825-09ff1924...
section_end:1705981902:prepare_script
e[0Ksection_start:1705981902:get_sources
e[0Ke[0Ke[36;1mGetting source from Git repositorye[0;me[0;m
e[32;1mFetching changes with git depth set to 50...e[0;m
Initialized empty Git repository in /builds/user/site/.git/
e[32;1mCreated fresh repository.e[0;m
e[32;1mChecking out 10244c35 as detached HEAD (ref is gitlab-pipeline)...e[0;m

e[32;1mSkipping Git submodules setupe[0;m
e[32;1m$ git remote set-url origin "${CI_REPOSITORY_URL}"e[0;m
section_end:1705981905:get_sources
e[0Ksection_start:1705981905:step_script
e[0Ke[0Ke[36;1mExecuting "step_script" stage of the job scripte[0;me[0;m
e[0KUsing docker image sha256:590358fd69fc516fb1a3929fbb8c2678b9249cf01e44e01edf3b90600678281f for kroniak/ssh-client with digest kroniak/ssh-client@sha256:df1fe1c506eb30c8bf37f080197c326ed99cacba5126fcfa6e8ba82ae71e67fb ...e[0;m
e[32;1m$ chmod og= $STAGE_ID_RSAe[0;m
BusyBox v1.36.1 (2023-07-27 17:12:24 UTC) multi-call binary.

Usage: chmod [-Rcvf] MODE[,MODE]... FILE...

MODE is octal number (bit pattern sstrwxrwxrwx) or [ugoa]{+|-|=}[rwxXst]

	-R	Recurse
	-c	List changed files
	-v	Verbose
	-f	Hide errors
section_end:1705981905:step_script
e[0Ksection_start:1705981905:cleanup_file_variables
e[0Ke[0Ke[36;1mCleaning up project directory and file based variablese[0;me[0;m
section_end:1705981906:cleanup_file_variables
e[0Ke[31;1mERROR: Job failed: exit code 1
e[0;m

Do wonder if rerunning a pipeline job does just run the old version. I do want to load the latest from our pipeline branch for testing purposes. And last error it seem $ chmod og= $STAGE_ID_RSA so old variable was being used.

How can I enforce manual run on pipeline branch of latest .gitlab-ci.yml ?

rhand · January 27, 2024, 10:22am

I ran the test again and added after_script with echoing of some variables to see if all would load but I get this error

Running with gitlab-runner 16.6.0~beta.105.gd2263193 (d2263193)
  on blue-1.saas-linux-small.runners-manager.gitlab.com/default j1aLDqxS, system ID: s_ccdc2f364be8
  feature flags: FF_USE_IMPROVED_URL_MASKING:true

Preparing the "docker+machine" executor
00:05
Using Docker executor with image kroniak/ssh-client ...
Pulling docker image kroniak/ssh-client ...
Using docker image sha256:590358fd69fc516fb1a3929fbb8c2678b9249cf01e44e01edf3b90600678281f for kroniak/ssh-client with digest kroniak/ssh-client@sha256:df1fe1c506eb30c8bf37f080197c326ed99cacba5126fcfa6e8ba82ae71e67fb ...

Preparing environment
00:00
Running on runner-j1aldqxs-project-27991478-concurrent-0 via runner-j1aldqxs-s-l-s-amd64-1706350794-118923fc...

Getting source from Git repository
00:03
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/user/site/.git/
Created fresh repository.
Checking out 10244c35 as detached HEAD (ref is gitlab-pipeline)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}"

Executing "step_script" stage of the job script
00:01
Using docker image sha256:590358fd69fc516fb1a3929fbb8c2678b9249cf01e44e01edf3b90600678281f for kroniak/ssh-client with digest kroniak/ssh-client@sha256:df1fe1c506eb30c8bf37f080197c326ed99cacba5126fcfa6e8ba82ae71e67fb ...
$ chmod og= $STAGE_ID_RSA
BusyBox v1.36.1 (2023-07-27 17:12:24 UTC) multi-call binary.
Usage: chmod [-Rcvf] MODE[,MODE]... FILE...
MODE is octal number (bit pattern sstrwxrwxrwx) or [ugoa]{+|-|=}[rwxXst]
	-R	Recurse
	-c	List changed files
	-v	Verbose
	-f	Hide errors

Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

We do not use $STAGE_ID_RSA so why do I see it pass by? Is that because of the Docker image?

we use

# - SSH_STAGING_PRIVATE_KEY: Base64-encoded private key for staging environment
# - SSH_PROD_PRIVATE_KEY: Base64-encoded private key for production environment
# - SSH_STAGING_HOST: Hostname or IP address of the staging server
# - SSH_PROD_HOST: Hostname or IP address of the production server
# - SSH_PORT: SSH port for both staging and production servers
# - SSH_STAGING_USER: SSH username for the staging server
# - SSH_PROD_USER: SSH username for the production server
# - WORK_DIR: Directory on the server where the Git repository is located
# - STAGING_BRANCH: Branch to be deployed on the staging server
# - PROD_BRANCH: Branch to be deployed on the production server

and we no longer use image kroniak/ssh-client

How can I manually use the current image ubuntu as in

# GitLab CI Pipeline Script
# Summary of Variables:
# - SSH_STAGING_PRIVATE_KEY: Base64-encoded private key for staging environment
# - SSH_PROD_PRIVATE_KEY: Base64-encoded private key for production environment
# - SSH_STAGING_HOST: Hostname or IP address of the staging server
# - SSH_PROD_HOST: Hostname or IP address of the production server
# - SSH_PORT: SSH port for both staging and production servers
# - SSH_STAGING_USER: SSH username for the staging server
# - SSH_PROD_USER: SSH username for the production server
# - WORK_DIR: Directory on the server where the Git repository is located
# - STAGING_BRANCH: Branch to be deployed on the staging server
# - PROD_BRANCH: Branch to be deployed on the production server

stages:
  - deploy-staging
  - deploy-prod

deploy-staging: 
    image: ubuntu:latest
    stage: deploy-staging
    rules:
        - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == $STAGING_BRANCH'
        - if: '$CI_COMMIT_BRANCH == $STAGING_BRANCH'
          when: manual
    before_script:
        # Update and install necessary packages
        - apt-get -yq update
        - apt-get -yqq install ssh

        # Configure SSH key for staging environment
        - install -m 600 -D /dev/null ~/.ssh/id_rsa
        - echo "$SSH_STAGING_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - ssh-keyscan -p $SSH_PORT $SSH_STAGING_HOST > ~/.ssh/known_hosts
    script:        
        # Deploy to staging
        - ssh $SSH_STAGING_USER@$SSH_STAGING_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $STAGING_BRANCH && git pull && exit"
    after_script:
        # Debugging information
        - 'echo "CI_COMMIT_BRANCH: $CI_COMMIT_BRANCH"'
        - 'echo "STAGING_BRANCH: $STAGING_BRANCH"'
        - 'echo "SSH_STAGING_HOST: $SSH_STAGING_HOST"'
        - 'echo "SSH_PORT: $SSH_PORT"'
        - 'echo "WORK_DIR: $WORK_DIR"'
    #     # Clean up sensitive information from CI/CD environment
    #     - rm -rf ~/.ssh

# deploy-prod: 
#     image: ubuntu:latest
#     stage: deploy-prod
#     rules:
#         - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == $PROD_BRANCH'
#         - if: '$CI_COMMIT_BRANCH == $PROD_BRANCH'
#           when: manual
#     before_script:
#         # Update and install necessary packages
#         - apt-get -yq update
#         - apt-get -yqq install ssh

#         # Configure SSH key for production environment
#         - install -m 600 -D /dev/null ~/.ssh/id_rsa
#         - echo "$SSH_PROD_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
#         - chmod 600 ~/.ssh/id_rsa
#         - ssh-keyscan -p $SSH_PORT $SSH_PROD_HOST > ~/.ssh/known_hosts
#     script:        
#         # Deploy to production
#         - ssh $SSH_PROD_USER@$SSH_PROD_HOST -p$SSH_PORT "cd $WORK_DIR && git checkout $PROD_BRANCH && git pull && exit"
#     after_script:
#         # Clean up sensitive information from CI/CD environment
#         - rm -rf ~/.ssh

rhand · January 28, 2024, 2:50am

Artifacts can be removed but then old pipeline and jobs are still run. Seems I need to run at https://gitlab.com/user/site/-/pipelines/new to run a new manual test and that I can add variables if need be. But may do this tomorrow.

rhand · January 31, 2024, 9:43am

Main issue in the end was that we needed to change ssh user rights for user deployment , had to adjust directory and file permissions and had to run a clean pipeline . All working now.