Very slow ssh copy (scp) during CI

GitLab CI/CD
1

Hi!
Final stage of my CD pipline is deploy to dev server. Now it is done by copying files over ssh. Overall size of files is approximately 45 Mb, ~ 2.8k files. When I copy thne from my computer (50Mbps internet connetction) it takes about 2 minutes to copy them all. But when they copied from gitlab.com runner it takes more than 1h and I get timeout error. Here is my .gitlab-ci.yml:

stages:
  - deploy

.deploy_template: &deploy_tpl
  stage: deploy
  allow_failure: false

.dev_deploy_template: &dev_deploy_tpl
  <<: *deploy_tpl
  variables:
    DEPLOY_HOST: myhost
    DEPLOY_USER: myuser
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - eval $(ssh-agent -s)
    - echo "$DEPLOY_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - ssh-keyscan $DEPLOY_HOST >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
  only:
    - develop

api to dev:
  <<: *dev_deploy_tpl
  image: php:7.1
  environment: api dev
  cache:
    untracked: true
    paths:
      - api/vendor
      - ~/.composer
  script:
    - apt-get install -y pkg-config git unzip
    - pecl install mongodb
    - docker-php-ext-enable mongodb
    - php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
    - php composer-setup.php --install-dir=/usr/bin --filename=composer
    - rm -f ./composer-setup.php
    - chmod +x /usr/bin/composer
    - cd api/
    - composer install
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "mkdir /var/www/api/$CI_COMMIT_SHA"
    - scp -r -C ./* $DEPLOY_USER@$DEPLOY_HOST:/var/www/api/$CI_COMMIT_SHA
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "ln -s /var/www/shared/.env /var/www/api/$CI_COMMIT_SHA/.env"
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "ln -s /var/www/shared/runtime /var/www/api/$CI_COMMIT_SHA/runtime"
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "unlink /var/www/api/current && ln -s /var/www/api/$CI_COMMIT_SHA /var/www/api/current"
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "/var/www/api/current/vendor/bin/phinx migrate"

front to dev:
  <<: *dev_deploy_tpl
  image: node:10
  environment: front dev
  cache:
    untracked: true
    paths:
      - app/node_modules
      - ~/.node-gyp
      - ~/.npm
  script:
    - cd app/
    - npm install --no-audit
    - npm run build
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "mkdir /var/www/html/$CI_COMMIT_SHA"
    - scp -r -C ./* $DEPLOY_USER@$DEPLOY_HOST:/var/www/html/$CI_COMMIT_SHA
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "unlink /var/www/html/current && ln -s /var/www/html/$CI_COMMIT_SHA /var/www/html/current"

chat server to dev:
  <<: *dev_deploy_tpl
  image: node:10
  environment: chat dev
  cache:
    untracked: true
    paths:
      - chat-server/node_modules
      - ~/.node-gyp
      - ~/.npm
  script:
    - cd chat-server/
    - npm install --no-audit
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "rm -rf /opt/chat-server/*"
    - scp -r -C ./* $DEPLOY_USER@$DEPLOY_HOST:/opt/chat-server/
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "rm -f /opt/chat-server/server.json && ln -s /opt/shared/server.json /opt/chat-server/server.json 2>&1 > /dev/null"
2

I have a similar speed issue when pulling from the registry. I had hoped that gitlab’s migration to Google cloud last weekend would have helped, but no luck.

3

Many years later on this post. I am experiencing same issue. It seems that the recursive scp is causing the issue, I tried single scp commands and performance is not affected. Wondering if the -r is causing extra latency for each files.

4

Maybe use rsync over ssh instead of scp? So using the example line from the earlier posts:

scp -r -C ./* $DEPLOY_USER@$DEPLOY_HOST:/var/www/api/$CI_COMMIT_SHA

instead of that:

rsync ./* $DEPLOY_USER@$DEPLOY_HOST:/var/www/api/$CI_COMMIT_SHA