Very slow ssh copy (scp) during CI

Final stage of my CD pipline is deploy to dev server. Now it is done by copying files over ssh. Overall size of files is approximately 45 Mb, ~ 2.8k files. When I copy thne from my computer (50Mbps internet connetction) it takes about 2 minutes to copy them all. But when they copied from runner it takes more than 1h and I get timeout error. Here is my .gitlab-ci.yml:

  - deploy

.deploy_template: &deploy_tpl
  stage: deploy
  allow_failure: false

.dev_deploy_template: &dev_deploy_tpl
  <<: *deploy_tpl
    DEPLOY_HOST: myhost
    DEPLOY_USER: myuser
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - eval $(ssh-agent -s)
    - echo "$DEPLOY_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - ssh-keyscan $DEPLOY_HOST >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
    - develop

api to dev:
  <<: *dev_deploy_tpl
  image: php:7.1
  environment: api dev
    untracked: true
      - api/vendor
      - ~/.composer
    - apt-get install -y pkg-config git unzip
    - pecl install mongodb
    - docker-php-ext-enable mongodb
    - php -r "copy('', 'composer-setup.php');"
    - php composer-setup.php --install-dir=/usr/bin --filename=composer
    - rm -f ./composer-setup.php
    - chmod +x /usr/bin/composer
    - cd api/
    - composer install
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "mkdir /var/www/api/$CI_COMMIT_SHA"
    - scp -r -C ./* $DEPLOY_USER@$DEPLOY_HOST:/var/www/api/$CI_COMMIT_SHA
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "ln -s /var/www/shared/.env /var/www/api/$CI_COMMIT_SHA/.env"
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "ln -s /var/www/shared/runtime /var/www/api/$CI_COMMIT_SHA/runtime"
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "unlink /var/www/api/current && ln -s /var/www/api/$CI_COMMIT_SHA /var/www/api/current"
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "/var/www/api/current/vendor/bin/phinx migrate"

front to dev:
  <<: *dev_deploy_tpl
  image: node:10
  environment: front dev
    untracked: true
      - app/node_modules
      - ~/.node-gyp
      - ~/.npm
    - cd app/
    - npm install --no-audit
    - npm run build
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "mkdir /var/www/html/$CI_COMMIT_SHA"
    - scp -r -C ./* $DEPLOY_USER@$DEPLOY_HOST:/var/www/html/$CI_COMMIT_SHA
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "unlink /var/www/html/current && ln -s /var/www/html/$CI_COMMIT_SHA /var/www/html/current"

chat server to dev:
  <<: *dev_deploy_tpl
  image: node:10
  environment: chat dev
    untracked: true
      - chat-server/node_modules
      - ~/.node-gyp
      - ~/.npm
    - cd chat-server/
    - npm install --no-audit
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "rm -rf /opt/chat-server/*"
    - scp -r -C ./* $DEPLOY_USER@$DEPLOY_HOST:/opt/chat-server/
    - ssh $DEPLOY_USER@$DEPLOY_HOST -C "rm -f /opt/chat-server/server.json && ln -s /opt/shared/server.json /opt/chat-server/server.json 2>&1 > /dev/null"

I have a similar speed issue when pulling from the registry. I had hoped that gitlab’s migration to Google cloud last weekend would have helped, but no luck.

Many years later on this post. I am experiencing same issue. It seems that the recursive scp is causing the issue, I tried single scp commands and performance is not affected. Wondering if the -r is causing extra latency for each files.

Maybe use rsync over ssh instead of scp? So using the example line from the earlier posts:

scp -r -C ./* $DEPLOY_USER@$DEPLOY_HOST:/var/www/api/$CI_COMMIT_SHA

instead of that:

rsync ./* $DEPLOY_USER@$DEPLOY_HOST:/var/www/api/$CI_COMMIT_SHA