Bulk migrate projects from one self hosted instance to another

Workflow: bulk export >> transfer >> bulk import all the projects.

Requirements:

• ssh access to both source and destination GitLab servers.
• Source GitLab server has ability to ssh to the destination GitLab server.

Grant SSH access from source GitLab server to the destination GitLab server:

• Access GitLab source server and check a directory for the ssh key ls ~/.ssh, it should contain a file with extension *.pub if this file does not exist generate SSH hey in the source GitLab server by running ssh-keygen -t rsa

• Copy ssh key from cat ~/.ssh/id_rsa.pub

• Access GitLab destination server and add copied ssh key to the following file vi ~/.ssh/authorized_keys

• Validate ssh access from the source GitLab server to the destination GitLab server ssh root@<your_server_host>

Export Projects

• Access source GitLab server’s rails console sudo gitlab-rails console and run the below code snippet, be sure to change values SERVER, KEY_FILE, SERVER_USERNAME to your own:

# Export

SERVER = <your_server_host>
KEY_FILE = <your_ssh_keyfile_path>
SERVER_USERNAME = <username> 
jobs = []
user = User.find_by_username('your_username')
projects = Project.all

projects = Project.all

projects.each do |p|
    # p.add_export_job(current_user: user)
    job_id = ProjectExportWorker.perform_async(user.id, p.id, {}, {})
    jobs << job_id
 end

def check_job_status(ids)
    if Gitlab::SidekiqStatus.running?(ids)
        wait_and_check_again(ids)
    end
end

def wait_and_check_again(ids)
    puts "checking jobs status"
    sleep(60)
    check_job_status(ids)
end

check_job_status(jobs)

exec("ssh #{SERVER_USERNAME}@#{SERVER} mkdir -p /tmp/exports && scp -pr /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/import_export_upload/export_file/* #{SERVER_USERNAME}@#{SERVER}:/tmp/exports")

• This should have exported all the projects to the following destination path /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/import_export_upload/export_file/
• It should have transfered all the files to the destination GitLab server location /tmp/exports

Import Projects

• Access destination GitLab server’s rails console sudo gitlab-rails console and run the below code snippet, be sure to change value user to your own:

To import projects by entering names on project import:

path_string = ''
user = User.find_by_username('your_username')
file_paths = Dir.glob('/tmp/exports/*').select { |fn| File.directory?(fn) }


file_paths.each do |path|
    path_string = Dir.glob(path + '/*')[0]
    puts "Enter project name for #{path_string} : "
    project_name = gets.chomp
    project_params = {
        :name => project_name, 
        :path => '',
        :namespace_id => user.namespace_id,
        :file => File.open(path_string)
    }
    @project = ::Projects::GitlabProjectsImportService.new(user, project_params).execute
    puts @project.name
end

To Import projects which name will be taken from the exported file name. Please note if project was nested in groups the new project name will not be correct:

# Import

path_string = ''
user = User.find_by_username('your_username')
file_paths = Dir.glob('/tmp/exports/*').select { |fn| File.directory?(fn) }

def format_name(name)
   final_name = ''
   char_array = name.scan /\w/
   name_array = ((char_array.drop(17))[0..-13]).join('').split("_")
   name_array.delete_at(0)
   name_array.each do |n|
      final_name += n 
      final_name += " "
   end 
   return final_name
end

file_paths.each do |path|
    path_string = Dir.glob(path + '/*')[0]
    project_params = {
        :name => format_name(path_string.split('/')[4]),
        :path => '',
        :namespace_id => user.namespace_id,
        :file => File.open(path_string)
    }
    @project = ::Projects::GitlabProjectsImportService.new(user, project_params).execute
    puts @project.name
end

• This will import all the projects in the /tmp/exports/ directory

Troubleshooting SSH access from source GitLab server to the destination GitLab server.

• In case ssh access does not work, check whether you have to add ssh key to GitLab specific folders /var/opt/gitlab/.ssh/ and /var/opt/gitlab/.ssh/authorized_keys

Hi,

I exported a couple of Projects on one server from the UI, and then scp’d the resulting export files over to the target GITLAB instance. As such, I skipped the step that creates the ‘export’ rails script and just created the ‘Import’ script from your example above. I used my username and a different exports directory in the script too. I also replaced the ‘Enter Project name for’ reference with the Project name to which I am importing the ‘export’ files. On the Destination instance, I created a new Group and subgroup.

I’m using the command “gitlab-rails runner /local/mnt/workspace/import_project.rb” to import the Projects into the new Group on the Destination instance. The script doesn’t throw any errors and I do see some activity in the ‘production.log’ as the script is running. However when the script finishes, I don’t see the Projects I’m trying to import on the Destination instance in the new subgroup from the UI.

Knowing very little about Ruby, would you be able to provide any insight as to what I’m doing wrong?

Thank you,

Jon

Hey,

What version of GitLab are you running?

Hi,

I’m using v12.8.1-ee.

Thanks,

Jon

So, I think this might be outdated can you try the below for the import:

user = User.find_by_username('your_username')
file_paths = Dir['/tmp/exports/*']

file_paths.each do |path|
    puts "Enter project name for #{path_string} : "
    project_name = gets.chomp
    project_params = {
        :name => project_name, 
        :path => '',
        :namespace_id => user.namespace_id,
        :file => File.open(path)
    }
    @project = ::Projects::GitlabProjectsImportService.new(user, project_params).execute
    puts @project.name
end

Can this be put in gitlab-ctl CLI so that it will be easier to do bulk import and export between instances?