Reducing Repo Size from 40GB

Hi
We’re currently migrating from GitLab self-managed to SaaS.
One of our project’s repositories is way too large for SaaS and I’m trying to figure out what to do.

Current Export Size: 40.2 GB
Repo Details:

image733×39 2.18 KB

I used git-sizer to analyze the repo size, and these are the results:

Processing blobs: 20685                        
Processing trees: 45556                        
Processing commits: 9403                        
Matching commits to trees: 9403                        
Processing annotated tags: 0                        
Processing references: 2869                        
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Biggest objects              |           |                                |
| * Trees                      |           |                                |
|   * Maximum entries      [1] |  1.66 k   | *                              |
| * Blobs                      |           |                                |
|   * Maximum size         [2] |   277 MiB | ****************************   |
|                              |           |                                |
| Biggest checkouts            |           |                                |
| * Maximum path depth     [3] |    13     | *                              |
| * Maximum path length    [4] |   151 B   | *                              |
| * Total size of files    [5] |  1.27 GiB | *                              |

[1]  e46316074e309c62b2e358b6b69ea7cc534c482f (refs/heads/***:***/Scripts/kendo/cultures)
[2]  b8348a5487a30d4ae121c4d4fedf1bc0413b084a (refs/heads/***:***_180103.vsp)
[3]  a35b40788de3e039b0709fcec5d09c3a24281946 (65e22de516f1d6cc5976040dcdde75157c25a3cc^{tree})
[4]  00eac94e70e4a2cabf67d93d189996c3c9dc8cf9 (831051b157da19e08dd867b774df387c9948d583^{tree})
[5]  a4b0c77facb0d611f79d427a6127f405f4e76808 (2567bfd7be843d0e8fee04487239189e5fe13440^{tree})

When I run ./git-sizer –verbose ig get:

Processing blobs: 20685                        
Processing trees: 45556                        
Processing commits: 9403                        
Matching commits to trees: 9403                        
Processing annotated tags: 0                        
Processing references: 2869                        
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Overall repository size      |           |                                |
| * Commits                    |           |                                |
|   * Count                    |  9.40 k   |                                |
|   * Total size               |  2.72 MiB |                                |
| * Trees                      |           |                                |
|   * Count                    |  45.6 k   |                                |
|   * Total size               |  70.1 MiB |                                |
|   * Total tree entries       |  1.71 M   |                                |
| * Blobs                      |           |                                |
|   * Count                    |  20.7 k   |                                |
|   * Total size               |  1.48 GiB |                                |
| * Annotated tags             |           |                                |
|   * Count                    |     0     |                                |
| * References                 |           |                                |
|   * Count                    |  2.87 k   |                                |
|                              |           |                                |
| Biggest objects              |           |                                |
| * Commits                    |           |                                |
|   * Maximum size         [1] |  1.36 KiB |                                |
|   * Maximum parents      [2] |     2     |                                |
| * Trees                      |           |                                |
|   * Maximum entries      [3] |  1.66 k   | *                              |
| * Blobs                      |           |                                |
|   * Maximum size         [4] |   277 MiB | ****************************   |
|                              |           |                                |
| History structure            |           |                                |
| * Maximum history depth      |  2.98 k   |                                |
| * Maximum tag depth          |     0     |                                |
|                              |           |                                |
| Biggest checkouts            |           |                                |
| * Number of directories  [5] |  1.31 k   |                                |
| * Maximum path depth     [5] |    13     | *                              |
| * Maximum path length    [6] |   151 B   | *                              |
| * Number of files        [7] |  14.9 k   |                                |
| * Total size of files    [8] |  1.27 GiB | *                              |
| * Number of symlinks         |     0     |                                |
| * Number of submodules       |     0     |                                |

[1]  4ff9933bd48cc3fc1bf9a65dfa973b438d8ee8b1
[2]  b7b785be52148515c1757fcec78aeefcda56269e (refs/merge-requests/625/merge)
[3]  e46316074e309c62b2e358b6b69ea7cc534c482f (refs/heads/***:***/Scripts/kendo/cultures)
[4]  b8348a5487a30d4ae121c4d4fedf1bc0413b084a (refs/heads/***:***_180103.vsp)
[5]  a35b40788de3e039b0709fcec5d09c3a24281946 (65e22de516f1d6cc5976040dcdde75157c25a3cc^{tree})
[6]  00eac94e70e4a2cabf67d93d189996c3c9dc8cf9 (831051b157da19e08dd867b774df387c9948d583^{tree})
[7]  f7616562b2592e9bec58e1dd2814959cbbed7822 (63384b14246d695457d01f3917975d95b80ff7f9^{tree})
[8]  a4b0c77facb0d611f79d427a6127f405f4e76808 (2567bfd7be843d0e8fee04487239189e5fe13440^{tree})

: is a branch name + file path, the same in all of the above.

I’m not sure what to do at this point.
My options are:

1. reducing this repository size somehow.
2. convert all 3rd party scripts to sub-modules. (If you think this is the best options and there is a guide please send it)

What should I do next?

Thanks!

Hi there
How are you migrating to saas?
1 - direct group transfer?
2 - project export / import?
3 - congregate?
4 - what version is your self hosted?
5 - where are you seeing 40gb?

I’m doing project import/export., and the 40GB was the export size of a single repository (1 project)

I’ve just found a solution:
Apparently, each one of the 116 Tags held 2 sets of “zip” files with our builds, and removing the references didn’t help, so I had to locate them on the self-managed server and delete those files. This cleared 38GB off of the repository.

Glad to hear it’s solved
Also worth trying DGT, it’s better than project export / import. It’d probably drop those files as well in the background. DGT can be CPU hungry, just an fyi, if you ever try it and it fails then just try again

Thanks you!!

I’m actually not familiar with DGT, what is this?

It’s another way of migrating projects from one gitlab instance to another, very easy to use, no need to download a project and then upload it, saves time.
you can create a temp group, move your project in there, migrate that group to another gitlab, job done. then move the project back to where it was. more info on DGT is here

Thanks!

This wouldn’t fix the issue but it might be a simpler way