I have set up Gitlab CE self hosted on my server. I wish to automatically mirror pull a public repo from github into my gitlab for archiving purposes.
Although when I go to Settings > Repository > Mirroring
the dropdown is stuck on “push” and I cant select pull.
I installed using docker-compose. Maybe there is an environment variable I need to enable to enable mirror pulling?
Hi,
that’s a feature available in the Starter/Bronze edition of GitLab EE.
https://docs.gitlab.com/ee/user/project/repository/repository_mirroring.html#pulling-from-a-remote-repository-starter
Cheers,
Michael
Wow, did not seem like an enterprise needed feature so I assumed it would be a CE one. Are there any non profit Starter or Bronze packages?
As I use Gitlab Selfhosted for my non profit open source work and would really like to start archiving old projects that were hosted on Github… this is a big issue for me and would be the deal breaker moving me back to finding another platform for my git home. 
Hi,
I don’t know about the specific reasons for this. Still, there’s always the chance to ask for moving specific EE features into Core. On gitlab.com that’s available atm.
With regard to open source projects, you can apply for getting a free license.
Cheers,
Michael
Reading that I see this:
I am not even sure what that is… I dont host my stuff on gitlab only my self hosted gitlab and I dont plan to host my projects in like 3 different places all at once as thatll jut be chaotic; I want to host it from my self hosted gitlab. Needing to have my repo on gitlab it slef too defeats the purpose to self host… I might as well ditch my git and use gitlabs at thaf rate.
Gitlab is not looking like something for little developers like me who like to self host & are barely known. Its sad.
Hi,
if the application cannot solve this, there’s another way to achieve this with native Git.
I had described this in this topic: Refreshing a Fork (the CLI ways).
A small cronjob on the CLI with that script should also work. I have been using this method for quite a while to sync everything between GitHub, GitLab, SVN repos and others.
Here’s an example we used in the past to sync from GitLab to GitHub (the other way around).
#!/bin/bash
REPO_HOME="/data/sync"
declare -A REPOS
# syntax is [github_repo]="gitlab_repo"
REPOS=([githubrepo]="gitlabgroup/gitlabrepo")
cd $REPO_HOME
for REPO_GITHUB in "${!REPOS[@]}"
do
REPO_LOCAL=${REPOS[$REPO_GITHUB]}
echo "### Processing repo $REPO_LOCAL"
if [ ! -d $REPO_LOCAL ]; then
git clone --bare --mirror https://gitlab.domain.org/$REPO_LOCAL.git $REPO_LOCAL
fi
(cd $REPO_LOCAL; git fetch --prune; git push --prune git@github.com:org/$REPO_GITHUB.git +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/*)
done
# su - git
$ crontab -e
# Sync repos to github
*/5 * * * * /data/sync/github/create_and_sync > /dev/null 2>&1
Cheers,
Michael
Thanks so much for the info. I will be using your CLI suggestion. Although if you dont mind Id like some help understanding how to turn it around to work with cloning a public github to my self hosted gitlab. From what I gathered it should be rather simple to change it and this is what I came up with:
#!/bin/bash
REPO_HOME="/data/sync"
declare -A REPOS
# syntax is [gitlab_repo]="gitlab_repo"
REPOS=([gitlabrepo]="gitlabgroup/gitlabrepo")
cd $REPO_HOME
for REPO_GITLAB in "${!REPOS[@]}"
do
REPO_LOCAL=${REPOS[$REPO_GITLAB]}
echo "### Processing repo $REPO_LOCAL"
if [ ! -d $REPO_LOCAL ]; then
git clone --bare --mirror https://github.com/$REPO_LOCAL.git $REPO_LOCAL
fi
(cd $REPO_LOCAL; git fetch --prune; git push --prune git@git.mydomain.com:org/$REPO_GITLAB.git +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/*)
done
I have a few questions though. The variable REPO_HOME is just a temp location, right? As Im thinking this will double my data stored as itll store it in my gitlab AND on the local file system in REPO_HOME? Would I be able to add a rm $REPO_LOCAL at the end of this script to clean up the data? As it would just recreate it during the sync anyways I dont think it would matter.
Here is an example I want to use, let me know if you think itll work:
#!/bin/bash
REPO_HOME="/data/sync"
declare -A REPOS
# syntax is [gitlab_repo]="gitlab_repo"
REPOS=([gitlabrepo]="callmekory/nezuko")
cd $REPO_HOME
for REPO_GITLAB in "${!REPOS[@]}"
do
REPO_LOCAL=${REPOS[$REPO_GITLAB]}
echo "### Processing repo $REPO_LOCAL"
if [ ! -d $REPO_LOCAL ]; then
git clone --bare --mirror https://github.com/$REPO_LOCAL.git $REPO_LOCAL
fi
(cd $REPO_LOCAL; git fetch --prune; git push --prune git@git.mydomain.com:org/$REPO_GITLAB.git +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/*)
rm $REPO_LOCAL
done
This should sync this repo: https://github.com/callmekory/nezuko
Into my gitlab at git.mydomain.com/callmekory/nezuko
If this is right then I for sure will be adjusting this to work with multiple repos, so i imagine adding repots to this since we have a for loop would be adding them here:
REPOS=([gitlabrepo]="callmekory/nezuko anothergroup/anotherrepo")
Ah, your repository groups and names are equal. We had the problem that it didn’t match for us, that’s why we created the REPOS key value pair.
To keep the code intact, just do it like this:
REPOS=(["callmekory/nezuko"]="callmekory/nezuko")
Also, rm $REPO_LOCAL won’t work, you’ll need to pass -r.
My Bash foo isn’t that accurate, I feel more familiar with other script languages. Though I thought I’d share it with you.
While looking into it again, ready-to-use scripts might be more applicable, the community is big enough 
This one looks tempting, also since it uses https with tokens as transport instead of SSH keys. Might be worth a try.
Cheers,
Michael
Ill give it a go latwr tonight and see if it works for me, if not Ill fall back onto the bash script and configure it to work for what I need. 
Hi @AnnaLisik,
there was a change for private repositories announced, please follow my comment here: Pull based Mirroring from GitHub for CI is no longer mirrored
Cheers,
Michael
@dnsmichi
Why cant we do it in self hosted gitlab for public repos too then?
I want to mirror FOSS from github into my self hosted gitlab so I can back them up incase they eventually disappear like Ive seen happen in the past.
I cant do this without custom scripts on the CLI level when its available on the GUI level for people who have the EE.
I understand why private repo mirroring may be for EE, but just like how its available on gitlab.com for public repos… why cant we do it for public repos in self hosted?
Id also like to mention that Github just recently dropped a lot of their pricing and made a lot of festures avilable for the free github over the enterprise.
I truthfully think Gitlab should rethink their enterprise structuring like Github did and realize that feaures like this is not an enterprise feature whereas like someone above mentioned that LDAP really is something enterprise would use… but for some reason is available on the community editiom but mirroring isnt? Bizzare
Hi,
Which self-hosted version of GitLab are you using? Please share the version string found at https://gitlab.server/help. If you are using the self-hosted Enterprise version, the pull option is available from Starter forward, as described in the docs.
Cheers,
Michael
@dnsmichi
I am using the free version. I am a student learning git and cant afford starter when I make no money from my projects.
Thats what Im talking about. I understand how itd be disabled for private repos like it is on gitlab.com but why is it totally disabled for free use?
I understand why Enterprise may need it for private repos and I fully support that like how it is on gitlab.com although you guys removed allowing to do it even on public repos in the free version. It really makes no sense how you guys think it is something free users can have on gitlab.com but not in the self hosted version.
Especially with how many users have already complained about how this feature makes no sense to be locked behind EE when theres more sense for it to be a free feature and theres other features in the free package that make more sense to be EE.
Hi,
I’m trying to help you here, just outlining the facts:
That being said, this feature never existed in the Core/CE version. Your request slightly differs from the original topic where this is headed towards GitLab.com.
Anyhow, since you are self-hosted, you also have self-control about the CLI and may install a cronjob. With the tools and scripts I had shared in this topic already, this should be doable on your own. Either in a cronjob or with leveraging CI scheduled jobs.
Cheers,
Michael
I appreciate the time youre taking to respond. Thanks.
Ill also break down what Im saying 
Now, what I am trying to point out is that I dont understand why this feature is enabled on gitlab.com for free users to be able to pull mirror into their Git accounts when you guys refuse to enable it for use in the CE of self hosted as well under the same pretenses.
I feel since its available on the actual website gitlab.com for free users, that I should also be able to get it for use in public repos in my CE edition. I dont see how mirror pulling into PUBLIC REPOS on my CE is an enterprise clause; especially when you guys already enabled it for free use on the gitlab.com website to non-paying users?
Sure, I could use CLI and cron jobs to do this (which I am currently) but I dont understand why I should have to add an extra layer of workflow to this when its already available in the GUI… and available for free users on gitlab.com.
I feel since free users can pull mirror into their gitlab repos from other places to PUBLIC REPOS then we should be able to in the CE as well.
As a student who also has some friends contributing on some websites and stuff we play around with learning together, i cant afford $25+ month to cover the seats JUST TO GET a feature that makes no sense to be EE… and the community has agreed with this fact:
Here is an issue I found regarding this exact thing, and you seem to have LOST potentional customers because of this…
Other git companies like Gitlea offer both pulling and pushing mirroring directly into even private repos, whereas you guys claim it is something EE need and CE doesnt… comparing Gitlab to other Git companies competitors you guys are starting to seem like you are falling behind… especially with Githubs new move with pushing towards being a lot more free.
A few years ago when Microsoft bought Github thousands of people swarmed gitlab looking for an alternative. Now Im seeing the opposite happen… and to me after reading many issue requests in your repo for Gitlab it shows me you guys dont listen to the community and just do things your way and that is most likely WHY the decline is happening.
It makes no sense how gitlab.com has this feature open for free use in public repos but the CE of gitlab self hosted doesnt. Why does the CE have LDAP? What company that isnt making profits uses LDAP? Why would a comoany making profits want to pull mirror into a PUBLIC repo? Theres also so many other festures in the CE that would be better fit in EE but for some reason you guys instead trap a feature better off for CE behind EE walls?
To me (and many others), this is all wrong… but it is your product and you guys can run it how you wish.
I just suggest you guys look into this instead of just pushing it off like it has been in this issue thread… as you guys are losing people over this. Gitlab will be just a name people say like AIM or MSN at this rate… my 2 cents.
Just noticed the OP of this thread is in the issue thread too, looks like hes one of the peoole who left Gitlab over this. Thats a shame.
Hi,
there’s no need to start yelling, nor continue with wrong accusations. I get that you want everything for free, this is why I recommend reading how GitLab operates on the pricing model and finances the work on the free open source version.
On a personal note: I’d recommend using the shell script to sync the repositories. Once you have solved your problem, and have taken a deep breath, please join the issue with your constructive feedback.
Cheers,
Michael
I was not yelling, sorry it came across that way.
Usually yelling involves capital letters. Im trying to have a discussion with you. Im asking why Gitlab believes this should be an EE feature where its available for free on gitlab.com?
I was pointing out facts backed with proof, like my screenshot. I proved that its available to free users on gitlab.com. Im not trying to lie or make stuff up so please inform me how what I sad is wrong so I can have better informstion? Maybe my account somehow got unlocked for Gitlab Pro on gitlab.com and thats why I have it? Hopefully we can get to the bottpm of that.
I also do not want everything free. I find your response very offensive actually when I was trying to gave a constructive conversation here.
I think you missed the part where I gave praise to the idea of locking the ability to mirror into private repos behind EE? I also said there are features given free in CE that would be better fit for paying EE customers… I dont quite understand where you think I think I should get everything free? So rude. Honestly, it makesme think you didnt even read what I had said.
Im trying to get my point across here that the public repo side of things makes little sense to the public, and no one has yet to go over /why/ that this exact feature was decided its EE?
