Mirror Dropdown Change from Push to Pull Not Working

Reading that I see this:

  1. You must have an existing GitLab project with an OSI-approved open source license

I am not even sure what that is… I dont host my stuff on gitlab only my self hosted gitlab and I dont plan to host my projects in like 3 different places all at once as thatll jut be chaotic; I want to host it from my self hosted gitlab. Needing to have my repo on gitlab it slef too defeats the purpose to self host… I might as well ditch my git and use gitlabs at thaf rate.

Gitlab is not looking like something for little developers like me who like to self host & are barely known. Its sad. :frowning:

Hi,

if the application cannot solve this, there’s another way to achieve this with native Git.

I had described this in this topic: Refreshing a Fork (the CLI ways).

A small cronjob on the CLI with that script should also work. I have been using this method for quite a while to sync everything between GitHub, GitLab, SVN repos and others.

Here’s an example we used in the past to sync from GitLab to GitHub (the other way around).

#!/bin/bash
REPO_HOME="/data/sync"
declare -A REPOS
# syntax is [github_repo]="gitlab_repo"
REPOS=([githubrepo]="gitlabgroup/gitlabrepo")
cd $REPO_HOME
for REPO_GITHUB in "${!REPOS[@]}"
do
	REPO_LOCAL=${REPOS[$REPO_GITHUB]}
        echo "### Processing repo $REPO_LOCAL"
        if [ ! -d $REPO_LOCAL ]; then
                git clone --bare --mirror https://gitlab.domain.org/$REPO_LOCAL.git $REPO_LOCAL
        fi
        (cd $REPO_LOCAL; git fetch --prune; git push --prune git@github.com:org/$REPO_GITHUB.git +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/*)
done
# su - git
$ crontab -e
 
# Sync repos to github
*/5 * * * * /data/sync/github/create_and_sync > /dev/null 2>&1

Cheers,
Michael

1 Like

Thanks so much for the info. I will be using your CLI suggestion. Although if you dont mind Id like some help understanding how to turn it around to work with cloning a public github to my self hosted gitlab. From what I gathered it should be rather simple to change it and this is what I came up with:

#!/bin/bash
REPO_HOME="/data/sync"
declare -A REPOS
# syntax is [gitlab_repo]="gitlab_repo"
REPOS=([gitlabrepo]="gitlabgroup/gitlabrepo")
cd $REPO_HOME
for REPO_GITLAB in "${!REPOS[@]}"
do
	REPO_LOCAL=${REPOS[$REPO_GITLAB]}
        echo "### Processing repo $REPO_LOCAL"
        if [ ! -d $REPO_LOCAL ]; then
                git clone --bare --mirror https://github.com/$REPO_LOCAL.git $REPO_LOCAL
        fi
        (cd $REPO_LOCAL; git fetch --prune; git push --prune git@git.mydomain.com:org/$REPO_GITLAB.git +refs/heads/*:refs/heads/*  +refs/tags/*:refs/tags/*)
done

I have a few questions though. The variable REPO_HOME is just a temp location, right? As Im thinking this will double my data stored as itll store it in my gitlab AND on the local file system in REPO_HOME? Would I be able to add a rm $REPO_LOCAL at the end of this script to clean up the data? As it would just recreate it during the sync anyways I dont think it would matter.

Here is an example I want to use, let me know if you think itll work:

#!/bin/bash
REPO_HOME="/data/sync"
declare -A REPOS
# syntax is [gitlab_repo]="gitlab_repo"
REPOS=([gitlabrepo]="callmekory/nezuko")
cd $REPO_HOME
for REPO_GITLAB in "${!REPOS[@]}"
do
	REPO_LOCAL=${REPOS[$REPO_GITLAB]}
        echo "### Processing repo $REPO_LOCAL"
        if [ ! -d $REPO_LOCAL ]; then
                git clone --bare --mirror https://github.com/$REPO_LOCAL.git $REPO_LOCAL
        fi
        (cd $REPO_LOCAL; git fetch --prune; git push --prune git@git.mydomain.com:org/$REPO_GITLAB.git +refs/heads/*:refs/heads/*  +refs/tags/*:refs/tags/*)
        rm $REPO_LOCAL
done

This should sync this repo: https://github.com/callmekory/nezuko
Into my gitlab at git.mydomain.com/callmekory/nezuko

If this is right then I for sure will be adjusting this to work with multiple repos, so i imagine adding repots to this since we have a for loop would be adding them here:

REPOS=([gitlabrepo]="callmekory/nezuko anothergroup/anotherrepo")

Ah, your repository groups and names are equal. We had the problem that it didn’t match for us, that’s why we created the REPOS key value pair.

To keep the code intact, just do it like this:

REPOS=(["callmekory/nezuko"]="callmekory/nezuko")

Also, rm $REPO_LOCAL won’t work, you’ll need to pass -r.

My Bash foo isn’t that accurate, I feel more familiar with other script languages. Though I thought I’d share it with you.

While looking into it again, ready-to-use scripts might be more applicable, the community is big enough :wink:

This one looks tempting, also since it uses https with tokens as transport instead of SSH keys. Might be worth a try.

Cheers,
Michael

Ill give it a go latwr tonight and see if it works for me, if not Ill fall back onto the bash script and configure it to work for what I need. :slight_smile:

@dnsmichi
Im affraid that this is not available on gitlab.com


Notice that dropdown is greyed-out.

Hi @AnnaLisik,

there was a change for private repositories announced, please follow my comment here: Pull based Mirroring from GitHub for CI is no longer mirrored

Cheers,
Michael

@dnsmichi
Why cant we do it in self hosted gitlab for public repos too then?
I want to mirror FOSS from github into my self hosted gitlab so I can back them up incase they eventually disappear like Ive seen happen in the past.
I cant do this without custom scripts on the CLI level when its available on the GUI level for people who have the EE.

I understand why private repo mirroring may be for EE, but just like how its available on gitlab.com for public repos… why cant we do it for public repos in self hosted?

Id also like to mention that Github just recently dropped a lot of their pricing and made a lot of festures avilable for the free github over the enterprise.

I truthfully think Gitlab should rethink their enterprise structuring like Github did and realize that feaures like this is not an enterprise feature whereas like someone above mentioned that LDAP really is something enterprise would use… but for some reason is available on the community editiom but mirroring isnt? Bizzare

Hi,

Which self-hosted version of GitLab are you using? Please share the version string found at https://gitlab.server/help. If you are using the self-hosted Enterprise version, the pull option is available from Starter forward, as described in the docs.

Cheers,
Michael

@dnsmichi
I am using the free version. I am a student learning git and cant afford starter when I make no money from my projects.
Thats what Im talking about. I understand how itd be disabled for private repos like it is on gitlab.com but why is it totally disabled for free use?
I understand why Enterprise may need it for private repos and I fully support that like how it is on gitlab.com although you guys removed allowing to do it even on public repos in the free version. It really makes no sense how you guys think it is something free users can have on gitlab.com but not in the self hosted version.

Especially with how many users have already complained about how this feature makes no sense to be locked behind EE when theres more sense for it to be a free feature and theres other features in the free package that make more sense to be EE.

Hi,

I’m trying to help you here, just outlining the facts:

  • The feature is part of the enterprise version
  • For the specific CI/CD mirroring feature, this was enabled on gitlab.com even for the free accounts
  • Since the blog post announcement happened, this is not true anymore (for private and public repositories that is, my mistake, I just checked it with a test account of mine)

That being said, this feature never existed in the Core/CE version. Your request slightly differs from the original topic where this is headed towards GitLab.com.

Anyhow, since you are self-hosted, you also have self-control about the CLI and may install a cronjob. With the tools and scripts I had shared in this topic already, this should be doable on your own. Either in a cronjob or with leveraging CI scheduled jobs.

Cheers,
Michael

@dnsmichi

I appreciate the time youre taking to respond. Thanks.

Ill also break down what Im saying :stuck_out_tongue:

  • Yeah, currently in the self hosted Gitlab you MUST have EE to pull mirror into either PUBLIC or PRIVATE repos.
  • Yeah, as stated above you can pull mirror into publix repos using a free user on the gitlab.com site.
    Picture for reference. You can see the option is NOT grayed out. Meaning I can pull mirror into my oublic repo on gitlab.com.

Now, what I am trying to point out is that I dont understand why this feature is enabled on gitlab.com for free users to be able to pull mirror into their Git accounts when you guys refuse to enable it for use in the CE of self hosted as well under the same pretenses.
I feel since its available on the actual website gitlab.com for free users, that I should also be able to get it for use in public repos in my CE edition. I dont see how mirror pulling into PUBLIC REPOS on my CE is an enterprise clause; especially when you guys already enabled it for free use on the gitlab.com website to non-paying users?

Sure, I could use CLI and cron jobs to do this (which I am currently) but I dont understand why I should have to add an extra layer of workflow to this when its already available in the GUI… and available for free users on gitlab.com.

I feel since free users can pull mirror into their gitlab repos from other places to PUBLIC REPOS then we should be able to in the CE as well.

As a student who also has some friends contributing on some websites and stuff we play around with learning together, i cant afford $25+ month to cover the seats JUST TO GET a feature that makes no sense to be EE… and the community has agreed with this fact:

Here is an issue I found regarding this exact thing, and you seem to have LOST potentional customers because of this…

Other git companies like Gitlea offer both pulling and pushing mirroring directly into even private repos, whereas you guys claim it is something EE need and CE doesnt… comparing Gitlab to other Git companies competitors you guys are starting to seem like you are falling behind… especially with Githubs new move with pushing towards being a lot more free.
A few years ago when Microsoft bought Github thousands of people swarmed gitlab looking for an alternative. Now Im seeing the opposite happen… and to me after reading many issue requests in your repo for Gitlab it shows me you guys dont listen to the community and just do things your way and that is most likely WHY the decline is happening.

It makes no sense how gitlab.com has this feature open for free use in public repos but the CE of gitlab self hosted doesnt. Why does the CE have LDAP? What company that isnt making profits uses LDAP? Why would a comoany making profits want to pull mirror into a PUBLIC repo? Theres also so many other festures in the CE that would be better fit in EE but for some reason you guys instead trap a feature better off for CE behind EE walls?
To me (and many others), this is all wrong… but it is your product and you guys can run it how you wish.
I just suggest you guys look into this instead of just pushing it off like it has been in this issue thread… as you guys are losing people over this. Gitlab will be just a name people say like AIM or MSN at this rate… my 2 cents.

Just noticed the OP of this thread is in the issue thread too, looks like hes one of the peoole who left Gitlab over this. Thats a shame.

Hi,

there’s no need to start yelling, nor continue with wrong accusations. I get that you want everything for free, this is why I recommend reading how GitLab operates on the pricing model and finances the work on the free open source version.

On a personal note: I’d recommend using the shell script to sync the repositories. Once you have solved your problem, and have taken a deep breath, please join the issue with your constructive feedback.

Cheers,
Michael

I was not yelling, sorry it came across that way.
Usually yelling involves capital letters. Im trying to have a discussion with you. Im asking why Gitlab believes this should be an EE feature where its available for free on gitlab.com?

I was pointing out facts backed with proof, like my screenshot. I proved that its available to free users on gitlab.com. Im not trying to lie or make stuff up so please inform me how what I sad is wrong so I can have better informstion? Maybe my account somehow got unlocked for Gitlab Pro on gitlab.com and thats why I have it? Hopefully we can get to the bottpm of that.

I also do not want everything free. I find your response very offensive actually when I was trying to gave a constructive conversation here.
I think you missed the part where I gave praise to the idea of locking the ability to mirror into private repos behind EE? I also said there are features given free in CE that would be better fit for paying EE customers… I dont quite understand where you think I think I should get everything free? So rude. Honestly, it makesme think you didnt even read what I had said.

Im trying to get my point across here that the public repo side of things makes little sense to the public, and no one has yet to go over /why/ that this exact feature was decided its EE?

Hi,

sorry about my tone, that was not intended to hurt you. I do think that there are many emotions running high here and we are getting at a point where a discussion is too heated. I cannot add anything more to it other than pointing to the feature request and encouraging you to add your arguments in there.

Have a nice day,
Michael

On the help page installed with the CE edition, help/user/project/repository/repository_mirroring.md, does have a small word “STARTER” up by the section head, but presumes that the reader will know that what’s installed isn’t “STARTER.” In much of the software world, the starter edition is the free one. This might all be more clearly indicated. Having the help pages of one version provide detailed procedures which can’t be done using it is not especially helpful, and so unlikely to dispose folks to want to someday become customers. In the context of the help pages bundled with GitLab, certainly they can be aware of which version of GitLab they’re installed for, and limit themselves to advice within the scope of that version.

I registered to say as a student it is very frustrating that I can’t pull mirror a public repository using the free version at gitlab.com, or a self hosted version of CE.

I spent 5 hours writing a pipeline before realizing that this feature is paid and behind a 100$ annual license paywall.

I needed this for a 1-month long team project to automate my deployments, and we were just a few days from the deadline, so this left me needing to find an alternative to gitlab.

Proposal: Give a 1-month free trial so I can see if I want to continue using gitlab in my personal project pipelines.

Alternative: Enable this feature as it’s been a frustration of your new users for over three years.

A trial is possible for free look under the menu in top right hand corner in web browser where its also possible to find user preferences. I have this and can enable it but if you trialled before then this option isnt available.

Alternative: pay for a subscription since gitlab dont have to give you everything for free. If you want functionality that is in pro or higher then pay for it. Otherwise use the push option and install gitlab on your own server. But obviously since you are complaining about it you dont want to pay and expect you should have everything you want for free. Doesnt work like that.