Best way to handle synching between standard and customization repos

We have a product that has its standard implementation and then there can be some customization made for each customer who decides to purchase that product.

I think this scenario can also be compared to open source projects with forks that users have customized (and some customizations may make it into the upstream project).

In order to have each customer project (with their customization) separated from the others, we decided to go down this route:

  • Have one repository containing the standard implementation
  • Have one fork for each customer, in which we implement any customization

‘til now, everything is fine, YES! :smile:

The problem arises when some customizations made for one customer become eligible for going back to the standard. How can we achieve this? What I thought about is cherry picking.

My idea is basically to

  • create a branch on the fork, starting from the main branch of the standard repo
  • cherry-pick the commits, related to the customization we want to make a standard
  • merge this branch into the standard repo
  • every customer gets that feature when they’re forks are rebased

In theory, this could work, but after some testing I encountered some problems.

Here is a step-by-step reconstruction of the tests I did.

Bringing back some changes on a file common to both the standard and the customization

  1. Cloned the fork locally
  2. Made one commit on the fork that won’t go back to the main repo
  3. Made another commit on the fork that will go back to the main repo
  4. Made another commit on the fork that won’t go back to the main repo
  5. Add the main repo (upstream) to the fork
> git remote add upstream <url to the main repo>
  1. Fetch the upstream to know where it is in regards of the fork
> git fetch upstream
  1. Create a branch starting from the latest commit on the upstream
> git checkout upstream/main
> git switch -c synch-back-to-standard
  1. Did a cherry pick of the commit that should go back to the main repo but got a merge conflict so I aborted
> git cherry-pick <commit sha>
Automatic merging of README.md
CONFLICT (content): merge conflict in README.md
error: unable to apply 0ac0656... This change will get to the shared project
suggestion: After resolving the conflicts, mark them with
suggestion: "git add/rm <pathspec>", then run
suggestion: "git cherry-pick --continue".
suggestion: You can instead skip this commit with "git cherry-pick --skip".
suggestion: To abort and get back to the state before "git cherry-pick",
suggestion: run "git cherry-pick --abort".
> git cherry-pick --abort
  1. No easy way around it, had to resolve the conflicts to move on with the cherry-pick
  2. Now I push the branch with the cherry-picked commit up and open an MR from the fork to the main (upstream) repo
  3. Merged the changes to the main project so it know has the commit form the fork
  4. Fetch the upstream on the local fork and fetch the origin pruning the branches
> git fetch upstream
> git fetch origin –prune
  1. Remove the previous branch. It complained about it not being fully merge so I force it
> git branch -D synch-back-to-standard
  1. Rebase the fork with the upstream
> git rebase upstream/main

This gave me a conflict with the first commit that did not make it to the standard project

  1. Rebase completed
  2. Force push to update the fork
> git push –force-with-lease
  1. The commit is there on both the main project and on the fork. It looks like it is duplicated but it’s not

The problems are:

  • the cherry pick had a conflict
  • the removal of the branch used to sync with the upstream had to be forced
  • most importantly, the rebase had a conflict!

Bringing a new file from the fork back to the standard

  1. Added a commit introducing a new file on the customization fork
  2. Update the upstream to fetch new changes
> git fetch upstream
  1. Checkout the upstream main and create a branch on that
> git checkout upstream/main
> git switch -c sync-with-standard-2
  1. Cherry pick the newly added commit
> git cherry-pick <commit sha>

It did not throw any strange message
5. Push the branch and create an MR from the fork to the standard branch

> git push -u origin sync-with-standard-2
  1. Merge the MR
  2. Fetch the upstream, the origin and remove the local branch
> git fetch upstream
> git fetch origin –prune
> git checkout main
> git branch -D sync-with-standard-2

I had to force-delete the branch again
8. Rebase the upstream main on the fork main

> git rebase upstream/main
warning: skipped previously applied commit 2b4f058
suggestion: use --reapply-cherry-picks to include skipped commits
suggestion: Disable this message with "git config advice.skippedCherryPicks false"
Rebase and update of refs/heads/main executed with success.
  1. Force push the fork to align it with the standard repo
> git push –force-with-lease

This case seems a bit better because the only problem was that the removal of the branch used to sync with the upstream had to be forced.

This is the history of the fork commits after all the tests:

As you can see, the “This change will get to the shared project” commit is there twice (it was the commit that had a conflict during the rebase).

This is the history of the standard repo commit after all the tests:

To sum it all up

  • If we bring back changes to an existing file, we have conflicts when cherry-picking and when rebasing the fork with the standard project
  • If we bring back a new file we won’t have any conflict, but only if we bring back ALL the commits that operate on said file
  • The branches used to bring the changes back to the standard repo have to be force-deleted

One other downside of this approach is that we have to be very careful when creating the commits, bearing in mind that we have to create a separate commit for changes that, one day, may become standard.

We have considered using branching, but think we will face the same problems (and will lose some of the benefits we get from forks).

We’re now thinking we need to either:

  • do it manually (e.g. pull out the bits of code and raise an MR in the upstream project)- we will still need to tackle the conflicts when rebasing the forks
  • look at language specific techniques (partial classes, extensions/plugins etc)- the obvious drawback being language dependant, but also we have to build the framework

We have also considered how this can be compared to GitLab itself and code separation for EE and JH.

Have any of you similar requirements and ideas how we might solve?

1 Like