Transcript from our AMA
@amoloney First a few thank yous - Thank you to both the Fedora and CentOS communities for adding your questions to the hackmd doc and GitLab forum in advance of this session. Any questions we do not have time to answer in today’s session will be answered over the next week and published next Friday as a blog post on both Fedora and CentOS Community blogs with links to the session and a wrap up of how it went.
And thank you to our GitLab panelists for joining today and providing answers to some really great technical questions. I look forward to what will be a really insightful session on how GitLab operates and will help guide me and my team through our investigation and technical scoping of the possible migration to GitLab from Fedora.
@nuritzi Hi everyone!
@siddharthvipul @amoloney, \o 
@greg Hi Fedora Community! 
@amoloney oh here we go hahahaha
@Arrfab let’s all use irc ? 
@griffin hello
@cverna Hi everyone
@amoloney I’d like to introduce the panel for today’s call:
@lgriffin Hey everyone
@bstinson hey all
@siddharthvipul .hello siddharthvipul1
@amoloney Aoife Moloney - Product Owner, CPE
@lgriffin Leigh Griffin - Engineering Manager, CPE
@Nuritzi Nuritzi Sanchez - Senior Community Manager, GitLab
@Linds Lindsay Olson - Senior Community Advocate, GitLab
@mattdm Matthew Miller - Fedora Project Leader, Red Hat
@andr3 
André Luís - Frontend Engineering Manager, Create: Source Code
@greg Greg Myers - Support Engineer and Community Relations Support Counterpart
@mgill Michelle Gill - Backend Engineering Manager - Create: Source Code
@jayo-gitlab Hi! I’m Jason Young, a Support Engineer, GitLab, Support Counterpart for Open Source Program
@cverna Clément Verna - Engineering Manager, Fedora CoreOS
@bcotton Ben Cotton - Fedora Program Manager (and CentOS Stream Program Manager)
@nick-thomas Nick Thomas - Backend Engineer - Create: Source Code
@siddharthvipul what a great panel Thank you all
@amoloney Firstly Id like to thank both the Fedora and CentOS communities for adding your questions in advance, and thank you to our panelists for attending todays session
@amoloney I will be adding questions from the hackmd and allowing a panelist to post their answer
@amoloney any questions that are not answered within this session will be answered over the next few days and we would like to publish the session as a blog post to the community forum
@amoloney so, lets begin!
@amoloney 
Question: Fedora has a group-based access system. People in the packager group have (commit) access to only the packages they maintain. People in the provenpackager group have (commit) access to all the active packages, but a few (for legal reason). People in the releng group have commit access to all the packages. Is this an access model that GitLab can support? If not, how would this work in a GitLab world?
@amoloney Question: How would notifications work (Esp consider people in the provenpackager or releng group do not want to be notified for all the projects they have access to)?
@lgriffin Feel free also to ask questions in flight and we can answer ad hoc (there are enough of us here) and any we can’t answer we can take from the log and follow up async on devel list
@mattdm ooh, let’s do zodbot topics for these?
@mboddu: amoloney: Use #topic and followed by the question
@cverna: #topic permission and access in gitlab
@mattdm: sure that’ll work
@mattdm: cverna++
@cverna: I’ll take that one
@mboddu: cverna: Well, if you are part of the chair it will work
@cverna: So what we have currently looked at mapping the FAS groups to the permission models in GitLab
@cverna: for now it looks like we will have something like :
-
Packager and Co-Maintainer having a Developer role (commit access)
-
Proven Packager having a Developer role on all projects expect 2 (firefox and thunderbird)
-
then sysadmin and releng engineer having a Owner role on all projects
-
that means that Packager would not have access to the project settings so we will need to find a way for them to access settings that are needed for them
@zbyszek: Does “Packager and Co-Maintainer” give permission management? Adding any packager? Not being able to add a non-packager? Not being able to remove PP?
@King_InuYasha: No
@jlanda: and can owner alter the settings?
@King_InuYasha: Yes
@decathorpe: What about the “shim” and “kernel” packages?
@jlanda: or force push to an repo?
@cverna: There is a gitlab ticket open that would allow to have a more granular permission model https://gitlab.com/gitlab-org/gitlab/-/issues/7626
@jlanda: a*
@pingou: what about notifications?
@King_InuYasha: @jlanda: kind o
@cverna: about notifications Gitlab’s notifications are quite granular and can be managed at the different levels (Merge Requests, Projects, Group, Global)
@King_InuYasha: force push, if protected branches are set, is disabled
@cverna: https://docs.gitlab.com/ee/user/profile/notifications.html#global-notification-settings
@King_InuYasha: notifications do not map to what we have today
@pingou: cverna: so provenpackager would have to adjust them for all projects (created and new ones)?
@pingou: or will this be set by default?
@pjones: @decathorpe: those are protected from getting unauthorized signed builds done on the koji side as well as whatever is done in the repo, fwiw
@pingou: or something we’ll have to automate?
@cverna: @pingou: no since notification can be adjusted at the group level, so the provenpackager group would have notification disable by default
@jlanda: @cverna: does that allow to bulk disable all the notifications where someone is a developer because is a provenpackager while it keeps the ones because hi is a (co)maintainer?
@King_InuYasha: the permission inheritance model in gitlab doesn’t map to allow per project exclusions
@pingou: @cverna: is there a hierarchie in the groups then?
@lgriffin: Topics are really going to slow us down we have 20+ questions and answers and I don’t believe we can get through them all with the pace of answers, I’m going to propose we run 2-3 questions at once and use the persons IRC handle who is designated as answering it in any responses. Is that ok?
@King_InuYasha: e.g., everyone in rpms/* group (which would be proven packagers) cannot not have permissions
@King_InuYasha: which means packages cannot block provenpackagers individually anymore
@cverna: @pingou: the same user can be in different groups
@defolos: @lgriffin: please no, the chat is already hard to follow
@BrendanGitLab: Users can be in many groups, and groups can also have subgroups
@lgriffin: Ok we are going to get maybe 3-4 questions covered if we keep this approach
@pingou: @cverna: I mean, if I’m provenpackager I’ve notification disabled, but I also maintain packages, so I need notification for these
@misc: @King_InuYasha: that do not sound like a big problem, no ?
@King_InuYasha: @misc: Firefox, Thunderbird, the kernel, and shim are exceptions that nobody wants to fix
@King_InuYasha: I personally hate those exceptions, but we’re stuck with those
@zbyszek: @misc: yeah, I’d say that losing this capability is a net win.
@cverna: @pingou: yeah so provenpackager group = no notification, then you can manage the notification for each project you maintain as you wish
@pingou: @cverna: ok, thanks
@misc: @King_InuYasha: couldn’t it be replaced by some CI / approval ?
@King_InuYasha: Nope
@misc: like, you can’t merge unless you are authorized ?
@King_InuYasha: Nope
@King_InuYasha: you’re authorized, full stop
@amoloney: Shall we move onto the next topic? Next question is around Message Bus
@jlanda: there are a bunch of questions about acls, will we return to them?
@rbowen: Hopefully questions can be answered async after our time runs out here.
@jlanda: and I asked about owners capabilities too :S
@King_InuYasha: @jlanda: owners can do anything
@King_InuYasha: including bypass any restrictions
@mhroncok: “Shall we move onto the next topic?” I don’t feel this was answered enough 
@amoloney: Im totally fine to wait if you want to focus on this topic a bit more, just bear in mind the time
@King_InuYasha: @jlanda: same goes for maintainers
@petersen: @amoloney: +1
@jlanda: and alter history, or invite someone outside packager group to a repo?
@King_InuYasha: @jlanda: absolutely
@jlanda: that’s a big concern :S
@King_InuYasha: maintainers and owners have that power
@misc: ACL is kinda the core of the community interaction, that’s what define what you can and cannot do, so that’s extra important
@gregmyers: @King_InuYasha: There are also Admin-only options, like custom push rules and server hooks. Maintainers and owners cannot modify these settings.
@mhroncok: @cverna: so a s aprovenpackager I need to manually enable notifications on the packages I actually maintain?
@cverna: @mhroncok: the thing is that we will need to have a Proof of Concept to test a lot of the special cases
@King_InuYasha: @gregmyers: yes, but those are fraught with peril on upgrades
@King_InuYasha: having done those for my own instances before, they easily and regularly break
@misc: break in what way ?
@misc: cause if that break and block push, that will be detected fast
@King_InuYasha: misc: misfire, not fire, api changes, etc.
@cverna: @mhroncok: no that will come by default, it is just that as a member of the proven packager group you will have notification disable for all the other projects
@mhroncok: @cverna: ack
@mhroncok: @cverna++
@rbowen: Question from CentOOS-land - Will git.centos.org be moving to gitlab?
@misc @King_InuYasha: wow
@Arrfab: just for my understanding, which git instances are we talking about ? src.fedoraproject.org, pagure.io and git.centos.org ? all ? , on e ?
@rbowen: What Arrfab said.
@King_InuYasha: @misc: there are no stability guarantees beyond the frontend UI
@cverna: @Arrfab: afaik src.fp.o
@mboddu: @Arrfab: This is for src.fp.o
@amoloney: #topic message bus
@amoloney: Question: Fedora uses a message bus to integrate different parts of its infrastructure. How should we onboard GitLab into this message bus?
@King_InuYasha: and it’s arguable that the frontend UI doesn’t have stability guarantees either
@pingou: there is also an ACL question for CentOS, and iirc there is a question for it
@Arrfab: @cverna: oh, so git.centos.org would stay on pagure ?
@gregmyers: @King_InuYasha: if updates/upgrades to GitLab break custom server hooks and push rules, this would be prioritized a high priority high severity bug by our product team as the impact and scope would be large. If this happens in the future, we could create a bug report with reproducible steps and get our devs/engineers involved in finding a permanent fix.
@cverna: @Arrfab: I mean this AMA is mostly about src.fp.o
@King_InuYasha: @gregmyers: riiight
@lgriffin: @rbowen not for this conversation, git.centos.org is tied up with CentOS Stream and that will be a different Gitlab instance / configuration based on their needs. One hard need for Fedora was the Community Edition
@cverna: #topic message bus
@Arrfab: @cverna: ah, because message to join was sent to centos community to join here for this ama
@King_InuYasha: @gregmyers: that’d be an interesting change from gitlab’s previous policy
@petersen: @Arrfab: I believe it is in the longer term scope
@King_InuYasha: I’m not sure that would be even sustainable given how much internal instability you have there (which is not necessarily a bad thing)
@cverna: so currently GitLab does not support sending events to a message bus and that’s unlikely to happen
@cverna: so we will have to have a bridge similiar to what we have with github2fedmsg
@bstinson: i was about to say what @lgriffin and @cverna already did so i won’t…but it’s good to hear from the centos community if they’re here. if nothing else because it may affect us at some point
@amoloney: @Arrfab: yes as its important for both communities to be involved as we (CPE) are in both communities
@cverna: we have 2 options to do that either use webhooks or polling the events api
@King_InuYasha: if we go down this road, you will want to do webhooks
@decathorpe: wouldn’t polling be brittle madness?
@King_InuYasha: polling the API will kill the system
@King_InuYasha: it can’t handle the load and sidekiq will freak out
@King_InuYasha: since all requests internally are async through sidekiq
@jlanda: how much messages produces src.fp.o per hour?
@cverna: I said that there are 2 possible way forward, once might be better than the other 
@jlanda: we have a bunch of things that depends on them :S
@King_InuYasha: @jlanda: enough that I have a filter that sends all that mail to trash hourly 
@cverna: indeed I think the webhooks way should be preferred
@mhroncok: how would webhooks handle outages? currently, I believe that pagure will “eventually” emit the message to the bus
@jcpunk: On the API front, will CPE build a tool to replace https://git.centos.org/centos-git-common/blob/master/f/centos.git.repolist.py to avoid too much polling?
@King_InuYasha: @mhroncok: they’re lost
@misc: that’s bad 
@pingou: are the web hook notifications time-based?
@King_InuYasha: there is no eventual consistency guarantee
@King_InuYasha: there is no ordering guarantee either
@mhroncok: @King_InuYasha: so assuming the gitlab2fedmsg service is borken for a day, all events that happened on that day will never reach the bus, right?
@King_InuYasha: Yup
@decathorpe: that’s awful
@King_InuYasha: it’s the same thing that we have with github2fedmsg
@lgriffin: @jcpunk we can certainly look at any tooling or service request from the community, route them through @amoloney – that’s a general statement, we always welcome community requests
@mboddu: Aouch, thats gonna hurt
@King_InuYasha: if github throws 500s for a day, we lose everything there
@mhroncok: @King_InuYasha: except we don’t use github for anything important in that regard
@King_InuYasha: sure
@King_InuYasha: welp, I need to step away
@mattdm: any comments from the gitlab folks on this?
@mboddu: Is there a way that we can reply these lost messages?
@King_InuYasha: looking forward to seeing how this goes
@pingou: could our gitlab guest weigh in?
@jlanda: so we moved away from fedmsg because we lose messages, to come back to a message losing sceneario?
@lgriffin: Gitlab folks are chatting to confirm bear with us
@pingou: grizlly or teddy?
@mhroncok: @mboddu: it could possible be webhooks + nightly api calls to retrieve the lost ones
@BrendanGitLab: I’m not sure it’s guaranteed to be chronological.
@siXy: Currently this doesn’t sound a sufficiently reliable solution. It’d be great to see a design of how this could be made to be eventually consistent, with some reasonable bounds on what “eventual” would look like.
@Nuritzi: Uploaded file: https://uploads.kiwiirc.com/files/450cc0392a741e8ee7fb3a84b1a3bef0/pasted.txt
@BrendanGitLab: Also, depending on the cause downtime you may or may not lose messages - they may queue up
@Nuritzi: As an FYI – Fedora is part of GitLab’s Open Source program and we have a migration tracker issue that we are using to keep track of feature requests, bugs, etc that are important to Fedora. The Fedora migration team has been working with us at GitLab to maintain that and community members can add relevant issues there so we can track them.
@Nuritzi: It’s also helpful for our product managers to hear about why particular issues are important for the Fedora use case, and to have upvotes, so doing that will help! Here are some relevant links:
@nick-thomas: webhooks are out-of-order and there’s no guarantee of them arriving. I wouldn’t rely on it for something that needs those properties, we should think about alternatives
@BrendanGitLab: We also have an engineer who is going to add more detail as well, joining now
@pingou: @nick-thomas: any proposal?
@nick-thomas: it would be useful to know what sort of events you’re interested in
@lgriffin: Thanks @nuritzi, this gives the Community (and CPE) a means to make requests for consideration on your roadmap for any gaps, correct?
@mhroncok: @nick-thomas: pretty much all
@Nuritzi: @lgriffin yes, exactly
@mhroncok: @nick-thomas: at least what is publicly visible
@pingou: @nick-thomas: if you ask 10 people, you’ll get 11 answer, so @mhroncok +1, you can assume: all
@gregmyers: For authentication and access control, there are a number of options and “best” solution will change depending on the specific needs and use case of the community.
If we can get a better understanding of how your auth/ACL are currently, and how you’d like them to work, we can narrow-in on a solution. I’d like to open up a conversation and discuss this further, would #fedora-devel be the best place to go?
@nick-thomas: the events feed might be better then. We do have an event log for geo that has in-order, guaranteed delivery, but it’s designed for other stuff
@amoloney: just as an fyi, our next topic coming up will be around namespaces 
@mboddu: @nirik: How does events feed work?
@mboddu: @nick-thomas
@mattdm: @gregmyers: probably Fedora devel mailing list is better than irc
@siddharthvipul: @mattdm, +1 on mailing list than IRC
@misc: I guess we could have a cron that pull the event feed and emit message ?
@pingou: a little while ago I drafted the diagram of how commits get allowed/rejected: http://ambre.pingoured.fr/public/packager_commit_workflow.jpg
@jlanda: an * * * * * one misc?
@siXy: @nick-thomas: One usecase I’m aware of from centos: Due to the unreliability of src rpms being made available, larger users are consuming messages to know when an update is pushed to git, then generating the src rpm for their internal repo.
@amoloney: Next topic will being in 5 mins
@mboddu: @misc: But that doesn’t reply in chronological order