I found the issue that may happen for some people. My /etc/apt/sources.list.d/gitlab_gitlab-ce.list file contained the following entries:
deb [signed-by=/usr/share/keyrings/gitlab_gitlab-ce-archive-keyring.gpg] https://packages.gitlab.com/gitlab/gitlab-ce/debian/ buster main
deb-src [signed-by=/usr/share/keyrings/gitlab_gitlab-ce-archive-keyring.gpg] https://packages.gitlab.com/gitlab/gitlab-ce/debian/ buster main
As you can see the signing key is specified here and it is the expired one. Removing the [signed-by=...] config or replacing the gpg key file with the proper one fixes the issue.