Most likely you exceeded the runner limits, see this post:
If the runner has 4gb limit, then you can’t exceed that - if using Gitlab.com - otherwise you can install and configure your own runners and connect them to gitlab.com or your own Gitlab instance if you are running it.
So your only real option is create your own runner with sufficient memory resources and connect it to gitlab.com or your own Gitlab instance: Install GitLab Runner | GitLab and if using Gitlab.com you can disable shared runners to force your project to use your own personal runner.