Cannot get cache with 2 stage Docker build of .NET Core 2.0

I have dockerfile like that:

FROM microsoft/aspnetcore-build:2.0 AS build-env WORKDIR /app

    # Copy csproj and restore as distinct layers 
COPY *.csproj ./ 
COPY *.config ./ 
RUN dotnet restore --configfile NuGet.config
    # Copy everything else and build 
COPY . ./ RUN dotnet publish -c Release -o out

# Build runtime image 
FROM microsoft/aspnetcore:2.0 
COPY --from=build-env /app/out . 
ENV ASPNETCORE_URLS http://+:5000 
ENTRYPOINT ["dotnet", "AuthService.dll"]

I cannot get cache for that on Gitlab build.

The whole Nuget cache sits in build-env under ~/.nuget/packages (when I run ls ~/.nuget/packages while still on the build-env it’s showing all nuget packages, but Gitlab somehow does not see that (I suppose it checks only last container.

Any idea how to solve it? It is making builds so long because of no NuGet cache…
I’d really appreciate any help with that.