#### Summary
When the error happened, can not access mattersmost service.
It w…ould redirect to login page and show "Token Error", like https://forum.gitlab.com/t/mattermost-token-error/65279/2.
Check mattermost.log, once this error happen, all db connection were broken.
Nov 11 09:41:13 messaging-server mattermost: {"timestamp":"2022-11-11 09:41:13.085 +08:00","level":"error","msg":"Failed to list plugin key values","caller":"app/plugin_key_value_store.go:167","page":0,"perPage":1000,"error":"failed to get PluginKeyValues with pluginId=playbooks: context deadline exceeded"}
Not sure it is related to https://github.com/jackc/pgx/issues/1313.
Bug report in one concise sentence
#### Steps to reproduce
Mattermost Version: 7.1.3
Database Schema Version: 89
Database: postgres (13.8)
A complete restart of the system solves the problem, but it comes back every 4-5 day.
We check db / memory / network status, can not find the problem.
#### Expected behavior
Even the error happened, it should broken that query not all of queries after that.
#### Observed behavior (that appears unintentional)
Once "context deadline exceeded" error happened, need to restart service.
```
[root@messaging-server /]# grep "context deadline exceeded" /opt/mattermost/log/mattermost.log | nl
1 Nov 11 09:40:51 messaging-server mattermost: {"timestamp":"2022-11-11 09:40:51.821 +08:00","level":"error","msg":"Unable to get the posts for the channel.","caller":"web/context.go:105","path":"/api/v4/channels/h9jjsc9cy3fauea87fif76jm9e/posts","request_id":"qpr5rhk1obbd5qj575soi7wrha","ip_addr":"61.227.52.141","user_id":"zcouasondpnxmjot3m7wg64tbw","method":"GET","err_where":"GetPostsSince","http_code":500,"err_details":"failed to find Posts with channelId=h9jjsc9cy3fauea87fif76jm9e: context deadline exceeded"}
2 Nov 11 09:40:51 messaging-server mattermost: {"timestamp":"2022-11-11 09:40:51.915 +08:00","level":"error","msg":"We encountered an error while finding user profiles.","caller":"web/context.go:105","path":"/api/v4/users/ids","request_id":"6xqay1wbd3gj8djc1d3ptcznww","ip_addr":"61.227.52.141","user_id":"zcouasondpnxmjot3m7wg64tbw","method":"POST","err_where":"GetUsersByIds","http_code":500,"err_details":"failed to find Users: context deadline exceeded"}
3 Nov 11 09:40:51 messaging-server mattermost: {"timestamp":"2022-11-11 09:40:51.915 +08:00","level":"error","msg":"Unable to get the posts for the channel.","caller":"web/context.go:105","path":"/api/v4/channels/9thezsgwmjfg8jn8p75oxxqz5r/posts","request_id":"bkq17jcc5iga3ru1yhi7m95m7o","ip_addr":"61.227.52.141","user_id":"zcouasondpnxmjot3m7wg64tbw","method":"GET","err_where":"GetPostsSince","http_code":500,"err_details":"failed to find Posts with channelId=9thezsgwmjfg8jn8p75oxxqz5r: context deadline exceeded"}
...
2038 Nov 11 10:05:15 messaging-server mattermost: {"timestamp":"2022-11-11 10:05:15.853 +08:00","level":"error","msg":"Unable to get the teams unread messages.","caller":"web/context.go:105","path":"/api/v4/users/me/teams/unread","request_id":"89tmianebj865ekh3hikg9uz9o","ip_addr":"182.150.116.153","user_id":"tz3o1c49abr85rp6t4mykrwbiw","method":"GET","err_where":"GetTeamsUnreadForUser","http_code":500,"err_details":"failed to find Channels with userId=tz3o1c49abr85rp6t4mykrwbiw and teamId!=: context deadline exceeded"}
2039 Nov 11 10:05:15 messaging-server mattermost: {"timestamp":"2022-11-11 10:05:15.897 +08:00","level":"error","msg":"We encountered an error while finding user profiles.","caller":"web/context.go:105","path":"/api/v4/users/ids","request_id":"wmiykmkqpjbjxm5kea6emctcwa","ip_addr":"182.150.116.153","user_id":"tz3o1c49abr85rp6t4mykrwbiw","method":"POST","err_where":"GetUsersByIds","http_code":500,"err_details":"failed to find Users: context deadline exceeded"}
2040 Nov 11 10:05:16 messaging-server mattermost: {"timestamp":"2022-11-11 10:05:16.686 +08:00","level":"warn","msg":"Error while creating session for user access token","caller":"app/session.go:89","error":"createSessionForUserAccessToken: Invalid or missing token., failed to get UserAccessToken with token=tkkicpgdw7bg8ew8uamcgqdjpo: context deadline exceeded"}
2041 Nov 11 10:05:17 messaging-server mattermost: {"timestamp":"2022-11-11 10:05:17.025 +08:00","level":"warn","msg":"Error while creating session for user access token","caller":"app/session.go:89","error":"createSessionForUserAccessToken: Invalid or missing token., failed to get UserAccessToken with token=tkkicpgdw7bg8ew8uamcgqdjpo: context deadline exceeded"}
2042 Nov 11 10:05:17 messaging-server mattermost: {"timestamp":"2022-11-11 10:05:17.106 +08:00","level":"warn","msg":"Error while creating session for user access token","caller":"app/session.go:89","error":"createSessionForUserAccessToken: Invalid or missing token., failed to get UserAccessToken with token=tkkicpgdw7bg8ew8uamcgqdjpo: context deadline exceeded"}
2043 Nov 11 10:05:17 messaging-server mattermost: {"timestamp":"2022-11-11 10:05:17.537 +08:00","level":"warn","msg":"Error while creating session for user access token","caller":"app/session.go:89","error":"createSessionForUserAccessToken: Invalid or missing token., failed to get UserAccessToken with token=tkkicpgdw7bg8ew8uamcgqdjpo: context deadline exceeded"}
2044 Nov 11 10:05:18 messaging-server mattermost: {"timestamp":"2022-11-11 10:05:18.724 +08:00","level":"error","msg":"Unable to save the token.","caller":"web/context.go:105","path":"/oauth/gitlab/login","request_id":"ece6t76n5jfcdmfkn5wteayuhe","ip_addr":"221.237.25.97","user_id":"","method":"GET","err_where":"CreateOAuthStateToken","http_code":500,"err_details":"failed to save Token: context deadline exceeded"}
2045 Nov 11 10:05:22 messaging-server mattermost: {"timestamp":"2022-11-11 10:05:22.237 +08:00","level":"error","msg":"Unable to update the last viewed at time.","caller":"web/context.go:105","path":"/api/v4/channels/members/me/view","request_id":"hzp7mf9fa3n89x3moz6pr3kfjw","ip_addr":"218.94.118.90","user_id":"pdjcf9h3ujymmn9qygc4m967so","method":"POST","err_where":"MarkChannelsAsViewed","http_code":500,"err_details":"failed to mark all threads as read by channels for user id=pdjcf9h3ujymmn9qygc4m967so: context deadline exceeded"}
2046 Nov 11 10:05:22 messaging-server mattermost: {"timestamp":"2022-11-11 10:05:22.275 +08:00","level":"error","msg":"Unable to update the last viewed at time.","caller":"web/context.go:105","path":"/api/v4/channels/members/me/view","request_id":"yqjn9dq5kfga3jgdxi7h1uq6cy","ip_addr":"218.94.118.90","user_id":"pdjcf9h3ujymmn9qygc4m967so","method":"POST","err_where":"MarkChannelsAsViewed","http_code":500,"err_details":"failed to mark all threads as read by channels for user id=pdjcf9h3ujymmn9qygc4m967so: context deadline exceeded"}
2047 Nov 11 10:05:22 messaging-server mattermost: {"timestamp":"2022-11-11 10:05:22.297 +08:00","level":"warn","msg":"Failed to save status","caller":"app/status.go:332","user_id":"uyb7inmwdtf18f799c7my1or1w","error":"failed to upsert Status: context deadline exceeded"}
```
We restarted mattermost at 10:05 after that no such error happen yet.
#### Possible fixes
Should we upgrade pgx to fix it?
https://github.com/jackc/pgx/issues/1313