In vteams “____” I had an issue of production servers crashing after every 15 mins.
Problem :
We have testing and other instances available to reproduce the same issue but we were never able to reproduce that issue on testing servers. Then to debug that issue we went one step ahead and enable all sorts of logging in our project identified a few API that may cause crash as we were also not having proper error why the server is crashing. After lots of efforts and no luck we thought there must be something wrong with our server so we also tried changing server for avoiding that issue and it didn’t helped as well.
Then we saw RDS and we were having low memory on production database that was the cause of server crashing and we updated that and this fix avoided or suppressed the issue for a couple of months but the issue was still there.
One day I was testing why the memory is consumed so much even after updating RDS mysql instance and I saw that there was a table session which was haing millions of records but no indexing both things were of concern for us.
Solution :
So I start debugging the sessions plugin and what we saw surprised us and we were having a session created in database for each http call even the issue is successfully logged in and that module is running a query to delete all unused sessions created 10 years ago to delete and that query is running, and table has no indexing and database is going busy and that query is scheduled to run every 15 minutes when I saw the plugins code this was getting our server crashed after waiting for database connection establishment timeout. We shifted to jwt tokens as we don’t need sessions, removed the sessions storage plugin cleared that table and the issue is fixed.
0 Comments