SWAP

Swap memory is space that is reserved via the hard disk and should only be utilized when necessary and in cases where the servers dedicated memory has been mostly used up and in your case, because the swappiness level of your server was set so high (60) it meant that the running processes on your server such as those from Nginx and Apache were running on swap memory rather than dedicated memory which is a lot slower.

root@serverbds:/var/log/atop# cat /proc/sys/vm/swappiness
60

As you can see from the above, prior to the change that I made, your server's swappiness level was set to 60 which meant that once 40% of the server's dedicated memory would be in use, the server would then start moving processes on to swap memory causing performance issues on the server.

I have since changed this value to 5 so that the server will not resort to swap memory as often and will utilize more of the dedicated memory when available which is crucial in impacting site performance which is what I suspect is what was causing the timeout issues that Nginx and Apache were logging.

root@serverbds:/var/log/atop# echo 5 > /proc/sys/vm/swappiness
root@serverbds:/var/log/atop# cat /proc/sys/vm/swappiness
5

root@serverbds:/var/log/atop# swapoff -a
root@serverbds:/var/log/atop# swapon -a