I have a high-load OLTP SQL Server 2008 Enterprise. 4 NUMA, 96 cores, 430 GB RAM allocated to SQL. Most data and logs are on NAND SSD, some logs on 10K drives. It's linked to a few other systems as Linkd Servers, but most remote queries run via OPENQUERY().
Occasionally, very memory-intensive ad-hoc queries come in (heavy in hash and sort) that slow the system down (I know this needs to be fixed). These queries usually remain within a single database. As a quick-and-dirty immediate solution, we restart SQL Server service. However, this does not fix performance issues, threads keep piling up, but there seems to be no memory load nor tempdb load, and CPU shows hardly any load at all.
A hard-reboot of the entire system fixes the problem, though. This leads us to believe that the issue is on the OS side more than SQL side.
P.S. While SQL doesn't seem to be actively using any memory, it is very slow to release it on the OS side.
Question is - how do I troubleshoot this and determine what prevents SQL from returning to peak performance after just a service restart vs. a hard reboot?