I had a mysterious problem a year or so ago, when a server started mysteriously to go hideously slowly. Having investigated everything to do with the network, the database and anything else we could see that had more than a trivial amount of processing to do, we decided to strip down the one server that had the most power but did the least work (it was brand new and we'd only installed one of the many apps it was intended to run). Needless to say, this server turned out to be the problem.

Fast-forward to yesterday, and I had a similar problem - in a different company and even in a different country. Users in one of our offices were reporting network slowdowns with their Citrix applications, so we looked into everything we could think of and found no problems.

Several hours later, the various techies on the ever-growing conference call started thinking laterally, and saying things like: "What's running on these machines which couldn't possibly be causing the problem?". After another couple of hours, we found the problem via the same approach of stripping things down.

The problem in both cases? Anti-virus software. In both cases AV software had been installed on the server, and in both cases it was configured not to scan anything except when commanded to run a manual process. Yet in both cases (and it's worth noting at this point that the AV packages were from different vendors in each case) having AV installed caused the world to fall apart. Turning it off, disabling all the AV-related Windows services and rebooting didn't even help - you had to physically uninstall the AV product to get performance back to normal.

The moral of the story: think twice before you install AV software on your servers, and test it to death, because it might well bite you when you're least expecting it.