Our site ran out of resources because of an overly-aggressive search spider, so we have tightened our network firewall filters to prevent this in the future. Your help is needed as we fine-tune the settings.
Like nearly all publicly-visible web sites, we place our server behind a network "firewall" device that helps block malicious traffic. One of the features of a firewall is that it can also limit the rate of legitimate traffic to prevent a server from being overloaded.
Earlier today, a search engine spider decided to scan all our pages at a very fast rate. This is normal activity from search engines like Google and Yahoo, and is how they gather links for their database. The better search engines try to be nice about fast they request pages, but a few are not so polite. It's not (usually) malicious, just poor technical practice. That's what happened to us this morning; a search spider ran our server out of resources, and I had to reboot (a very rare event with Linux) to clean up the problem.
After the reboot, I reset our firewall to more strictly limit the rate at which pages can be requested. This applies only to new connections (which happen when your browser does a page refresh); the speed at which the page appears after the request is not affected.
As we fine-tune this setting, you may occasionally receive a connection refused error from the site, if we've set the limit too low. Over the next 48 hours or so, if you get such an error please send an email to email@example.com to let me know.
Our settings were updated at noon U.S. Eastern time (17:00 GMT), so errors that occurred before that have hopefully already been addressed.
Other errors besides connection refused are not related to this change -- our system is set to issue that very specific error code if it denies the connection. So if you get some other error, it may be your ISP or other more typical causes, so you should investigate those in the usual way.
If you do report a connection refused error, please specify the date and time when it occurred, approximately how many errors you received and how frequently, and if you are able, the IP address from which you were connecting. We do not -- ever -- need or want you to send your SCAtoday.net password or other private information through email.
Thank you for your assistance and understanding. We hope that this change will help make our site more reliable over the long term, and we thank you for your patience as we fine-tune the limit over the next day or two.
Webmaster and Publisher, SCAtoday.net