Lame corporate proxies
Like a lot of weblogs out there, we use filtering technology to detect and block spambots, dodgy referrers and automated link spam specialists.
While our anti-spam software effectively blocks around 99% of all comment spam, we occasionally get some false positives and inadvertently block some humans as well. As of last week, this is what our stats reveal about some of the reasons why both spambots and human visitors are denied access to our website:
| Ratio | Reason |
|---|---|
| 21.61% | Required header "Accept" missing. A B |
| 9.02% | Header "Pragma" without "Cache-Control" prohibited for HTTP/1.1 requests. A |
| 9.02% | Prohibited header "Proxy-Connection" present. A |
| 9.02% | Prohibited header "via" present. A |
| 8.53% | User-Agent string is required but none was provided. A B |
| 4.59% | Header "Referer" present but blank. A B |
| 1.14% | Header "Connection" contains invalid values. A B |
| 0.88% | Header "Referer" is corrupt. A B |
| 0.42% | Header "TE" present but TE not specified in "Connection" header. A |
| 35.77% | OTHER REASONS. |
A Often caused by a misconfigured web proxy or corporate firewall.
B Often caused by a misconfigured personal firewall or browser privacy software.
As you can see, around two out of three of all blocked attempts are due to misconfigured web proxies, corporate and personal firewalls which often mangle the HTTP headers on behalf of the users browser.
While a percentage of these problems are caused by crappy personal firewall default settings, the majority eminate from visitors sitting behind lame corporate proxy servers. The RFC 2616 specification defines a corporate "transparent proxy" as:
"A proxy that does not modify the request or response beyond what is required for proxy authentication and identification."
In other words, a properly configured proxy should not obstruct browser authentication at all. So why do so many of them manage to do exactly that?
Due to the large volume of spam, comments are disabled. If you have anything to say, please feel free to contact me directly.
About the author
Ivan Lutrov is the owner of Lutrov Interactive. He creates cost effective business websites that are simple, engaging and very easy to use. When not busy working on client and personal projects, he's into photography, fishing, cricket, tennis, music from the 70s, cooking, good wine, and apparently knows "way too much" about movies. He tells it like it is, whether you like it or not. Subscribe to the Lutrov Interactive feed via RSS and follow Ivan on Twitter.