Lame corporate proxies
Like a lot of weblogs out there, we use filtering technology to detect and block spambots, dodgy referrers and automated link spam specialists.
While our anti-spam software effectively blocks around 99% of all comment spam, we occasionally get some false positives and inadvertently block some humans as well. As of last week, this is what our stats reveal about some of the reasons why both spambots and human visitors are denied access to our website:
| Ratio | Reason |
|---|---|
| 21.61% | Required header "Accept" missing. A B |
| 9.02% | Header "Pragma" without "Cache-Control" prohibited for HTTP/1.1 requests. A |
| 9.02% | Prohibited header "Proxy-Connection" present. A |
| 9.02% | Prohibited header "via" present. A |
| 8.53% | User-Agent string is required but none was provided. A B |
| 4.59% | Header "Referer" present but blank. A B |
| 1.14% | Header "Connection" contains invalid values. A B |
| 0.88% | Header "Referer" is corrupt. A B |
| 0.42% | Header "TE" present but TE not specified in "Connection" header. A |
| 35.77% | OTHER REASONS. |
A Often caused by a misconfigured web proxy or corporate firewall.
B Often caused by a misconfigured personal firewall or browser privacy software.
As you can see, around two out of three of all blocked attempts are due to misconfigured web proxies, corporate and personal firewalls which often mangle the HTTP headers on behalf of the users browser.
While a percentage of these problems are caused by crappy personal firewall default settings, the majority eminate from visitors sitting behind lame corporate proxy servers. The RFC 2616 specification defines a corporate "transparent proxy" as:
"A proxy that does not modify the request or response beyond what is required for proxy authentication and identification."
In other words, a properly configured proxy should not obstruct browser authentication at all. So why do so many of them manage to do exactly that?
Due to the large volume of spam, comments are disabled. If you have anything relevant to say, you can leave a comment via Twitter, or contact me directly.
About the author
I'm Ivan Lutrov and I'm the owner of Lutrov Interactive. I have 25 years of experience producing interactive work and I create cost effective business websites that are simple, engaging and easy to use. I practice what I preach and I say what I really think, even if it's sometimes not what you want to hear. Subscribe to the Lutrov Interactive feed via RSS and follow me on Twitter.