Always something ...Router Filtering

1 post / 0 new
puravida's picture
puravida
Jedi Warrior
Offline
Joined: 09/01/2007
Visit puravida's Website

So our roll out of the new generators has successfully squashed the remaining, irritating, and lingering issues that we have known about for awhile. Things are generally improved. The main issues fixed were failed captures of HTTPS sites when using Comodo CA SSL certificates and also an intermittent "shortening" of captures that were not the exact size they were supposed to be (affected custom sized and default sizes alike).

The roll out itself did not go so smoothly and we just now got the new gens to a stable point (long story). Things should be running much better now, though.

At any rate, the new gens have shed some light on a "sticky" problem. We still have about 0.00003% failed captures (CAPTURE_FAILED) when the site is actually working. We have long said that these were routing issues. However, after much research, we are finding that the IP ranges of our datacenters (with Ubiquity) are being filtered out by certain sites. This "Router Filtering" is not targeting us specifically but does affect us. Our datacenter colocation vendor, Ubiquity, will not assist in this because they say it is outside of their network. For some reason, large blocks of their IPs are being filtered.

This wasn't such a big issue for us when we were running in the Rackspace Cloud, but we migrated away from Rackspace due to their unrealistic pricing model.

Unfortunately, there is nothing that we can do but point the finger at the sites doing the filtering and tell people to ask them to stop blocking Ubiquity's IP ranges. Even twitter.com and constantcontact.com were in this boat until we got in touch with them and finally got them to remove the filter record. Then, we were able to capture those sites. Those were examples of very high profile sites that we couldn't capture. However, there is no way that we can monitor for these relatively rare occurrences and then reach out to thousands of web hosting companies to get them to fix their filtering. It's a bit of a catch-22.

ShrinkTheWeb® (About STW) is another innovation by Neosys Consulting
Contact Us | PagePix Benefits | Learn More | STW Forums | Our Partners | Privacy Policy | Terms of Use

Announcing Javvy, the best crypto exchange and wallet solution (coming soon!)

©2018 ShrinkTheWeb. All rights reserved. ShrinkTheWeb is a registered trademark of ShrinkTheWeb.