PagePix adds "invalid encoding" detection

2 posts / 0 new
Last post
puravida's picture
puravida
Jedi Warrior
Offline
Joined: 09/01/2007
Visit puravida's Website

It has been a monumental challenge to properly detect invalid encoding issues over the years, because it has been nearly impossible to reproduce the variety of possible issues. To make matters worse, most of the encoding issues came from overseas users who did not speak English and would not reply to requests for more information.

However, with our recent improvements and increased logging, we have finally tracked down some of these occurrences in the wild. One of the biggest offenders ended up being a very specific combination of browser encoding support vs web page encoding vs screenshot request URL encoding. Read about the nitty gritty details here.

The changes released today to the delivery code, on our side, do not require any end-user changes and should prevent HASH_MISMATCH errors from occurring again. There may still be rare cases of encoding weirdness that gets through and causes that message, but so far, the changes have all but eliminated those errors.

One important note, though, is that we have implemented this invalid encoding check for referrers to the service as well. Since the recent upgrade to our delivery scripts, we have more security related to referrers, and since invalid encoding "could" be problematic; we now block any access from referrers with invalid encoding (should never be a problem, since web page encoding is not involved in referrer tracking). The important takeaway is that it is possible that our "invalid referrer detection" could unintentionally block a "good" referrer (or screenshot URL request), and it may not be entirely clear why the requests are failing. This should never happen, in practice, and we tested dozens of scenarios to ensure there were no false positives that we could reproduce; but the code to detect is extremely complex, so it is good to be aware of the possibiity of a false positive (if your requests are dropped for no obvious reason).

puravida's picture
puravida
Jedi Warrior
Offline
Joined: 09/01/2007
Visit puravida's Website

Update: This new code has been working very well over the past few days. The database is no longer cluttered with tons of HASH_MISMATCH errors now. Excellent!

However, I had an epiphany while I was hunting down some other issues and I have made a slight tweak to the encoding/decoding algorithm that will also correct a number of NS_ERROR_UNKNOWN_HOST and NS_ERROR_NET_TIMEOUT errors caused by unnecessary '+' (plus signs). There were tons of requests like http://mobileagca.com++ that would fail. Now, the system gracefully cleans up this and about 23 new cases of bad data. Those requests will work now, instead of showing the expected error.

Since I'm on a rampage to investigate "bad data" in the database, something I do every once in awhile, the system is up to over 100 cases of "bad data" that are supported. While we do not advocate ignoring problems with coding and websites, when I see a lot of users across the world with the same kinds of "bad data"; I figure that I might as well make the system "just work" for them. It means they may never fix their bad code but at least they will be blissfully happy with the service. Smile

ShrinkTheWeb® (About STW) is another innovation by Neosys Consulting
Contact Us | PagePix Benefits | Learn More | STW Forums | Our Partners | Privacy Policy | Terms of Use

Announcing Javvy, the best crypto exchange and wallet solution (coming soon!)

©2018 ShrinkTheWeb. All rights reserved. ShrinkTheWeb is a registered trademark of ShrinkTheWeb.