Occasional Broken Images when Caching Locally

4 posts / 0 new
Last post
puravida's picture
puravida
Jedi Warrior
Offline
Joined: 09/01/2007
Visit puravida's Website

Hello,

It was reported by a couple of users over the past few weeks that images will occasionally be stored in a broken state. We determined that this was caused by the behavior of our advanced API image delivery script (xian). It has always output text error messages for various error conditions, but we realize now that this causes most integrations to save that error text output as an image, which then shows up "broken" if displayed in a browser.

If you open the image file in a text editor, you will see the error message given by our script. It is useful for troubleshooting but makes for a poor user experience when a broken image placeholder is displayed.

Therefore, we have changed the behavior of this script to no longer output text error messages except in certain cases that only occur when testing, debugging, or possibly while writing a new integration. For most cases, the script will no longer output anything (allowing end user scripts to gracefully fail and retry) but will log the error to the ShrinkTheWeb account log from now on. This way, users will get the same benefit of error reporting but no negative user experience and no having to open files to determine the error message.

The most common reasons for this situation have been:

"Lock to account: domain or IP not listed"
"Image Retrieval Error"

The "Lock to account" condition is already reported in the logs, so this change will more gracefully handle this situation.

The "Image Retrieval Error" has been handled with some changes to the script behavior. You may read more about that here:
https://shrinktheweb.com/content/new-logging-output.html

If you notice anything unexpected from these changes, please let us know. Smile

Best regards,

Brandon

puravida's picture
puravida
Jedi Warrior
Offline
Joined: 09/01/2007
Visit puravida's Website

I have been closely monitoring the expanded output when these image retrievals fail, and it appears to be an on-going issue after all. It appears that the images DO exist in the storage facility but are failing to download at random. No error is provided by the storage provider. This is occurring at a rate of about 1 in 150,000 requests. For some reason, it seems to be happening more often with a small subset of users. That may lend a clue if we are able to pinpoint some similarities between the requests and/or images. However, it may also be coincidence, based on usage and a specific type of integration.

We have opened a ticket with our storage provider and will work with them to determine the root cause of the failures and correct the issue.

puravida's picture
puravida
Jedi Warrior
Offline
Joined: 09/01/2007
Visit puravida's Website

There may be more than one issue going on here, but I have identified one major cause that may be the only culprit. Several months ago, I made a slight change to the "Rolling Refresh" query that controls which "stale" requests to refresh (in order to keep our cached images fresh).

This change included refreshing certain error types, in case broken sites may be working after some time, but I did not account for those error types in the update query. So the system was treating them as if they were "good" and the XML response was telling users to download images that should not exist --hence the error. I have corrected this now and it should not happen again.

If we continue to see errors for some other reason, then I will have to dig deeper to see why. However, I have a feeling that this may be the fix that was needed.

puravida's picture
puravida
Jedi Warrior
Offline
Joined: 09/01/2007
Visit puravida's Website

I have been monitoring this issue for weeks, and it appears that there are still reasons that cause these errors. After in-depth testing and enabling logging in our third-party storage provider, it appears to be a systematic issue with the third-party storage provider (possibly related to our high rate of access).

There are two conditions that have been causing the errors we log as "Image Retrieval Error":

1. Cached Image Missing - for whatever reason, the image no longer exists in the third-party cache. This should not be possible, so we have been collecting logs and will open a ticket with the storage provider. In the meantime, we have modified the code to detect this specific condition and log it in our system as "Cached Image Missing" from now on (instead of "Image Retrieval Error"). This condition will result in an automatic, free refresh request for that request.

2. Connection Error / No Response - for an unidentified reason, the remaining errors are happening with no data response from the storage provider to indicate what went wrong. Subsequent attempts succeed, so the image exists in cache. These failures are intermittent, so we have been collecting logs for awhile and will open a ticket with our storage provider to attempt to identify the root cause of this issue and prevent it. These errors will continue to be logged in our system as "Image Retrieval Error".

I suspect that our high rate of access may be to blame, but our storage provider says that they can handle quite a bit more volume than we currently generate. We have also worked with them to optimize the image per storage volume distribution, but that has had no noticeable effect.

We will continue to work on this issue to find a suitable resolution.

ShrinkTheWeb® (About STW) is another innovation by Neosys Consulting
Contact Us | PagePix Benefits | Learn More | STW Forums | Our Partners | Privacy Policy | Terms of Use

Announcing Javvy, the best crypto exchange and wallet solution (coming soon!)

©2018 ShrinkTheWeb. All rights reserved. ShrinkTheWeb is a registered trademark of ShrinkTheWeb.