Resolved: Cataloging Fix and Postmortem!

I am pleased to announce that the cataloging issues effecting the project have been resolved. A silent hotfix has been released so effected users can simply re-download the project to get the latest changes. In addition, to minimize future disruptions the scraping engine has been updated to fully support browser simulation. This allows us to bypass many forms of anti-scraping techniques as well as the potential to add additional scraping sources. A one time download of support modules may be required for your OS (If you don’t already have them installed). But don’t worry, absolutely everything is done automatically for you.

So what happened?

Sometime during the morning of Jul 27, one of the largest sources for scraping updated their website to use a new format and layout for video entries. This change was completely unexpected and caused many scrapers to stop working correctly. In addition, video entries are now formatted in such a way (which I can only imagine) is intentionally designed to block scraping attempts.

How was this problem solved?

Since traditional methods of scraping no longer work, the scraping engine was upgraded to support full browser simulation. That means when requests are made to a website, the project can blend in by accurately simulating what a real user would do. In addition, this opens the future possibility of supporting more scraping sources, since the basic foundation has already been laid. If anyone notices any issues with the new system, please feel free to let me know.

Can this be prevented in the future?

Unfortunately, changes like this are totally out of our control. To be sure, we are very much unwelcome guests. Like a cat and mouse game, the only real thing you can do is wait until something happens, then try to react to it with a solution as soon as possible. Since the groundwork has been laid for browser simulation, hopefully changes in the future will be less disruptive. As a single developer (supported by a full time job), naturally there is only so much I can do during a single day. While I always try to respond to people and issues as soon as possible, in these cases, it may take me a bit more time. (^_^);