Downloader 1.6 Released

One week after the release of Update All 2.0 we now have an upgrade on its brain. Downloader 1.6 is a substantial change, that will make our updating process quite a bit quicker (also when using Update All).

But first some history. On September 2021, MiSTer OS was upgraded to include a new Python version with new standard libraries (among other things). This had serious implications for the tools I was developing, for example before that downloader had to depend on Linux CLI utilities for downloading files which introduced some overhead, and now this wasn't necessary. But I didn't want to leverage on top of that yet, because it would have caused crashes on users that were not so up-to-date. I wanted to give them time to update to a later OS version, so I decided to give them one year.

Fast-forward one year, here we are. Now thanks to Python 3.9 and its new libraries, I'm reimplementing these parts in a much more optimal way. As a result of this work, Downloader is now a cross-platform tool and quite a bit faster.

Speed improvements

The speed improvements come from using Python Threads. They are not true multithreading since they don't let you use more CPU cores, but that's not important. Python Threads allow you to optimize code that is heavy on IO operations, and IO operations such as requesting files from the internet are by far the most time-consuming part of Downloader so they are a perfect fit for us.

The new implementation allowed me to process all files from a DB in a single go, instead of having to split them into batches of 100-200 files. On top of that, I moved the file validation to the main thread which saved a bit of extra time for each file (that's why you won't see a "Checking files..." message anymore). Both these things contributed to the speed improvement.

In my setup, I measured two scenarios multiple times. In the first one I'm installing a few databases from scratch (including Distribution MiSTer). And in the second one I'm rerunning Downloader after deleting all folders starting with the `_` character. Here are the results:

• In the first scenario, I observed a speed improvement between 1.47x and 1.68x. I did 6 full runs to get these numbers and removed one outlier.
• In the second one, I had more extreme results. The speed improvement was between 1.56x and 2.51x, also after running the scenario 6 times.

These numbers are highly dependent on network conditions, but regardless of that show very significant and consistent speed improvements. I hope you observe similar results too (please let me know in the comments!)

The Downloader PC Launcher (Windows, Mac & Linux)

This is another outcome of the refactor I talked previously about. Since the code was now cross-platform I just had to tweak a few things to have Downloader running successfully on Windows with a new launcher.

That new launcher is called downloader_pc_launcher.py and it comes with this release. When you call it from your PC, it will download all the files to the same location where it's placed in. A very common use case will be to keep it in the root of the MiSTer SD card so that you can install files by inserting that SD in your PC from time to time.

This is the perfect replacement for the old updater-pc. I recommend everyone using that solution to move to this new launcher as soon as possible because the old updater-pc will no longer work correctly pretty soon.

More information on where to download the PC Launcher and how to run it is here.

Other changes

There have been quite a few more changes in this release. To check all of them, please take a look at the CHANGELOG.md file.

And that's all for now. Thanks for reading and please let me know any feedback about the recent updates. I'll be happy to read it.

Have a good update!