It works by first taking a list of URLs into its text area. These are parsed line by line looking for a regex pattern of application type and an ID. For instance,
sub/75498. This simple detection method means most Steam URLs work, including community links and those with query strings.
Whenever a request finishes, the game is marked as “fetched” and the downloaded information stored. This then triggers a new function to build a markdown table with the current information. The asynchronous requests already significantly speed things up, but seeing the table building in real time also makes it feel faster.
The script knows it’s completed when all games have been marked “fetched”, and it indicates to the user that the process is finished.
As simple as a concept as this project was, it ended up being far more complicated than I first anticipated. I had to work around problems such as multiple regional currencies, each of which required an HTTP request; getting around same-origin policy, which meant writing a companion PHP script to curl the requests; and writing a thread manager to keep track of asynchronous HTTP requests. Writing thread-safe code was a completely new idea to me, and it took a complete change in thinking to get my head around the idea.
This was also my first real contribution to open source. It’s something I’ve wanted to do for a long time, and I plan to do so again whenever possible.
Check out Steampunk Spider on Github and feel free to set up an instance yourself.