Best way to download huge threads for offline viewing?

[Cdd89]

I've spent a couple of hours looking at options and thought I'd come here for suggestions

I want to download a few 200-page threads off here for offline viewing - it would make good reading material while on flights/trains without internet service!

It turns out this is slightly harder than I remember it being. Years ago I'd have used WinHTTrack with crawling rules, but that doesn't seem particularly well-suited to the modern Internet.

I experimented with ArchiveWeb.Page + a script to keep clicking next, but this is incredibly slow and resource intensive as it's network based, therefore it wants to download every asset before moving on even though a lot of the assets are actually shared.

Finally I looked at XenForo export utilities, like XenForo Export Thread, which look promising but these all require being installed server side. (Maybe the Admins would consider that though?)

Before I spend more time digging, I just wondered if anyone else has been around this and found a good solution. Happy for it to be technical and involve scripting, if required.

 

[najaB]

That sounds like a job for the API, but that would require an admin to create an API key for you. It's something I'd be interested in too, for what it's worth.

 

[styles]

I expect the quickest way will be writing a small script to just loop through the pagination links, download the content of the page, open the next link in the pagination, until end.

It's probably actually something you ask an LLM to write a script for you to be perfectly honest.

Either that, or, as @najaB says, ask the site admins to either export it for you or help you gain access to the API, assuming the forums are backed by one. Edit: looks like the forum software does have such an option: https://docs.xenforo.com/api

 

[najaB]

Either that, or, as @najaB says, ask the site admins to either export it for you or help you gain access to the API, assuming the forums are backed by one. Edit: looks like the forum software does have such an option: https://docs.xenforo.com/api

Indeed there is. (I checked before mentioning it ).

 

[pokemonsuper9]

I experimented with ArchiveWeb.Page + a script to keep clicking next, but this is incredibly slow and resource intensive as it's network based, therefore it wants to download every asset before moving on even though a lot of the assets are actually shared.

While the images are useful, in the bottom bar where it says RailUK Forums there is a "worksafe" option which hides all images, that'd make your download probably quite a bit quicker.

 

[najaB]

While the images are useful, in the bottom bar where it says RailUK Forums there is a "worksafe" option which hides all images, that'd make your download probably quite a bit quicker.

Also, if you are forced to go down the "script to scrape" route it would be worthwhile to take note of the Cache-Control header and not re-download content every time.

 

[Cdd89]

Thanks for the suggestions, in particular the WorkSafe style option from @pokemonsuper9 which dramatically reduced the overhead.

Programmatic tools tended to give 403 errors even with browser useragents and cookies passed through, thanks to CloudFlare, so I settled on a browser extension called SingleFile which accepts a bulk list of URLs and downloads each URL and all its dependencies to a single .html file. (Interestingly it rewrites all the content as local resources rather than the classic MIME-style MHT formats!)

 

[Mike395]

That sounds like a job for the API, but that would require an admin to create an API key for you. It's something I'd be interested in too, for what it's worth.

Unfortunately we aren’t really in a position to give out API keys to non staff; it’s open to abuse (as we have no way of tracking how that data is used once it leaves our network) and as a small volunteer team we don’t really have the ability to actively monitor the use of these in the way that would really be needed.

My apologies! Please also bear in mind forum content is copyrighted to the member who wrote it. So it’s not fair on our membership to provide an easy way to pull data in bulk.

Also, our forum is supported by ads and this wouldn’t support ad data to also be pulled through.

 

[najaB]

Unfortunately we aren’t really in a position to give out API keys to non staff; it’s open to abuse (as we have no way of tracking how that data is used once it leaves our network) and as a small volunteer team we don’t really have the ability to actively monitor the use of these in the way that would really be needed.

My apologies! Please also bear in mind forum content is copyrighted to the member who wrote it. So it’s not fair on our membership to provide an easy way to pull data in bulk.

Also, our forum is supported by ads and this wouldn’t support ad data to also be pulled through.

That's fair enough, and pretty much the response I expected to be honest. But, as they say: if you never ask you'll never get.