xuejianxianzun

Ways to bypass the 429 (crawl frequency) limit (not perfect)

Added 2023-08-31 06:31:20 +0000 UTC

If you initiate a large number of crawl requests in a short period of time, Pixiv will return a 429 status code and reject your request for about 3 minutes.

How to bypass 429 restrictions? The easiest way is to log out of your Pixiv account so that the crawl frequency is not limited.

But this is not perfect, because when you are not logged in, Pixiv will not return some key data of R-18(G) works to you, such as the original image URL.

So, if you want to scrape illustrations for all ages (and no sexual depictions), or novels for all ages, you can log out and start scraping. (Also remember to turn off the "slow down crawling speed" function of the downloader)

This is especially effective when you need to do a lot of crawling. For example, if you want to crawl 100,000 (or more) consecutive works, you can log out and then use the "Crawl ID range" function on the home page.