New scraping limits?
Posted: Fri Aug 28, 2020 6:33 pm
Hey, just saw it started asking for verification after 1k scrapped followers (CSV details). Used to do 30k without a problem.
Anyone noticed this ?
Anyone noticed this ?
Holy cow. Thankfuly I managed to scrape a lot already, and I'd love to hear Martin's take on thateddielp123 wrote: ↑Fri Aug 28, 2020 6:52 pm I just realized in general that since yesterday API limits seem to have been reduced drastically.. can barely do anything at the moment without getting a 429 login error..
The thing is, to get CSV details you need to make a 1 to 1 request per result. So even a 1k is a lot compared to what 99% of real users do. All IG need to do is correlate what do most users do within 2 standard deviations then block everything else. No real user was ever pulling 30k, so why let what are obviously bots do it?
I guess, I was just asking if you noticed any change as I used to scrape 30k per acc for monthsmartin@rootjazz wrote: ↑Sat Aug 29, 2020 1:01 amThe thing is, to get CSV details you need to make a 1 to 1 request per result. So even a 1k is a lot compared to what 99% of real users do. All IG need to do is correlate what do most users do within 2 standard deviations then block everything else. No real user was ever pulling 30k, so why let what are obviously bots do it?
To pull that many you will have to start using multiple accounts for your scrapes so those 30k requests can be shared between 20, 50 or 100 other accounts, limiting what a single account can do