Hey, just saw it started asking for verification after 1k scrapped followers (CSV details). Used to do 30k without a problem.
Anyone noticed this ?
New scraping limits?
-
- Posts: 510
- Joined: Tue Feb 11, 2020 2:53 pm
Re: New scraping limits?
I just realized in general that since yesterday API limits seem to have been reduced drastically.. can barely do anything at the moment without getting a 429 login error..
Re: New scraping limits?
Holy cow. Thankfuly I managed to scrape a lot already, and I'd love to hear Martin's take on thateddielp123 wrote: ↑Fri Aug 28, 2020 6:52 pm I just realized in general that since yesterday API limits seem to have been reduced drastically.. can barely do anything at the moment without getting a 429 login error..
- martin@rootjazz
- Site Admin
- Posts: 34674
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: New scraping limits?
The thing is, to get CSV details you need to make a 1 to 1 request per result. So even a 1k is a lot compared to what 99% of real users do. All IG need to do is correlate what do most users do within 2 standard deviations then block everything else. No real user was ever pulling 30k, so why let what are obviously bots do it?
To pull that many you will have to start using multiple accounts for your scrapes so those 30k requests can be shared between 20, 50 or 100 other accounts, limiting what a single account can do
Re: New scraping limits?
I guess, I was just asking if you noticed any change as I used to scrape 30k per acc for monthsmartin@rootjazz wrote: ↑Sat Aug 29, 2020 1:01 amThe thing is, to get CSV details you need to make a 1 to 1 request per result. So even a 1k is a lot compared to what 99% of real users do. All IG need to do is correlate what do most users do within 2 standard deviations then block everything else. No real user was ever pulling 30k, so why let what are obviously bots do it?
To pull that many you will have to start using multiple accounts for your scrapes so those 30k requests can be shared between 20, 50 or 100 other accounts, limiting what a single account can do
- martin@rootjazz
- Site Admin
- Posts: 34674
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: New scraping limits?
No one else reporting it (yet?)