New scraping limits?

Ask any support / help / issues / problem or question related to any of our other products
Post Reply
Bartekef
Posts: 677
Joined: Thu Sep 22, 2016 12:24 pm

New scraping limits?

Post by Bartekef »

Hey, just saw it started asking for verification after 1k scrapped followers (CSV details). Used to do 30k without a problem.
Anyone noticed this ?
eddielp123
Posts: 510
Joined: Tue Feb 11, 2020 2:53 pm

Re: New scraping limits?

Post by eddielp123 »

I just realized in general that since yesterday API limits seem to have been reduced drastically.. can barely do anything at the moment without getting a 429 login error..
Bartekef
Posts: 677
Joined: Thu Sep 22, 2016 12:24 pm

Re: New scraping limits?

Post by Bartekef »

eddielp123 wrote: Fri Aug 28, 2020 6:52 pm I just realized in general that since yesterday API limits seem to have been reduced drastically.. can barely do anything at the moment without getting a 429 login error..
Holy cow. Thankfuly I managed to scrape a lot already, and I'd love to hear Martin's take on that
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: New scraping limits?

Post by martin@rootjazz »

Bartekef wrote: Fri Aug 28, 2020 6:33 pm Hey, just saw it started asking for verification after 1k scrapped followers (CSV details). Used to do 30k without a problem.
Anyone noticed this ?
The thing is, to get CSV details you need to make a 1 to 1 request per result. So even a 1k is a lot compared to what 99% of real users do. All IG need to do is correlate what do most users do within 2 standard deviations then block everything else. No real user was ever pulling 30k, so why let what are obviously bots do it?

To pull that many you will have to start using multiple accounts for your scrapes so those 30k requests can be shared between 20, 50 or 100 other accounts, limiting what a single account can do
Bartekef
Posts: 677
Joined: Thu Sep 22, 2016 12:24 pm

Re: New scraping limits?

Post by Bartekef »

martin@rootjazz wrote: Sat Aug 29, 2020 1:01 am
Bartekef wrote: Fri Aug 28, 2020 6:33 pm Hey, just saw it started asking for verification after 1k scrapped followers (CSV details). Used to do 30k without a problem.
Anyone noticed this ?
The thing is, to get CSV details you need to make a 1 to 1 request per result. So even a 1k is a lot compared to what 99% of real users do. All IG need to do is correlate what do most users do within 2 standard deviations then block everything else. No real user was ever pulling 30k, so why let what are obviously bots do it?

To pull that many you will have to start using multiple accounts for your scrapes so those 30k requests can be shared between 20, 50 or 100 other accounts, limiting what a single account can do
I guess, I was just asking if you noticed any change as I used to scrape 30k per acc for months
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: New scraping limits?

Post by martin@rootjazz »

No one else reporting it (yet?)
Post Reply