Hi I need to scrape like 20K profiles/day with full details but I get limited or banned

[varza]

Hi I need to scrape a large amount of users that need to have mail and phone (20K/day) . So I use to target followers of target account but the main issue is if I check on filter to have public mail and phone it give a lot less results and need a lot of time to scrape just a few users and after about an hour I get a termporarly block and ask to verify my mail or phone . How should I set my campaign to work without issues? I have rotating mobile proxies ( i can set the time to rotate) and can use multiple accounts but I'm not sure what is the best way to set the bot so to avoid the blocks? Is anyone who can recommend a good settings for my needs?

[varza]

Hi can anyone help me? thank you

[martin@rootjazz]

You need to test your searches to see what % give what you want.

Some searches may return 50% with mail / phone.
Some may give 1%.

Remember what you are doing .IG does NOT want you to do. Throwing accounts at the problem will help.

If you need to 20k results a day and yo are getting 25% of your searches with email / phone. Then you need to test 100k results.
With 1 account that is 100k requests to get the user_details endpoint per account
with 1000 accounts that is 100 requests per account. Much easier.

How many proxies do you have for your accounts?

Right now, the search module is NOT best optimised for rotating proxies, so trying to utilise 1 proxy for a LOT of accounts is could be difficult.

I will need to take a look at trying to find a way to allow the search module to allow each account once selected to call the GLOBAL CMDS to trigger an IP change.

Remind me about that next week and I'll take a look



Regards,
Martin

[varza]

Hi I have available to be honest 25 proxies mobile that have the option to change IP at a set time or by API. So how I set them proper I dont understand can I put more accounts on 1 proxy and how many? What you recommend? Also how I set the campaign to use more accounts and give unique results so to not scrape same profiles with different accounts? I will like to add a source file with the target profiles I want to target followers of these accounts and use a lot of accounts to do the scraping but to give unique results and not scrape same profiles. Right now with the filters I'm using is give around 15% valid results from the profiles it scrape ........Thank you

[varza]

The biggest issue is instagram after half an hour of an hour of scraping ask for phone or mail verification......

[martin@rootjazz]

The biggest issue is instagram after half an hour of an hour of scraping ask for phone or mail verification......

Make sure your accounts have POP ENABLED EMAILS, then the program can verify as they required. SMS verification is more complex as you need access to the same number which makes it much more complicated.

Have your accounts ever been email / phone verified? If not, then yes, they will always get a verification challenge at some point. It's not possible to use accounts that have never verified. Even on your phone, you will get hit with this



Regards,
Martin