Can I scrape faster / Can I scrape 1m followers with multiple accounts to scrape faster?

Post Reply
chupakabra
Posts: 15
Joined: Sun Feb 09, 2020 3:29 pm

Can I scrape faster / Can I scrape 1m followers with multiple accounts to scrape faster?

Post by chupakabra » Tue Mar 24, 2020 12:56 pm

Hey Admin,

I want to scrape an IG handle with 1mn follower. All I have done before is scrape 1 handle with one or 2 account but the time taken is same.

Can I speed up the time? I think scraping of 1mn followers will take huge time, how do I shorten it with resources like multiple accounts?

User avatar
martin@rootjazz
Site Admin
Posts: 23658
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Can I scrape faster / Can I scrape 1m followers with multiple accounts to scrape faster?

Post by martin@rootjazz » Tue Mar 24, 2020 7:07 pm

chupakabra wrote:
Tue Mar 24, 2020 12:56 pm
Hey Admin,

I want to scrape an IG handle with 1mn follower. All I have done before is scrape 1 handle with one or 2 account but the time taken is same.

Can I speed up the time? I think scraping of 1mn followers will take huge time, how do I shorten it with resources like multiple accounts?
No you cannot speed it up, as you cannot get the next page of results until you have scraped the previous. Also, be careful intended to scrape a million results. No real user has ever request results from a search 999,900 to 1,000,000 - only a bot would ever do that. Also, do you need these million results? Can you do anything with that many profiles? Have you tested these results, do they convert well for you? Are you sure they are real users / are they active?

I have seen people scrape through a large result set, only to find out they just scraped 200,000+ fake profiles an account purchased as fake followers.

chupakabra
Posts: 15
Joined: Sun Feb 09, 2020 3:29 pm

Re: Can I scrape faster / Can I scrape 1m followers with multiple accounts to scrape faster?

Post by chupakabra » Tue Mar 24, 2020 10:49 pm

Can you do anything with that many profiles?
^ With my experience of 7 years, with data you can do 3 things mainly - email marketing, Custom audience for social media, whatsapp/sms marketing

Have you tested these results, do they convert well for you? Are you sure they are real users / are they active?
^ So here's the thing - 95% of all the celebrities verified/non v verified profiles I tested have many many fake users. The trick is to find accounts in a given niche who have more engagements...spend 5 mins and then start scraping.

I have found that the ratio present is 1:5 to 1:20 for presence of contact details for 50000 data scraped from any IG profile. Range is very wide.

Also, for marketing purposes, it is really really imp to have lots of scraping.

User avatar
martin@rootjazz
Site Admin
Posts: 23658
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Can I scrape faster / Can I scrape 1m followers with multiple accounts to scrape faster?

Post by martin@rootjazz » Wed Mar 25, 2020 4:05 pm

chupakabra wrote:
Tue Mar 24, 2020 10:49 pm
Can you do anything with that many profiles?
^ With my experience of 7 years, with data you can do 3 things mainly - email marketing, Custom audience for social media, whatsapp/sms marketing
With just a profile URL / ID? You cannot do the above. You would need to scrape the profile URL and check for phone numbers, which would require A LOT of proxies to pull a million web pages. Do you have access to those proxies?
Have you tested these results, do they convert well for you? Are you sure they are real users / are they active?
^ So here's the thing - 95% of all the celebrities verified/non v verified profiles I tested have many many fake users. The trick is to find accounts in a given niche who have more engagements...spend 5 mins and then start scraping.
very wise
I have found that the ratio present is 1:5 to 1:20 for presence of contact details for 50000 data scraped from any IG profile. Range is very wide.

Also, for marketing purposes, it is really really imp to have lots of scraping.
ok, if you are sure you are have a need for such a scrape.


But as above, scraping is as follow:

scrape page1: extract ID for page2
scrape page2: extract ID for page3
....
scrape page10000: extract ID for page: 10001

so it cannot be threaded, it has to be scraped one page at a time. Just start your scrape with your account and leave it to run.

If you want to scrape big numbers, your best option is to run multiple different scrapes at the same time, as those can be thread - one thread for each scrape.




Regards,
Martin

chupakabra
Posts: 15
Joined: Sun Feb 09, 2020 3:29 pm

Re: Can I scrape faster / Can I scrape 1m followers with multiple accounts to scrape faster?

Post by chupakabra » Wed Mar 25, 2020 9:34 pm

ok, if you are sure you are have a need for such a scrape.


But as above, scraping is as follow:

scrape page1: extract ID for page2
scrape page2: extract ID for page3
....
scrape page10000: extract ID for page: 10001

so it cannot be threaded, it has to be scraped one page at a time. Just start your scrape with your account and leave it to run.

If you want to scrape big numbers, your best option is to run multiple different scrapes at the same time, as those can be thread - one thread for each scrape.
Okay. Seems like this is the only way now.
With just a profile URL / ID? You cannot do the above. You would need to scrape the profile URL and check for phone numbers, which would require A LOT of proxies to pull a million web pages. Do you have access to those proxies?
I don't think we are on the same page. You asked me what will I do with the data scraped? I said email, SMS and custom audience. Again, where did proxy come into the picture. Can you please connect the dots here I am lost...

User avatar
martin@rootjazz
Site Admin
Posts: 23658
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Can I scrape faster / Can I scrape 1m followers with multiple accounts to scrape faster?

Post by martin@rootjazz » Thu Mar 26, 2020 4:28 pm

chupakabra wrote:
Wed Mar 25, 2020 9:34 pm

I don't think we are on the same page. You asked me what will I do with the data scraped? I said email, SMS and custom audience. Again, where did proxy come into the picture. Can you please connect the dots here I am lost...
You said you wanted to scrape a million profile URLs.

With just a the URL, there isn't much you can with that. If you want the emails, you can feed the profile URLs into the email scraper module, but you need a lot of proxies to do this now due to IG protections.

If you are saving the CSV / JSON DETAILS during the scrape, then it just became more difficult to scrape a million results.

When you scrape just IDs / URLs, the program can pull 100 results per request. But if you want DETAILS, then the program must make 1 request per results.

so 1 request get 100 results
then another 100 requests to get the details for each result.

so for just urls, you perform 1 request for 100 results
with details , you perform 101 requests for 100 results.

So you need A LOT of accounts to even begin to think about pulling a million results with DETAILS. Again as above, not real account actually will ever pull a million results and pull the user details end point for each result.

chupakabra
Posts: 15
Joined: Sun Feb 09, 2020 3:29 pm

Re: Can I scrape faster / Can I scrape 1m followers with multiple accounts to scrape faster?

Post by chupakabra » Fri Mar 27, 2020 2:38 pm

I think im getting a hang of it!

Thank you

Post Reply