Hi,
so i just bought SCM and i´m a bit confused about all the options since mine is probably not a typical use case.
Let me explain please.
I want be able to scrape the following data from sc profiles/urls
- artist name
- follower count
- all social media links
I would like to be able to search for profiles fitting specific characteristics if possible.
Plus i would like to IMPORT my own LIST of URLS/sc profiles.
What would be the best way to get started ?
Other questions:
Are there any file size limits or quantity limits when it comes to importing data?
Do i need to have multiple SC accounts to scrape the data needed? I already have professional proxies
If so what is a good ratio for scaling up.
Help/Problems getting started
- martin@rootjazz
- Site Admin
- Posts: 34375
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Help/Problems getting started
SCRAPER TAB > SCRAPE PROFILE DETAILS > load in your filescrapedog wrote:Hi,
so i just bought SCM and i´m a bit confused about all the options since mine is probably not a typical use case.
Let me explain please.
I want be able to scrape the following data from sc profiles/urls
- artist name
- follower count
- all social media links
I would like to be able to search for profiles fitting specific characteristics if possible.
Plus i would like to IMPORT my own LIST of URLS/sc profiles.
What would be the best way to get started ?
This will save lots of information to a csv file, which you can load into excel and extract just what you want.
What do you mean? Importing data? What data? What? For what purpose. Are you talking 10mbs or 1gb files?Other questions:
Are there any file size limits or quantity limits when it comes to importing data?
No, you can scrape with your local IP, or provide the proxies to the SCRAPE tab and they are used in sequenceDo i need to have multiple SC accounts to scrape the data needed? I already have professional proxies
If so what is a good ratio for scaling up.
Re: Help/Problems getting started
I´m asking if a 1 gig file would be a problem
Purpose: Importing URLs from another scraper.
Purpose: Importing URLs from another scraper.
- martin@rootjazz
- Site Admin
- Posts: 34375
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Help/Problems getting started
Well if you import a gb file, the program holds it in memory. So probably is on the large side.
But how many images is that? (lines)
I imagine quite a few, do you have the proxy capability to scrape that many items?
But how many images is that? (lines)
I imagine quite a few, do you have the proxy capability to scrape that many items?
Re: Help/Problems getting started
i have no idea...
Just bought stormproxies backconnect for 39$
40 Threads
(40 Simultaneous Connections)
$39
Billed Monthly
70,000+ Proxies Pool.
Max. 40 Simultaneous Connections.
1 Access IP.
Unlimited Bandwidth.
From your experience how many URLS could i scrape with that setup and how
would i need to scrape six digit numbers?
They have these other offers with up to 150 threads... plus all have the 70k ip pool
I´ve also never had SC block my ips for some reason even when not using proxies. (Happened with other domains all the time)
Please tell me about your experiences/best practices, Martin..
Thank You!
Just bought stormproxies backconnect for 39$
40 Threads
(40 Simultaneous Connections)
$39
Billed Monthly
70,000+ Proxies Pool.
Max. 40 Simultaneous Connections.
1 Access IP.
Unlimited Bandwidth.
From your experience how many URLS could i scrape with that setup and how
would i need to scrape six digit numbers?
They have these other offers with up to 150 threads... plus all have the 70k ip pool
I´ve also never had SC block my ips for some reason even when not using proxies. (Happened with other domains all the time)
Please tell me about your experiences/best practices, Martin..
Thank You!
- martin@rootjazz
- Site Admin
- Posts: 34375
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Help/Problems getting started
No idea, never used them. but 6 digits is fine. That isn't so many. A 1gb file is going to be how many lines? I imagine you cannot even open it to find out. But I would imagine 10s of millions if not more.scrapedog wrote:
From your experience how many URLS could i scrape with that setup and how
would i need to scrape six digit numbers?
Why do you think you need to scrape 10s of millions of records?
Basically you have to try and see how it goes. The only way to know how it will go is for someone to do it and tell you. I haven't done it, so you'll have to be the trailblazerThey have these other offers with up to 150 threads... plus all have the 70k ip pool
years ago you'd get an IP block after about 300k pages pulled. Things are probably more strict.I´ve also never had SC block my ips for some reason even when not using proxies. (Happened with other domains all the time)
Please tell me about your experiences/best practices, Martin..
Thank You!
But load in your proxies, and see what happens (don't forget about the SCRAPING THREADS on the SETTINGS tab, you'll want to crank that up
Regards,
Martin