Help/Problems getting started

Ask any support / help / issues / problem or question related to Soundcloud Manager
Post Reply
scrapedog
Posts: 23
Joined: Tue Aug 01, 2017 10:11 am

Help/Problems getting started

Post by scrapedog »

Hi,
so i just bought SCM and i´m a bit confused about all the options since mine is probably not a typical use case.
Let me explain please.

I want be able to scrape the following data from sc profiles/urls

- artist name
- follower count
- all social media links

I would like to be able to search for profiles fitting specific characteristics if possible.

Plus i would like to IMPORT my own LIST of URLS/sc profiles.

What would be the best way to get started ?
Other questions:
Are there any file size limits or quantity limits when it comes to importing data?
Do i need to have multiple SC accounts to scrape the data needed? I already have professional proxies
If so what is a good ratio for scaling up.
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Help/Problems getting started

Post by martin@rootjazz »

scrapedog wrote:Hi,
so i just bought SCM and i´m a bit confused about all the options since mine is probably not a typical use case.
Let me explain please.

I want be able to scrape the following data from sc profiles/urls

- artist name
- follower count
- all social media links

I would like to be able to search for profiles fitting specific characteristics if possible.

Plus i would like to IMPORT my own LIST of URLS/sc profiles.

What would be the best way to get started ?
SCRAPER TAB > SCRAPE PROFILE DETAILS > load in your file

This will save lots of information to a csv file, which you can load into excel and extract just what you want.
Other questions:
Are there any file size limits or quantity limits when it comes to importing data?
What do you mean? Importing data? What data? What? For what purpose. Are you talking 10mbs or 1gb files?
Do i need to have multiple SC accounts to scrape the data needed? I already have professional proxies
If so what is a good ratio for scaling up.
No, you can scrape with your local IP, or provide the proxies to the SCRAPE tab and they are used in sequence
scrapedog
Posts: 23
Joined: Tue Aug 01, 2017 10:11 am

Re: Help/Problems getting started

Post by scrapedog »

I´m asking if a 1 gig file would be a problem

Purpose: Importing URLs from another scraper.
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Help/Problems getting started

Post by martin@rootjazz »

Well if you import a gb file, the program holds it in memory. So probably is on the large side.

But how many images is that? (lines)

I imagine quite a few, do you have the proxy capability to scrape that many items?
scrapedog
Posts: 23
Joined: Tue Aug 01, 2017 10:11 am

Re: Help/Problems getting started

Post by scrapedog »

i have no idea...
Just bought stormproxies backconnect for 39$

40 Threads
(40 Simultaneous Connections)

$39
Billed Monthly

70,000+ Proxies Pool.
Max. 40 Simultaneous Connections.
1 Access IP.
Unlimited Bandwidth.

From your experience how many URLS could i scrape with that setup and how
would i need to scrape six digit numbers?

They have these other offers with up to 150 threads... plus all have the 70k ip pool

I´ve also never had SC block my ips for some reason even when not using proxies. (Happened with other domains all the time)
Please tell me about your experiences/best practices, Martin..

Thank You!
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Help/Problems getting started

Post by martin@rootjazz »

scrapedog wrote:

From your experience how many URLS could i scrape with that setup and how
would i need to scrape six digit numbers?
No idea, never used them. but 6 digits is fine. That isn't so many. A 1gb file is going to be how many lines? I imagine you cannot even open it to find out. But I would imagine 10s of millions if not more.

Why do you think you need to scrape 10s of millions of records?

They have these other offers with up to 150 threads... plus all have the 70k ip pool
Basically you have to try and see how it goes. The only way to know how it will go is for someone to do it and tell you. I haven't done it, so you'll have to be the trailblazer

I´ve also never had SC block my ips for some reason even when not using proxies. (Happened with other domains all the time)
Please tell me about your experiences/best practices, Martin..

Thank You!
years ago you'd get an IP block after about 300k pages pulled. Things are probably more strict.

But load in your proxies, and see what happens (don't forget about the SCRAPING THREADS on the SETTINGS tab, you'll want to crank that up



Regards,
Martin
Post Reply