Scrape Emails from United Kingdom Profiles

Discussions to do with Soundcloud Manager. Do not use for support, use the dedicated support forum for help requests
Post Reply
deepGC
Posts: 4
Joined: Wed Oct 26, 2016 7:39 pm

Scrape Emails from United Kingdom Profiles

Post by deepGC »

Hi guys,

First of all, kudos to the developers here, this is a great piece of kit which I'm going to have lots of fun with :D

I've ran into one problem. I am trying to scrape emails from UK users only, I've looked on the YouTube guides but it seems that they are old and the software has been upgraded since.

The process I tried:

Soundcloud Modules -> Scraper
Using 5 proxies
Set Artist filter to United Kingdom
Scrape Followers of: "filepath of four popular UK profiles"
-> Scrape Users

This quickly scraped the users, but it did NOT scrape all users. For example, some accounts had a total of 400k followers, my TOTAL results from all four accounts was around 10k.

Using the user data scraped, I then ran another process:
Soundcloud Modules -> Scraper
Using 5 proxies
Set Artist filter to United Kingdom
Scrape Emails from Profiles: "filepath of the scrape above"
-> Scrape


The process has just about finished now, I've checked the output file and there are a LOT of US users on there, it seems the filter I applied did not work for some reason.

Can someone help me in the right direction please?

Thanks in advance, and all the best for the new year guys
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Emails from United Kingdom Profiles

Post by martin@rootjazz »

There are two methods for trying to get users from a specific location.

SEARCH USERS (this is basically a username search) and you can apply Soundcloud provided location filter: united kingdom.

for example, this page:
https://soundcloud.com/search/people?q= ... ace=london

you will see it is a people search in london. Now whether there are specific location filters you can use I am not sure, but it is up to you to provide the correct input values for it to work

United kingdom does seem to work
https://soundcloud.com/search/people?q= ... %20kingdom


The above search will ONLY return users from soundcloud matching the specified location. However you are limited to USER SEARCH / PEOPLE SEARCH. How soundcloud decides what people to return is not known. I think it used to be a username search only, but seems they have improved to return other profiles as well now


Additionally, there is a SOUNDCLOUD MANAGER location filter. This will allow you to filter an existing list of artists. So you could perform a search for "all artists who have uploaded a track with the genre "hiphop" in the last 24 hours, then filter artists from united kingdom. This allows you to perform more complex searches, but the issue is, maybe of your 1000 results, only 20 users are from the UK, thus 98% are filtered out.



If you can provide me with the EXACT searches you made, I can test to ensure things are working as they should
This quickly scraped the users, but it did NOT scrape all users. For example, some accounts had a total of 400k followers, my TOTAL results from all four accounts was around 10k.
It is not possible to scrape ALL followers of an account if it is a large number. The limit (from soundcloud) used to be 8k, meaning you could only ever see 8k results of ANY search. This has been increased and I do not know the actual limit, but it is not possible to scrape unlimited results from a search
deepGC
Posts: 4
Joined: Wed Oct 26, 2016 7:39 pm

Re: Scrape Emails from United Kingdom Profiles

Post by deepGC »

Thanks for the advice, I'll try again tomorrow.

The max users you can scrape per account seems to be 10k.c annoying as like I said some users has 400k + followers.

I think I need to read around the forum and guides a little more,

Thanks again
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Emails from United Kingdom Profiles

Post by martin@rootjazz »

there is a global max results limit on the SETTINGS tab, that *might* be limiting you as well as I had a feeling it was more than 10k, but these things change so....
deepGC
Posts: 4
Joined: Wed Oct 26, 2016 7:39 pm

Re: Scrape Emails from United Kingdom Profiles

Post by deepGC »

martin@rootjazz wrote:there is a global max results limit on the SETTINGS tab, that *might* be limiting you as well as I had a feeling it was more than 10k, but these things change so....
Hi Martin,

I'm running another scrape after updating the hard limit from 10k to 500k. So far so good, it's now scraping all of the users from LARGE accounts, thanks for the heads up
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Emails from United Kingdom Profiles

Post by martin@rootjazz »

Thanks for letting me know :)
deepGC
Posts: 4
Joined: Wed Oct 26, 2016 7:39 pm

Re: Scrape Emails from United Kingdom Profiles

Post by deepGC »

Further feedback @Martin:

Scraping user accounts using five proxies and I would say that 60% of results show:

"FAILED: Couldn't pull page: -1 SC URL
Failed pulled details for SC URL"

Any idea what the issue is here?
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Emails from United Kingdom Profiles

Post by martin@rootjazz »

failed to get the page that was requested.

proxy failed
page doesn't exists
network issue
Post Reply