Email Scraper Not Working

Ask any support / help / issues / problem or question related to Soundcloud Manager
shaner4042
Posts: 86
Joined: Sun Sep 04, 2016 5:25 pm

Email Scraper Not Working

Post by shaner4042 »

Hey,

The email scraper in SCM randomly stopped working for me this morning. I've tried from multiple PC's on different networks, with/without proxies. I keep getting this error message: * FAILED: 87345 : Couldn't scrape profile ID

I'm scraping from users who've uploaded a track based on a certain search term, which I usually do successfully everyday. This problem just started happening randomly, not after a particular update or anything; so maybe this was due to a change done by SoundCloud?

* ERRROR: building profile details of: https://soundcloud.com/davemell: Error converting value {null} to type 'System.Int32'. Path 'reposts_count', line 1, position 715.
Failed pulled details for: https://soundcloud.com/davemell
* FAILED: 87345 : Couldn't scrape profile ID
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Email Scraper Not Working

Post by martin@rootjazz »

shaner4042
Posts: 86
Joined: Sun Sep 04, 2016 5:25 pm

Re: Email Scraper Not Working

Post by shaner4042 »

Great, thanks
Bri
Posts: 524
Joined: Fri Nov 13, 2015 10:53 am

Re: Email Scraper Not Working

Post by Bri »

Hi, sorry if i should make a new topic, but im scraping emails, and its not finishing the whole amount, am i missing something? see screen shot

Image

I also have another processing running, and its still going, been going for 14 hours + lol

Image
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Email Scraper Not Working

Post by martin@rootjazz »

if you are trying to scrape 1.5 million posts you can expect an IP ban from Soundcloud. No one has a reasonable use case for pulling 1.5 million pages. That is why there is a proxy option to specify proxies to spilt the page requests up.


But even with proxies, it will take a while, that is a lot of pages to pull
Bri
Posts: 524
Joined: Fri Nov 13, 2015 10:53 am

Re: Email Scraper Not Working

Post by Bri »

Ok, just some compete with all, and some dont, thanks for the reply.
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Email Scraper Not Working

Post by martin@rootjazz »

What do you think it won't compete eventually?
Bri
Posts: 524
Joined: Fri Nov 13, 2015 10:53 am

Re: Email Scraper Not Working

Post by Bri »

Complete*

some BIG scapes finish with all results, and some dont.
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Email Scraper Not Working

Post by martin@rootjazz »

Don't suppose you still have the logs?

Or you performed them within the last 7 days so you could submit logs?
Bri
Posts: 524
Joined: Fri Nov 13, 2015 10:53 am

Re: Email Scraper Not Working

Post by Bri »

Could be somewhere in my logs folder. But on with other stuff now, keep up the amazing work. I love your stuff. Glad you never started the vine bot. Haha. Vine = R.I.P.....
Post Reply