Hey,
The email scraper in SCM randomly stopped working for me this morning. I've tried from multiple PC's on different networks, with/without proxies. I keep getting this error message: * FAILED: 87345 : Couldn't scrape profile ID
I'm scraping from users who've uploaded a track based on a certain search term, which I usually do successfully everyday. This problem just started happening randomly, not after a particular update or anything; so maybe this was due to a change done by SoundCloud?
* ERRROR: building profile details of: https://soundcloud.com/davemell: Error converting value {null} to type 'System.Int32'. Path 'reposts_count', line 1, position 715.
Failed pulled details for: https://soundcloud.com/davemell
* FAILED: 87345 : Couldn't scrape profile ID
Email Scraper Not Working
- martin@rootjazz
- Site Admin
- Posts: 34712
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
-
- Posts: 86
- Joined: Sun Sep 04, 2016 5:25 pm
Re: Email Scraper Not Working
Great, thanks
Re: Email Scraper Not Working
Hi, sorry if i should make a new topic, but im scraping emails, and its not finishing the whole amount, am i missing something? see screen shot
I also have another processing running, and its still going, been going for 14 hours + lol
I also have another processing running, and its still going, been going for 14 hours + lol
- martin@rootjazz
- Site Admin
- Posts: 34712
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Email Scraper Not Working
if you are trying to scrape 1.5 million posts you can expect an IP ban from Soundcloud. No one has a reasonable use case for pulling 1.5 million pages. That is why there is a proxy option to specify proxies to spilt the page requests up.
But even with proxies, it will take a while, that is a lot of pages to pull
But even with proxies, it will take a while, that is a lot of pages to pull
Re: Email Scraper Not Working
Ok, just some compete with all, and some dont, thanks for the reply.
- martin@rootjazz
- Site Admin
- Posts: 34712
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Email Scraper Not Working
What do you think it won't compete eventually?
Re: Email Scraper Not Working
Complete*
some BIG scapes finish with all results, and some dont.
some BIG scapes finish with all results, and some dont.
- martin@rootjazz
- Site Admin
- Posts: 34712
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Email Scraper Not Working
Don't suppose you still have the logs?
Or you performed them within the last 7 days so you could submit logs?
Or you performed them within the last 7 days so you could submit logs?
Re: Email Scraper Not Working
Could be somewhere in my logs folder. But on with other stuff now, keep up the amazing work. I love your stuff. Glad you never started the vine bot. Haha. Vine = R.I.P.....