Thank you! If you still want to see my logs I submitted them a while ago and put the log ID in one of my earlier posts.
Here it is again:
13598
Having some trouble with the search scraper
-
- Posts: 66
- Joined: Thu Dec 03, 2015 5:36 am
-
- Posts: 66
- Joined: Thu Dec 03, 2015 5:36 am
Re: Having some trouble with the search scraper
I just updated to 3.0.5.5 and tested the Scrape Followers Of feature.
ISSUE 3
Scrape Lists tab
Scrape Followers of Box
enter value: https://soundcloud.com/20syl
set Artist Filter:
min track 1, max track 300, min followers 50, max followers 1,000, min comments 0, max comments 9999999, min followings 30, max followings 1500, min favorites 1, max favorites 9999999, user active in last 30 days, has avatar box checked.
problem is: 0 results.
Logs:
There is NO file saved.
ISSUE 3
Scrape Lists tab
Scrape Followers of Box
enter value: https://soundcloud.com/20syl
set Artist Filter:
min track 1, max track 300, min followers 50, max followers 1,000, min comments 0, max comments 9999999, min followings 30, max followings 1500, min favorites 1, max favorites 9999999, user active in last 30 days, has avatar box checked.
problem is: 0 results.
Logs:
That's ALL that shows up when I run the test and it shows the action as "complete."Processing search: https://soundcloud.com/20syl scrape_followers
There is NO file saved.
- martin@rootjazz
- Site Admin
- Posts: 34712
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Having some trouble with the search scraper
Did the action complete?
Will test again
Will test again
- martin@rootjazz
- Site Admin
- Posts: 34712
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Having some trouble with the search scraper
is it really worth filtering
max comments 9,999,999
Max favorites 9,999,999
you are trying to set not limited? If so - 0 will do, or do you want to filter out >10 millions?
max comments 9,999,999
Max favorites 9,999,999
you are trying to set not limited? If so - 0 will do, or do you want to filter out >10 millions?
- martin@rootjazz
- Site Admin
- Posts: 34712
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Having some trouble with the search scraper
I run the action and processing action runs correctly, showing the logs of what is recorded / filtered
If you restart the bot and try again - do you still have the same issue?
If you restart the bot and try again - do you still have the same issue?
-
- Posts: 66
- Joined: Thu Dec 03, 2015 5:36 am
Re: Having some trouble with the search scraper
Oh ok, thank you! I didn't know that. I thought that an upper limit HAD to be set, so I was setting an arbitrarily high number for the upper limits.martin@rootjazz wrote:is it really worth filtering
max comments 9,999,999
Max favorites 9,999,999
you are trying to set not limited? If so - 0 will do, or do you want to filter out >10 millions?
Alright, I restarted the program and ran the search again, this time replacing my upper limits with zero and it looks like it's now working!
Now, I notice that it doesn't scrape all of the followers and I saw you post in the forum that soundcloud imposes a restriction of 8,000 maximum for scraping. Is it possible to set up a feature so proxies are rotated such that if a user supplies enough good proxies all of an artists followers will eventually be scraped in one go?
So like, maybe Proxy 1 scrapes the first 8K, then proxy 2 takes over and scrapes the next 8K etc.?
- martin@rootjazz
- Site Admin
- Posts: 34712
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Having some trouble with the search scraper
not possible. SC show the same 8000. All searches / scrapes have an 8000 limit.
As an account gets new followers, they will be showed first, so if you scrape again in a week, you may find a few new users, but you will never be able to get all
As an account gets new followers, they will be showed first, so if you scrape again in a week, you may find a few new users, but you will never be able to get all