Having some trouble with the search scraper

[PrismaPhonic]

Thank you! If you still want to see my logs I submitted them a while ago and put the log ID in one of my earlier posts.

Here it is again:

13598

[PrismaPhonic]

I just updated to 3.0.5.5 and tested the Scrape Followers Of feature.

ISSUE 3
Scrape Lists tab
Scrape Followers of Box
enter value: https://soundcloud.com/20syl
set Artist Filter:

min track 1, max track 300, min followers 50, max followers 1,000, min comments 0, max comments 9999999, min followings 30, max followings 1500, min favorites 1, max favorites 9999999, user active in last 30 days, has avatar box checked.

problem is: 0 results.

Logs:

Processing search: https://soundcloud.com/20syl scrape_followers

That's ALL that shows up when I run the test and it shows the action as "complete."

There is NO file saved.

[martin@rootjazz]

Did the action complete?


Will test again

[martin@rootjazz]

is it really worth filtering

max comments 9,999,999
Max favorites 9,999,999

you are trying to set not limited? If so - 0 will do, or do you want to filter out >10 millions?

[martin@rootjazz]

I run the action and processing action runs correctly, showing the logs of what is recorded / filtered


If you restart the bot and try again - do you still have the same issue?

[PrismaPhonic]

is it really worth filtering

max comments 9,999,999
Max favorites 9,999,999

you are trying to set not limited? If so - 0 will do, or do you want to filter out >10 millions?

Oh ok, thank you! I didn't know that. I thought that an upper limit HAD to be set, so I was setting an arbitrarily high number for the upper limits.

Alright, I restarted the program and ran the search again, this time replacing my upper limits with zero and it looks like it's now working!


Now, I notice that it doesn't scrape all of the followers and I saw you post in the forum that soundcloud imposes a restriction of 8,000 maximum for scraping. Is it possible to set up a feature so proxies are rotated such that if a user supplies enough good proxies all of an artists followers will eventually be scraped in one go?

So like, maybe Proxy 1 scrapes the first 8K, then proxy 2 takes over and scrapes the next 8K etc.?

[martin@rootjazz]

not possible. SC show the same 8000. All searches / scrapes have an 8000 limit.


As an account gets new followers, they will be showed first, so if you scrape again in a week, you may find a few new users, but you will never be able to get all