Scrape Lists

Discussions to do with Soundcloud Manager. Do not use for support, use the dedicated support forum for help requests
Post Reply
zenithventures7
Posts: 2
Joined: Mon Dec 03, 2018 6:15 am

Scrape Lists

Post by zenithventures7 »

Hi, when trying to scrape the likes or reposts it sets the Max Items to 8000 by default. Where can I change this? Also when the list is big, because it doesn't continuously save the scraped users I lost the data many a times. Is there a way to change that?
User avatar
martin@rootjazz
Site Admin
Posts: 34369
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Lists

Post by martin@rootjazz »

8000 used to be a Soundcloud limit, I don't think the program forces this (although it might). Can you show logs showing this?
because it doesn't continuously save the scraped users I lost the data many a times. Is there a way to change that?
No way to change it. However, I will note it as a feature suggestion for a future potential update
zenithventures7
Posts: 2
Joined: Mon Dec 03, 2018 6:15 am

Re: Scrape Lists

Post by zenithventures7 »

Starting: 15/12/2018 23:58 PM
Search: https://soundcloud.com/marshmellomusic/projectdreams
Max: 0
Processing search: https://soundcloud.com/marshmellomusic/projectdreams search_likers
Max: 0 globalmax: 20000
Max items not specified: setting to 8000
Page scraped
Current Results: 0
Results : 60 https://soundcloud.com/marshmellomusic/projectdreams
Page scraped

Not able to scrape anything more than 8k results except for the Scrape Followers option where I can specify recursive count too. What am I missing? Thank you!
User avatar
martin@rootjazz
Site Admin
Posts: 34369
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Lists

Post by martin@rootjazz »

Let me run some tests, but I think maybe:
Max: 0
this is causing some limiter to come in. Always try and specify what you want, even if is is a huge number: 9999999999999999999 for example, to avoid any limits kicking in.

Same for CUSTOM SEARCH / SEARCH BUILDER. for each specify a limit for the results you want at each step to avoid the program making decisions for you.

But I think there is some old code runing, which is setting the limit to 8000 even though you have a global limit. I'll need to run tests to confirm and find out
Post Reply