Scrape Lists
-
- Posts: 2
- Joined: Mon Dec 03, 2018 6:15 am
Scrape Lists
Hi, when trying to scrape the likes or reposts it sets the Max Items to 8000 by default. Where can I change this? Also when the list is big, because it doesn't continuously save the scraped users I lost the data many a times. Is there a way to change that?
- martin@rootjazz
- Site Admin
- Posts: 34640
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scrape Lists
8000 used to be a Soundcloud limit, I don't think the program forces this (although it might). Can you show logs showing this?
No way to change it. However, I will note it as a feature suggestion for a future potential updatebecause it doesn't continuously save the scraped users I lost the data many a times. Is there a way to change that?
-
- Posts: 2
- Joined: Mon Dec 03, 2018 6:15 am
Re: Scrape Lists
Starting: 15/12/2018 23:58 PM
Search: https://soundcloud.com/marshmellomusic/projectdreams
Max: 0
Processing search: https://soundcloud.com/marshmellomusic/projectdreams search_likers
Max: 0 globalmax: 20000
Max items not specified: setting to 8000
Page scraped
Current Results: 0
Results : 60 https://soundcloud.com/marshmellomusic/projectdreams
Page scraped
Not able to scrape anything more than 8k results except for the Scrape Followers option where I can specify recursive count too. What am I missing? Thank you!
Search: https://soundcloud.com/marshmellomusic/projectdreams
Max: 0
Processing search: https://soundcloud.com/marshmellomusic/projectdreams search_likers
Max: 0 globalmax: 20000
Max items not specified: setting to 8000
Page scraped
Current Results: 0
Results : 60 https://soundcloud.com/marshmellomusic/projectdreams
Page scraped
Not able to scrape anything more than 8k results except for the Scrape Followers option where I can specify recursive count too. What am I missing? Thank you!
- martin@rootjazz
- Site Admin
- Posts: 34640
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scrape Lists
Let me run some tests, but I think maybe:
Same for CUSTOM SEARCH / SEARCH BUILDER. for each specify a limit for the results you want at each step to avoid the program making decisions for you.
But I think there is some old code runing, which is setting the limit to 8000 even though you have a global limit. I'll need to run tests to confirm and find out
this is causing some limiter to come in. Always try and specify what you want, even if is is a huge number: 9999999999999999999 for example, to avoid any limits kicking in.Max: 0
Same for CUSTOM SEARCH / SEARCH BUILDER. for each specify a limit for the results you want at each step to avoid the program making decisions for you.
But I think there is some old code runing, which is setting the limit to 8000 even though you have a global limit. I'll need to run tests to confirm and find out