Profile Scraper

[shaner4042]

Hey,

I just updated my SCM to the most recent version last night, and after doing so, I have found that the profile scraper is MUCH less effective. I am just wondering if this is due to a change you made, or maybe I have to go in and change a certain setting. Prior to the update, I would scrape users who uploaded tracks in the past 24 hours via certain key words (ie. hip hop, trap). Hip hop and trap were generally the most popular keywords, resulting in about 4000 - 5000 scraped profiles each time I ran the process; this had been consistent everyday for the past 6 months. However, when I ran the same process this morning, it is only scraping 200 - 300 profiles. When opening up the log for the process, I noticed its only scraping 1 profile at a time, whereas before it would usually scrape 20 profiles at a time. This decline in simultaneous scrapes seems to account for the drop in total scraped profiles, as it is scraping 20 times less simultaneously, and resulting in 20 times less total results.

My log ID is: 88000

I have been extremely pleased using your software to date, but this would be a huge issue for me if it continues to do this. Thanks in advance, appreciate your continued support!

[martin@rootjazz]

Do you know what version you updated from?

If it was quite an old version, then the lack of multi threading (doing many things at once) was intentional. This is because Soundcloud have got more strict with bot detections, and running many threads to scrape as quickly as possible was hypothesised as being risky for accounts (more so accounts on proxies than main accounts on local / home IPs). So it was slowed down.

But that's all it was, a slow down. It should still scrape the same amount, just take longer.

So this raises the question. Why are you getting less.

1) there was a bug before and you were getting more results, which were wrong
2) there is a bug now meaning scrape less
3) there are just less tracks uploaded
4) a limit is kicking in, stopping your scrape.


A lot of the above could be answered if I knew your previous version, but unless you happen to know it, I don't think we can find it out.

Let me check your logs first though, as perhaps there might be some info there in regards to the above

[martin@rootjazz]

Never mind, I think I have found the issue. The next update will fix this. I shall let you know when it is ready.



Regards,
Martin

[shaner4042]

Thank you!

I was fairly certain it was some sort of bug; I don't think a 20x decline in results could be justified by users uploading less tracks. That would be a huge fluctuation from one day to the next. If it is of any assistance, I updated from the previous version.

[martin@rootjazz]

https://soundcloudmanager.com/updatetesting.html

[rugboy2012]

Hey Martin,
Not sure if this is related question...So I've also downloaded the most recent version of your software, and it seems to be working fine, but my question is...I'm not sure I'm able to scrape more than about a 1000 emails for each keyword search for uses that I try (ie, rock, EDM, etc)...I thought it may be only scraping the past 30 days, so I set the "artist filter" to "365" days...but I'm still only getting the same results?...I read your post about Soundcloud making limitations...so do you have any suggestions for how to gather more results?

Log ID: 26244

Cheers,

J

[martin@rootjazz]

can you describe the EXACT searches you are making. A screenshot probably the easiest way.

.I thought it may be only scraping the past 30 days, so I set the "artist filter" to "365" days.

wouldn't change anything. The filter works on the result of the search.

So you run the search and get results 1-1000
you run the search with the filter and you get the same 1-1000 results, which are then put through the filter (which requires pulling the profile details for each results - thus 1000 requests from Soundcloud)

[martin@rootjazz]

also your logs are too huge to go through, maybe I don't need them if can find out exactly what you are doing.

But if I do:



Please perform the following:

Open the program. Delete or pause all existing actions. If actions are already running due to autorun on startup, disable autorun at startup and restart the app. All actions must be waiting, then pause all (or delete)


Delete existing logs
HELP > LOGS > VIEW
delete all files in the folder that opens


Perform the action:
***
run your scrape/s that are having this issue
***

PLEASE NOTE: Do not perform other actions at the same time. Only use one thread. Do not run any actions after this action.


Then submit logs:
HELP > LOGS > SUBMIT

and let me know the LOGS ID (just the number part is sufficient)




Regards,
Martin

[rugboy2012]

Hey Martin,
I'm sorry...As I mentioned before...I'm really PC illiterate here (I'm pretty much mac for everything), so I was unable to work out how to upload any images here (I've pm them to you)
The Log is: 71992

I'm simply put in a search for "singer-songwriters", and then put in that link into the scraper to pull email addresses, and I seem to just get the same results - and I'm wondering if there is some kind of "limit"? If it only pulls say.."1000" emails (or soundcloud profiles) a day? or something?

Sorry if these questions are really dumb....

J

[martin@rootjazz]

just a request, in future when you send an email in regards to a forum post, include the forum post link so I can read up on your issue, otherwise I have no idea of what the email is referring to.

Let me find the email and check through this thread now