Issue with "Share Follows" Over army of 1k accounts

Discussions to do with Soundcloud Manager. Do not use for support, use the dedicated support forum for help requests
Post Reply
Happynewnow
Posts: 1
Joined: Fri Nov 02, 2018 12:21 am

Issue with "Share Follows" Over army of 1k accounts

Post by Happynewnow »

Hi there,

First off, I've been enjoying this amazing software. It's been running very smoothly so far. Thank you for creating this...

Everything has been running smoothly so far.

However, I do have this one issue I'd like to get to the bottom of...

Most likely its more an issue on my side,(settings) not an issue with the software.

Basically, I have an army of over 1k accounts, I'd like each account to follow at least 50/day active followers from a list of usernames.

When I run the follow action for all accounts (even with 100 threads) the process takes over a week to process...

In order for me to process 50 follow actions per account, I can only check 100 accounts at a time with 100 threads, then it takes about 6 hours to run.

However this is kind of a hassle, it should be able to process all 1k accounts with over 100 threads... With 100 threads and my dedicated server it shouldn't take more than half a day to process...

Here are my settings:
I have all 1,000 accounts checked
Follow followers of: permalink added with user with over 300k followers
a total of 50,000 set for #Max follow (50 per account)
random pause of 10 to 20 secs
repeat every 24 hrs
stop at 1800
all boxes check under that favorite, repost, share follows over all accounts
# of threads 100

Anything you see here that I'm doing wrong? Or could do better?

Thank you very much in advance for the help...
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Issue with "Share Follows" Over army of 1k accounts

Post by martin@rootjazz »

Happynewnow wrote: Fri Nov 02, 2018 12:54 am
Basically, I have an army of over 1k accounts, I'd like each account to follow at least 50/day active followers from a list of usernames.

When I run the follow action for all accounts (even with 100 threads) the process takes over a week to process...
ouch, that isn't very good. In fact is terrible.

Without seeing your logs it is hard to say what is the actual cause. But at a guess I would say the SEARCH is the bottleneck
If you are running filters as well, that will delay things. Specifically when you need 50k results.

Also your searches will effect things.

For example, due to the infinite scroll search used by SC, the search most start at the beginning each time.

So first time you run the search, the program can return results: 0-50,000
Second run, it needs to scroll through 50,000 results so it can return 50,001 - 100,000
after 10 runs, it will need to scroll through 500,000 results so can return 500,001-550,000

So if you can detail your search for me that would help if we can rule this in / out as a possible cause.

When you need big numbers, a custom search is best returning results that would differ each time

something like
posted track with tag X in last 24 hours (1000)
followers (10000)

The (x) numbers above as telling the custom search how many to pull per step, this means the above would try and pull 10 million, however, the program will stop once it has your 50k, it just means we try and pull as many as we can from each step to get your max_items as quickly as possible

scraping followers is a good search as the program can get a large amount of results per request, about 1000 IIRC, whereas a tag search will only return 10 per request

HOWEVER as above, if you have a filter, each result, must have it's user_details pulled, which is one request per user



The above should pull your 50k results fairly quickly (with no filter).

Another way to improve speed, would be to divide your 1000 accounts into 10.
So have 10 actions, each with 100 accounts, so you only need 5000 results per action


In order for me to process 50 follow actions per account, I can only check 100 accounts at a time with 100 threads, then it takes about 6 hours to run.
ah, probably should have read this before my reply above :)
However this is kind of a hassle, it should be able to process all 1k accounts with over 100 threads... With 100 threads and my dedicated server it shouldn't take more than half a day to process...
True, but the search stage is single threaded, the search cannot request the next page of results in a multi threaded way as they don't exist yet.
scrape page
get next page id (a)
scrape page with id (a)
get next page id(b)
scrapepage with id (b)

etc


What would help, would be the possibility of specifying multiple searches for a SHARED action, ie 100 searches, then these could be threaded. But this functionality doesn't exist in the app and would not be a quick and simple add, but a rewrite of the full search mechanism in order to have threaded search communication.

I understand how it can be a hassle for you to set up 10 actions compared to 1, but at this time it is probably your best option. Set the actions to repeat every 24 hours and you will only have to set them up once. Specify the inputs as a spintax block with a selection of 10 options and that should be enough variation for you to just leave it running and forget about it

Here are my settings:
I have all 1,000 accounts checked
Follow followers of: permalink added with user with over 300k followers
ok so you search is going to require paging each run, but as can only run 6 times, it still shouldn't take a week!

Anything you see here that I'm doing wrong? Or could do better?
Even now knowing what you are doing, taking a week is just wrong, it does seem to imply the program is not doing something correcetly. Send me your logs and I will take a look and see what I can find

HELP > LOGS > SUBMIT

then send your logs ID - the first 4 numbers is sufficient (displayed after successful uploading of logs)




Regards,
Martin
Post Reply