Multisearch flagging accounts?

Support / help / discussion forum for twitter bot
jamberry
Posts: 73
Joined: Sat Nov 09, 2019 6:18 pm

Re: Multisearch flagging accounts?

Post by jamberry »

Okay.. In this single process I have 468 searches and 300 accounts and 10 threads - if I understand it right it means 10 accounts (threads) are going to scrape all 468 searches for specified amount of results and then they start following?

This is no way - its extremely slow - its currently scraping for 3 hours (NOT stucked) and followed 3 users.. LOG ID: 14942 - its going to take days before all accounts finish scraping.

Also Twitterdub is using during this scraping only 1-5% of CPU... please is there any way how to do it faster? Any way how to make all accounts scraping? I need to follow every day about 12k people in total.. its impossible to reach it if scraping is going to take this long..

EDIT: Maybe run 3 instances of Twitterdub? Each has 100 threads.. so 100 accs per instance and this way I could run all accs at once? Or create 30 multi search processes - each with 10 accounts.. what would be better?
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Multisearch flagging accounts?

Post by martin@rootjazz »

jamberry wrote: Wed Jan 08, 2020 11:31 am Okay.. In this single process I have 468 searches and 300 accounts and 10 threads - if I understand it right it means 10 accounts (threads) are going to scrape all 468 searches for specified amount of results and then they start following?
No :)

A MULTI SEARCH can be thought of as 468 separate actions, stored in a single action that will process consecutively.

so search1 will search, then process results and it is the results that are threaded (the searches within the multisearch are NOT threaded)
then search2 runs, gets the results then processes results
....

This is no way - its extremely slow - its currently scraping for 3 hours (NOT stucked) and followed 3 users.. LOG ID: 14942 - its going to take days before all accounts finish scraping.
Are you sure you are just not looking at the logs, and seeing the results from search_123 after 3 hours, seeing the (n) follows from that action and assuming that is all that has happened? The real logs will only show the last (n) lines of logs, click RAW LOGS to view the days full logs
Also Twitterdub is using during this scraping only 1-5% of CPU... please is there any way how to do it faster? Any way how to make all accounts scraping? I need to follow every day about 12k people in total.. its impossible to reach it if scraping is going to take this long..
It is not CPU intensive, it is just some HTTP requests and storing a list.

If you want to speed it up, use multiple actions. so 50 accounts per action (so 1/6)and give each action 468/6 inputs
EDIT: Maybe run 3 instances of Twitterdub? Each has 100 threads.. so 100 accs per instance and this way I could run all accs at once? Or create 30 multi search processes - each with 10 accounts.. what would be better?
Test and see if you are happier with the results ultimately. As you seem to want speed above all else, then 30 actions with 10 accounts would probably be fastest.
jamberry
Posts: 73
Joined: Sat Nov 09, 2019 6:18 pm

Re: Multisearch flagging accounts?

Post by jamberry »

martin@rootjazz wrote: Wed Jan 08, 2020 8:35 pm so search1 will search, then process results and it is the results that are threaded (the searches within the multisearch are NOT threaded)
then search2 runs, gets the results then processes results
And thats it - Im using relatively very strict filters so sometimes scraping of some users can take long -> if search2 will wait until search1 is finished it can take a lot of time... thats the reason why I need multiple accounts work simultaneously and independently on each other
Are you sure you are just not looking at the logs, and seeing the results from search_123 after 3 hours, seeing the (n) follows from that action and assuming that is all that has happened? The real logs will only show the last (n) lines of logs, click RAW LOGS to view the days full logs
Yes im sure because when I canceled that process after 3 hours it showed me in "Results" tab that 3 users has been followed.
Test and see if you are happier with the results ultimately. As you seem to want speed above all else, then 30 actions with 10 accounts would probably be fastest.
Its not like I want speed above all else - I just need all accounts to work as set and not have X accounts waiting till Y accounts finishes scraping and following and preactions.

Thank you very much for your time and explaining all that stuff to me - I really appreciate your help. And sorry for so many questions. I will try to play with settings for now and hopefully find some sweet spot.
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Multisearch flagging accounts?

Post by martin@rootjazz »

jamberry wrote: Thu Jan 09, 2020 9:16 am
martin@rootjazz wrote: Wed Jan 08, 2020 8:35 pm so search1 will search, then process results and it is the results that are threaded (the searches within the multisearch are NOT threaded)
then search2 runs, gets the results then processes results
And thats it - Im using relatively very strict filters so sometimes scraping of some users can take long -> if search2 will wait until search1 is finished it can take a lot of time... thats the reason why I need multiple accounts work simultaneously and independently on each other
If you are using strict filters then this is the cause.

When a search request is made, one request can receive 100 results.
With a filter, the program needs to make an ADDITIONAL 1 request per result, so instead of 1 request, the program has to make 101 requests now.

Also, if your filter is strict, you may be ignoring 20% 50% or even 99% of results (if your search and filter combination is not well thought out - the search is just not a good search for your filter).

If searching is taking a long time, THIS is the cause. You can:

1) accept the long time searching
2) remove / loosen the filter
3) run tests and find searches that return results that meet your filter with a good percentage
Are you sure you are just not looking at the logs, and seeing the results from search_123 after 3 hours, seeing the (n) follows from that action and assuming that is all that has happened? The real logs will only show the last (n) lines of logs, click RAW LOGS to view the days full logs
Yes im sure because when I canceled that process after 3 hours it showed me in "Results" tab that 3 users has been followed.
Look at the raw logs, that is the only way to be sure.
Test and see if you are happier with the results ultimately. As you seem to want speed above all else, then 30 actions with 10 accounts would probably be fastest.
Its not like I want speed above all else - I just need all accounts to work as set and not have X accounts waiting till Y accounts finishes scraping and following and preactions.

Thank you very much for your time and explaining all that stuff to me - I really appreciate your help. And sorry for so many questions. I will try to play with settings for now and hopefully find some sweet spot.
As above, run tests. Try the multi action WITHOUT a filter. Maybe you still get good results without filtering. Perhaps with a filter you get 50% conversions, but it takes a LONG time. Without a filter, maybe you get 10% conversions, but due to the increased speed, that is a better.

If you want to keep the filter and have more things running at the same time, then multiple actions is the way to go. As I said multi search are consequtive searches > processing that happens one by one.

Splitting up to 10 / 30 / (n) actions will allow you to thread the processing actions, which will be a speed up. Which is different to the multi-searching action threads, which will just thread your accounts over the results - which won't be that much as IIRC you are pulling 5-10 results per search, so 100 threads there isn't go to do anything as you don't have 100 results per search to thread over.
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Multisearch flagging accounts?

Post by martin@rootjazz »

Thank you very much for your time and explaining all that stuff to me - I really appreciate your help. And sorry for so many questions. I will try to play with settings for now and hopefully find some sweet spot.
No problem, I am hear to answer your questions, so please do ask questions if you don't understand something. And if my responses do not make sense, please keep asking. Also, if you think my answers are wrong, please let me know as it is possible I misunderstood the question



Regards,
Martin
Post Reply