Best practices: Search Follow / Like / DM

[Alexiom]

Hi Team,

I'm using slave accounts to do actions on twitter (follow, like & DM) to push traffic to a main acct.

These processes are all GRINDING extremely slowly. Watching the logs, there is a lot going on, for example, the DM process seems to be scraping the tweets of each target and then "ignoring retweets" even though I have no conditions set for tweet activity.

I've submitted logs 68594.

The setup for all 3 actions is similar. Select my slave accounts. Point them at a public list of about 130 accounts I want to emulate, and have them do actions against the followers of those target accounts. Some questions:

1) In a custom search to get 1k targets, would you be better to pull 10 followers from 100 targets, or 100 followers from 10 targets? Which would be more efficient?
2) If I wanted 100 accounts to all do 10 actions, can I simply select them all, "share across accounts", and search for at least 1k targets? At what kind of numbers does this process become inefficient? Can I tell 500 accounts do a combined 50000 actions in one job?
3) Any other best practices or tips to get these jobs running smoothly? Right now running 24/7 on 20+ threads I can even get all of my slaves to do a handful of actions each day. I'm also using a TONE of network data, like 15gb yesterday according to my proxy provider.

Any tips would be much appreciated!

[Alexiom]

Still testing and trying to get to the bottom of this.

The "Search Direct Message" module is running extremely slowly, it takes several hours to send 20 DM's (2 each on 10 accounts).

When I just use the search/scraper to scrape 20 profiles with the same filters it still takes some time, but only about 40 minutes.

I'm providing custom search a list from Twitter. Telling it to pull 2 random accounts, and then 10 random followers from each account to send the DM's to.

I think I worked out already that the "filter" applies to the custom search, so the custom search is only returning results that match the filter, originally I thought I needed custom search to bring back a pool of results far larger which would then be filtered down.

I'm filtering followers (10-2k), following 10-5k), min account age (30d), gender (M or unknown), ignore default av, ignore follows me, ignore verified, ignore private, and can DM.

I'm aware that's a lot of filters but want to get rid of the obvious bots, incorrect genders, and inactive accounts. Does anything stand out there as likely crushing my results and making the DM process take so long?

Also... is a DB maintained of which target accounts have already been DM'd by any of my accounts? I've noticed that several of my accounts have DM'd the same targets even tho I have "Share Across Accounts" and "Unique Items" checked. Possibly the targets are not unique across different jobs?

Could someone please confirm what exactly "Unique items" and "Profiles per DM" mean on the Search Direct Message control?

Any help much appreciated!

[martin@rootjazz]

These processes are all GRINDING extremely slowly. Watching the logs, there is a lot going on, for example, the DM process seems to be scraping the tweets of each target and then "ignoring retweets" even though I have no conditions set for tweet activity.

How many accounts?
How many items you request to process?
Are you SURE you did not set a filter? If you did not, then the only reason items are ignored for DMs are:
* you've processed them before (if turned UNIQUE on)
* the profile does not allow you to DM them

The setup for all 3 actions is similar. Select my slave accounts. Point them at a public list of about 130 accounts I want to emulate, and have them do actions against the followers of those target accounts. Some questions:

So you set a search with:
* file input of 130 profiles
* followers

Total items wanted? How many followers per profile?

1) In a custom search to get 1k targets, would you be better to pull 10 followers from 100 targets, or 100 followers from 10 targets? Which would be more efficient?

Probably 100 followers from 10 targets as each page of followers should give you 40 results, so 3 page requests and you get 100
with 10 from 100, you need to do 10 page requests to get a 100

2) If I wanted 100 accounts to all do 10 actions, can I simply select them all, "share across accounts", and search for at least 1k targets? At what kind of numbers does this process become inefficient? Can I tell 500 accounts do a combined 50000 actions in one job?

The program will fulfil the search+filter requirement first. so it may take a while to grab 50k results. More time than if you wanted to do 5 actions with a 100 accounts each

3) Any other best practices or tips to get these jobs running smoothly? Right now running 24/7 on 20+ threads I can even get all of my slaves to do a handful of actions each day. I'm also using a TONE of network data, like 15gb yesterday according to my proxy provider.

threading only comes into play once you have scraped the profiles. As searching has to happen single threaded.

Twitter is quite data heavy on the network of the results it sends back, 15gb may be reasonable or not, depends what you are doing.


One thing to say, if you are using a FILTER, when you request 40 pages of results, each result needs to be checked over, one by one to see if it passes the filter. If your goal is simply throughput, get rid the filter and just blast everyone. Filtering will SEVERELY slow things down

[martin@rootjazz]

Still testing and trying to get to the bottom of this.

The "Search Direct Message" module is running extremely slowly, it takes several hours to send 20 DM's (2 each on 10 accounts).

This might be correct, or indicate a bug, impossible to say.
What search
what filtering.

But simply, it shouldn't.

1 search will find 20 results.
Then send the DM with zero pause, lets say it takes 30 seconds.

Shouldn't be more than 20 minutes for 10 accounts to post twice. Assuming no issue with proxies, no issues with accounts having to verify, no extreme filter.


right click the action > select RAW LOGS >

email me the file that opens

Code: Select all

support[at]rootjazz[dot]com

When I just use the search/scraper to scrape 20 profiles with the same filters it still takes some time, but only about 40 minutes.

ok, so it's the filters, to get 20 results, if it's taking 40 minutes, your searches are not a good fit, or your filter is too strict. Find a better search that gives better results to meet your filter, or reduce your filters.

The same search in the SCRAPER module, or any action module, runs exactly the same, it's the same code.


I'm providing custom search a list from Twitter. Telling it to pull 2 random accounts, and then 10 random followers from each account to send the DM's to.

I'm filtering followers (10-2k), following 10-5k), min account age (30d), gender (M or unknown), ignore default av, ignore follows me, ignore verified, ignore private, and can DM.

I'm aware that's a lot of filters but want to get rid of the obvious bots, incorrect genders, and inactive accounts. Does anything stand out there as likely crushing my results and making the DM process take so long?

filtering takes a while. Each results must have it's end point pulled, but as above, right click the action > RAW LOGS and I can go through them. Everything you are filtering should be in the USERDETAILS endpoint.

Also... is a DB maintained of which target accounts have already been DM'd by any of my accounts? I've noticed that several of my accounts have DM'd the same targets even tho I have "Share Across Accounts" and "Unique Items" checked. Possibly the targets are not unique across different jobs?

To not process duplicate items across multiple accounts, you have multiple options.

1) SHARED ACTIONS.
When you run a SHARED ACTION, the "already processed" lists for each account in the shared action is used to ensure, no item is processed if any of the accounts in the shared action have already processed the item. Additionally, the items that are processed in the shared action are recorded by all accounts and not processed by any account (in the shared action) in the future (even if an account is running a single account action)


2) Manual MERGE PROCESSED ITEMS.
On the action tabs (eg follow tab) you can select multiple accounts and select MERGE PROCESSED ITEMS. This will run a function to merge the processed items lists of all selected accounts at that point in time. Meaning no item processed by any account in the merge *already* will be processed again. This is a one-off merge, meaning the processed lists will diverge as you run more separate actions in future. Please note, if you perform this on the follow tab, only the followed processed items are merged.

3) Global processed logs - (most of the time people want this option)
Again, on the action tabs (eg follow tab) select your accounts, right click GLOBAL PROCESSED LOGS > ASSIGN (add)
What this does is sync all select accounts together their processed lists. This means the synced accounts will never process duplicate items regardless of what type of actions is used (shared / single). The difference between (2) "manual merge" is that this is a permanent sync, there will never be any duplication. Again this only applies to the action tab applied to (so only follow items if performed on the follow tab, only like items if performed on like tab)

viewtopic.php?f=15&t=11027

Could someone please confirm what exactly "Unique items" and "Profiles per DM" mean on the Search Direct Message control?

unique is per action
profiles per DM, if you set 5, you will send a group DM with 5 people in.




Regards,
Martin

[Alexiom]

Thanks for the great info Martin. Have implemented some of your advice and sent you an email with updated results, logs, and screenshots.

[Alexiom]

Poor proxies I think... switched to a mix of fast LTE and datacentre proxies and the process runs 10's of times faster.

I was using cheap residential for all accts previously. Twitter seems to have no issues with datacentre ipv4 proxies

[martin@rootjazz]

I was using cheap residential for all accts previously. Twitter seems to have no issues with datacentre ipv4 proxies

There is no system to it sometimes, it's just random. If it was as simple as "just use XYZ" it would be much easier. But in 2023 there is definitely some trial and error to find something that works, and if it works, it works.

[Alexiom]

Hi Martin.

If using a custom search to scrape targets from a list such as: ListMembers x10, FollowersOf x40 = 400 targets.

Does the tool likely page the same (or almost the same) list members every time the same query is used? Except for some churn when I add new list members or they gain new followers.

Assuming that's a problem if I cap the job to run 400 times still, but increase it to say 50 ListMembers and randomize them before the 2nd step, will that dramatically slow the process down? Or will it stop scraping once it has enough targets?

Thank you!

[martin@rootjazz]

Hi Martin.

If using a custom search to scrape targets from a list such as: ListMembers x10, FollowersOf x40 = 400 targets.

Does the tool likely page the same (or almost the same) list members every time the same query is used? Except for some churn when I add new list members or they gain new followers.

if you run the search again, it will start at the beginning and scrape latest new results

Assuming that's a problem if I cap the job to run 400 times still, but increase it to say 50 ListMembers and randomize them before the 2nd step, will that dramatically slow the process down?

Too specific to say, try it and see

Or will it stop scraping once it has enough targets?

yes, if you want a total of 400 results

yet tell the steps:
List members = 100
followers = 100

so 10k total, the searching stops once you have MAX ITEMS = 400



Regards,
Martin