Duplicate DM Actions

Support / help / discussion forum for twitter bot
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Duplicate DM Actions

Post by Alexiom »

Hi Team,

Can anyone assist with troubleshooting duplicate actions? I'm using the mother/slave method sending DM's from slave accounts.

I'm getting a lot of reports and complaints back from target accounts that they've received DM's from sometimes dozens of my slave accounts.

I have global logs added to all accounts for follow, like and DM actions.

Doing some investigation and comparing logs between two instances of TwitterDub. In this (my main) instance I've sent probably 100x more DM's but the DM Global log file is very small, only 15.4kb. In my test instance only having sent a few K DM's in total, the global DM
log is 60kb. Also, in my test instance, I frequently see in the logs "item was already processed", I'm not seeing that in my main instance.

So to my untrained eye, it seems that the global logs are not being populated. As far as I can tell all settings are the same, I always select "Unique Items" and I run each account's DM process as its own job.

And guidance on troubleshooting would be much appreciated.
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Duplicate DM Actions

Post by Alexiom »

I ran "Convert Sharedto Global" and it generated quite a large global log file for DM's. Re-testing now.
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Duplicate DM Actions

Post by Alexiom »

So set up a DM test using custom search to return 10 records max, and set the max actions for the job to 10 also. The thinking was, there SHOULD be some accounts that have already been processed, so I'd see less than 10/10 success in the completed jobs.

Ran this job across many accounts and checked the logs.

Every single account I found that completed less than 10/10 actions retrieved about 151k processed items in the first lines of the raw logs, eg:

"Loaded previous db logs: 151340 processed: 602 within 24hrs: 10"

Every single account I found that completed all 10/10 actions, not finding any previously processed items returned about 4k processed items in the first lines of the raw logs, eg:

Loaded previous db logs: 3932 processed: 617 within 24hrs: 11

So it seems some accounts are pulling the global logs, and some are not.

I checked all of the accounts that did not skip any processed items, and probably sent duplicate DM's, and they all have both global DM logs assigned (the original log, and the one I created with Coverd Shared to Global)

So... re-rand one of the accounts that didn't pull all logs, this time it only pulled 800:

Loaded previous db logs: 876 processed: 627 within 24hrs: 20

Seems like logs were purged, but I'm not sure, I see this in the logs:

Finish removing already processed: results :10 removed already processed: 0:

Any suggestions on where to look next to get these global logs working correctly and avoid duplicates?

PS: I always have "unique items" checked when configuring the DM job.
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Duplicate DM Actions

Post by Alexiom »

Logs submitted: 41644
User avatar
martin@rootjazz
Site Admin
Posts: 34390
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Duplicate DM Actions

Post by martin@rootjazz »

Alexiom wrote: Fri May 26, 2023 3:19 pm
Doing some investigation and comparing logs between two instances of TwitterDub. In this (my main) instance I've sent probably 100x more DM's but the DM Global log file is very small, only 15.4kb. In my test instance only having sent a few K DM's in total, the global DM
log is 60kb. Also, in my test instance, I frequently see in the logs "item was already processed", I'm not seeing that in my main instance.
it sounds like the GLOBAL LOG is not working.

goto DM tab
select / check all accounts
right click
GLOBAL PROCESS LOGS > APPEND

add a new ID to assign to all accounts.

Run a DM action (just a test, 2-3 items)


goto:
%appdata%\rootjazz\twitterdub\database\dbloggeddata\

you should see the file

<global_id>.bin

check the file size

run another DM action (again a test 2-3 items)

you should see the global log grow
User avatar
martin@rootjazz
Site Admin
Posts: 34390
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Duplicate DM Actions

Post by martin@rootjazz »

Alexiom wrote: Fri May 26, 2023 3:47 pm I ran "Convert Sharedto Global" and it generated quite a large global log file for DM's. Re-testing now.
ok this may help, it will take all the logs from the selected accounts and create a new global log for them, removing any duplicates from the account_specific_processed_log.

it's more an efficiency function, but it could help...
User avatar
martin@rootjazz
Site Admin
Posts: 34390
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Duplicate DM Actions

Post by martin@rootjazz »

Alexiom wrote: Fri May 26, 2023 11:56 pm Logs submitted: 41644
your logs are massive, I've not going to find anything there.

clear existing logs:

HELP >LOGS >DELETE


Run a DM action with one account that is not apparently loading all the previous records. Just process 1-2 items

then send me the RAW LOGS

right click the action > RAW LOGS > copy and send me the contents of the file that opens.

Code: Select all

support[at]rootjazz[dot]com

and:
HELP > LOGS > SUBMIT LAST DAY

then send your logs ID - the first 4 numbers is sufficient (displayed after successful uploading of logs)
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Duplicate DM Actions

Post by Alexiom »

Hi Martin, I've done a ton of testing and recorded everything. Sorry for the massive reply, but hope the info is helpful:

1) Deleted existing raw logs

2) Confirmed there are two existing global DM logs, the original, and the one that got added when I ran "Convert Shared to Global"

3) Checked size of both log files (DM_Logs = 78kb and auto = 49kb)

4) Had 5 accts send 5 DM's each: 5 custom search results, 5 max actions, unique items selected, 1 job per acct.

This ran much faster than expected, I confirmed that all filters were set as usual.
No actions were skipped, and no previously processed items were found.

5) Checked global log sizes: Now DM_Logs = 80kb and auto = 51kb

6) Added new global log to all accts (now 3 each)

7) Ran same DM action on same 5 accts.

8) Checked global log sizes, now DM_Logs = 82kb, auto = 53kb, DM_Logs2 = 3kb (all growing)

9) Checked how many previously processed jobs were loaded in all raw logs for both runs:

3832 -> 3992
4077 -> 4237
3959 -> 4119
3878 -> 4038
4116 -> 4264 (This acct had previously been returning 150k+ processed items)

These numbers all seem very small, I've sent many thousands of DM's across all accts, but much less than 4k on each acct.

10) Submitted logs: 31806

11) Since the number of previously processed items seems very low, I ran "Convert Shared to Global" again across my 810 active accounts and this time got a global log of 3.5mb!!! Now all accts have 4 global logs.

12) For a 3rd time, ran the same DM process sending 5 DM's from the same 5 accts.

This pulled in WAYYY more "previous db logs"

3832 -> 3992 -> 85353
4077 -> 4237 -> 85598
3959 -> 4119 -> 85480
3878 -> 4038 -> 85399
4116 -> 4264 -> 85353

All jobs are still completed without finding a duplicate processed item.

Unlike the first time I converted shared to global, the new global log did not shrink when I ran the jobs. Last time I got a similar size file, about 3.5 MB, then sent my DM's, and the global log had shrunk to a few KB, like all of the previously processed items were blown away.

I've been running 1 job per acct, so 800 + jobs, each job sends just a few DM's but repeats 5 times. I run 150+ threads in the processor. I'm wondering if so many parallel jobs all trying to update the same log file could corrupt it? Maybe I should use "Share Across Accounts" to run fewer parallel jobs?

13) Submitted logs again: 99119

11) Emailed you 15x raw logs and before and after snapshots of the 4 global logs.

Hope this is useful. Thanks again.
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Duplicate DM Actions

Post by Alexiom »

Hi Martin,

Quick update, things were running fine until I tried to run 20 parallel threads (1 acct per thread sending DM's). Within a few minutes something happened and shrunk the newest log file down to 22.7 KB (from 3541 KB).

So possibly there is an issue with parallel threads clashing...
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Duplicate DM Actions

Post by Alexiom »

Resubmitted logs just after I found the above issue: 27187
Post Reply