Hi Team,
Can anyone assist with troubleshooting duplicate actions? I'm using the mother/slave method sending DM's from slave accounts.
I'm getting a lot of reports and complaints back from target accounts that they've received DM's from sometimes dozens of my slave accounts.
I have global logs added to all accounts for follow, like and DM actions.
Doing some investigation and comparing logs between two instances of TwitterDub. In this (my main) instance I've sent probably 100x more DM's but the DM Global log file is very small, only 15.4kb. In my test instance only having sent a few K DM's in total, the global DM
log is 60kb. Also, in my test instance, I frequently see in the logs "item was already processed", I'm not seeing that in my main instance.
So to my untrained eye, it seems that the global logs are not being populated. As far as I can tell all settings are the same, I always select "Unique Items" and I run each account's DM process as its own job.
And guidance on troubleshooting would be much appreciated.
Duplicate DM Actions
Re: Duplicate DM Actions
I ran "Convert Sharedto Global" and it generated quite a large global log file for DM's. Re-testing now.
Re: Duplicate DM Actions
So set up a DM test using custom search to return 10 records max, and set the max actions for the job to 10 also. The thinking was, there SHOULD be some accounts that have already been processed, so I'd see less than 10/10 success in the completed jobs.
Ran this job across many accounts and checked the logs.
Every single account I found that completed less than 10/10 actions retrieved about 151k processed items in the first lines of the raw logs, eg:
"Loaded previous db logs: 151340 processed: 602 within 24hrs: 10"
Every single account I found that completed all 10/10 actions, not finding any previously processed items returned about 4k processed items in the first lines of the raw logs, eg:
Loaded previous db logs: 3932 processed: 617 within 24hrs: 11
So it seems some accounts are pulling the global logs, and some are not.
I checked all of the accounts that did not skip any processed items, and probably sent duplicate DM's, and they all have both global DM logs assigned (the original log, and the one I created with Coverd Shared to Global)
So... re-rand one of the accounts that didn't pull all logs, this time it only pulled 800:
Loaded previous db logs: 876 processed: 627 within 24hrs: 20
Seems like logs were purged, but I'm not sure, I see this in the logs:
Finish removing already processed: results :10 removed already processed: 0:
Any suggestions on where to look next to get these global logs working correctly and avoid duplicates?
PS: I always have "unique items" checked when configuring the DM job.
Ran this job across many accounts and checked the logs.
Every single account I found that completed less than 10/10 actions retrieved about 151k processed items in the first lines of the raw logs, eg:
"Loaded previous db logs: 151340 processed: 602 within 24hrs: 10"
Every single account I found that completed all 10/10 actions, not finding any previously processed items returned about 4k processed items in the first lines of the raw logs, eg:
Loaded previous db logs: 3932 processed: 617 within 24hrs: 11
So it seems some accounts are pulling the global logs, and some are not.
I checked all of the accounts that did not skip any processed items, and probably sent duplicate DM's, and they all have both global DM logs assigned (the original log, and the one I created with Coverd Shared to Global)
So... re-rand one of the accounts that didn't pull all logs, this time it only pulled 800:
Loaded previous db logs: 876 processed: 627 within 24hrs: 20
Seems like logs were purged, but I'm not sure, I see this in the logs:
Finish removing already processed: results :10 removed already processed: 0:
Any suggestions on where to look next to get these global logs working correctly and avoid duplicates?
PS: I always have "unique items" checked when configuring the DM job.
Re: Duplicate DM Actions
Logs submitted: 41644
- martin@rootjazz
- Site Admin
- Posts: 34390
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Duplicate DM Actions
it sounds like the GLOBAL LOG is not working.Alexiom wrote: ↑Fri May 26, 2023 3:19 pm
Doing some investigation and comparing logs between two instances of TwitterDub. In this (my main) instance I've sent probably 100x more DM's but the DM Global log file is very small, only 15.4kb. In my test instance only having sent a few K DM's in total, the global DM
log is 60kb. Also, in my test instance, I frequently see in the logs "item was already processed", I'm not seeing that in my main instance.
goto DM tab
select / check all accounts
right click
GLOBAL PROCESS LOGS > APPEND
add a new ID to assign to all accounts.
Run a DM action (just a test, 2-3 items)
goto:
%appdata%\rootjazz\twitterdub\database\dbloggeddata\
you should see the file
<global_id>.bin
check the file size
run another DM action (again a test 2-3 items)
you should see the global log grow
- martin@rootjazz
- Site Admin
- Posts: 34390
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Duplicate DM Actions
ok this may help, it will take all the logs from the selected accounts and create a new global log for them, removing any duplicates from the account_specific_processed_log.
it's more an efficiency function, but it could help...
- martin@rootjazz
- Site Admin
- Posts: 34390
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Duplicate DM Actions
your logs are massive, I've not going to find anything there.
clear existing logs:
HELP >LOGS >DELETE
Run a DM action with one account that is not apparently loading all the previous records. Just process 1-2 items
then send me the RAW LOGS
right click the action > RAW LOGS > copy and send me the contents of the file that opens.
Code: Select all
support[at]rootjazz[dot]com
and:
HELP > LOGS > SUBMIT LAST DAY
then send your logs ID - the first 4 numbers is sufficient (displayed after successful uploading of logs)
Re: Duplicate DM Actions
Hi Martin, I've done a ton of testing and recorded everything. Sorry for the massive reply, but hope the info is helpful:
1) Deleted existing raw logs
2) Confirmed there are two existing global DM logs, the original, and the one that got added when I ran "Convert Shared to Global"
3) Checked size of both log files (DM_Logs = 78kb and auto = 49kb)
4) Had 5 accts send 5 DM's each: 5 custom search results, 5 max actions, unique items selected, 1 job per acct.
This ran much faster than expected, I confirmed that all filters were set as usual.
No actions were skipped, and no previously processed items were found.
5) Checked global log sizes: Now DM_Logs = 80kb and auto = 51kb
6) Added new global log to all accts (now 3 each)
7) Ran same DM action on same 5 accts.
8) Checked global log sizes, now DM_Logs = 82kb, auto = 53kb, DM_Logs2 = 3kb (all growing)
9) Checked how many previously processed jobs were loaded in all raw logs for both runs:
3832 -> 3992
4077 -> 4237
3959 -> 4119
3878 -> 4038
4116 -> 4264 (This acct had previously been returning 150k+ processed items)
These numbers all seem very small, I've sent many thousands of DM's across all accts, but much less than 4k on each acct.
10) Submitted logs: 31806
11) Since the number of previously processed items seems very low, I ran "Convert Shared to Global" again across my 810 active accounts and this time got a global log of 3.5mb!!! Now all accts have 4 global logs.
12) For a 3rd time, ran the same DM process sending 5 DM's from the same 5 accts.
This pulled in WAYYY more "previous db logs"
3832 -> 3992 -> 85353
4077 -> 4237 -> 85598
3959 -> 4119 -> 85480
3878 -> 4038 -> 85399
4116 -> 4264 -> 85353
All jobs are still completed without finding a duplicate processed item.
Unlike the first time I converted shared to global, the new global log did not shrink when I ran the jobs. Last time I got a similar size file, about 3.5 MB, then sent my DM's, and the global log had shrunk to a few KB, like all of the previously processed items were blown away.
I've been running 1 job per acct, so 800 + jobs, each job sends just a few DM's but repeats 5 times. I run 150+ threads in the processor. I'm wondering if so many parallel jobs all trying to update the same log file could corrupt it? Maybe I should use "Share Across Accounts" to run fewer parallel jobs?
13) Submitted logs again: 99119
11) Emailed you 15x raw logs and before and after snapshots of the 4 global logs.
Hope this is useful. Thanks again.
1) Deleted existing raw logs
2) Confirmed there are two existing global DM logs, the original, and the one that got added when I ran "Convert Shared to Global"
3) Checked size of both log files (DM_Logs = 78kb and auto = 49kb)
4) Had 5 accts send 5 DM's each: 5 custom search results, 5 max actions, unique items selected, 1 job per acct.
This ran much faster than expected, I confirmed that all filters were set as usual.
No actions were skipped, and no previously processed items were found.
5) Checked global log sizes: Now DM_Logs = 80kb and auto = 51kb
6) Added new global log to all accts (now 3 each)
7) Ran same DM action on same 5 accts.
8) Checked global log sizes, now DM_Logs = 82kb, auto = 53kb, DM_Logs2 = 3kb (all growing)
9) Checked how many previously processed jobs were loaded in all raw logs for both runs:
3832 -> 3992
4077 -> 4237
3959 -> 4119
3878 -> 4038
4116 -> 4264 (This acct had previously been returning 150k+ processed items)
These numbers all seem very small, I've sent many thousands of DM's across all accts, but much less than 4k on each acct.
10) Submitted logs: 31806
11) Since the number of previously processed items seems very low, I ran "Convert Shared to Global" again across my 810 active accounts and this time got a global log of 3.5mb!!! Now all accts have 4 global logs.
12) For a 3rd time, ran the same DM process sending 5 DM's from the same 5 accts.
This pulled in WAYYY more "previous db logs"
3832 -> 3992 -> 85353
4077 -> 4237 -> 85598
3959 -> 4119 -> 85480
3878 -> 4038 -> 85399
4116 -> 4264 -> 85353
All jobs are still completed without finding a duplicate processed item.
Unlike the first time I converted shared to global, the new global log did not shrink when I ran the jobs. Last time I got a similar size file, about 3.5 MB, then sent my DM's, and the global log had shrunk to a few KB, like all of the previously processed items were blown away.
I've been running 1 job per acct, so 800 + jobs, each job sends just a few DM's but repeats 5 times. I run 150+ threads in the processor. I'm wondering if so many parallel jobs all trying to update the same log file could corrupt it? Maybe I should use "Share Across Accounts" to run fewer parallel jobs?
13) Submitted logs again: 99119
11) Emailed you 15x raw logs and before and after snapshots of the 4 global logs.
Hope this is useful. Thanks again.
Re: Duplicate DM Actions
Hi Martin,
Quick update, things were running fine until I tried to run 20 parallel threads (1 acct per thread sending DM's). Within a few minutes something happened and shrunk the newest log file down to 22.7 KB (from 3541 KB).
So possibly there is an issue with parallel threads clashing...
Quick update, things were running fine until I tried to run 20 parallel threads (1 acct per thread sending DM's). Within a few minutes something happened and shrunk the newest log file down to 22.7 KB (from 3541 KB).
So possibly there is an issue with parallel threads clashing...
Re: Duplicate DM Actions
Resubmitted logs just after I found the above issue: 27187