Alexiom wrote: ↑Sun May 28, 2023 5:58 pm
Hi Martin, I've done a ton of testing and recorded everything. Sorry for the massive reply, but hope the info is helpful:
1) Deleted existing raw logs
2) Confirmed there are two existing global DM logs, the original, and the one that got added when I ran "Convert Shared to Global"
3) Checked size of both log files (DM_Logs = 78kb and auto = 49kb)
4) Had 5 accts send 5 DM's each: 5 custom search results, 5 max actions, unique items selected, 1 job per acct.
This ran much faster than expected, I confirmed that all filters were set as usual.
No actions were skipped, and no previously processed items were found.
5) Checked global log sizes: Now DM_Logs = 80kb and auto = 51kb
6) Added new global log to all accts (now 3 each)
7) Ran same DM action on same 5 accts.
8) Checked global log sizes, now DM_Logs = 82kb, auto = 53kb, DM_Logs2 = 3kb (all growing)
9) Checked how many previously processed jobs were loaded in all raw logs for both runs:
3832 -> 3992
4077 -> 4237
3959 -> 4119
3878 -> 4038
4116 -> 4264 (This acct had previously been returning 150k+ processed items)
These numbers all seem very small, I've sent many thousands of DM's across all accts, but much less than 4k on each acct.
10) Submitted logs: 31806
11) Since the number of previously processed items seems very low, I ran "Convert Shared to Global" again across my 810 active accounts and this time got a global log of 3.5mb!!! Now all accts have 4 global logs.
12) For a 3rd time, ran the same DM process sending 5 DM's from the same 5 accts.
This pulled in WAYYY more "previous db logs"
3832 -> 3992 -> 85353
4077 -> 4237 -> 85598
3959 -> 4119 -> 85480
3878 -> 4038 -> 85399
4116 -> 4264 -> 85353
All jobs are still completed without finding a duplicate processed item.
Unlike the first time I converted shared to global, the new global log did not shrink when I ran the jobs. Last time I got a similar size file, about 3.5 MB, then sent my DM's, and the global log had shrunk to a few KB, like all of the previously processed items were blown away.
I've been running 1 job per acct, so 800 + jobs, each job sends just a few DM's but repeats 5 times. I run 150+ threads in the processor. I'm wondering if so many parallel jobs all trying to update the same log file could corrupt it? Maybe I should use "Share Across Accounts" to run fewer parallel jobs?
13) Submitted logs again: 99119
11) Emailed you 15x raw logs and before and after snapshots of the 4 global logs.
Hope this is useful. Thanks again.