Duplicate DM Actions

Support / help / discussion forum for twitter bot
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Duplicate DM Actions

Post by Alexiom »

Tried again, restored my backups of the global logs, and created 4 jobs this time, each shared across approx 50 accounts. The same problem, within 5 minutes I noticed the global logs had been shrunk to a few KB.

Tried running a single job across 810 accts overnight, just enough to send 1 DM from each account, but it's been running for 11 hrs now so there's no way this is scalable. But, log files are still intact.
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Duplicate DM Actions

Post by Alexiom »

Testing more, running smaller jobs again, 1 acct per job sending between 3 and 15 DM's. I queued up about 50 jobs, but only let the processor run 3 threads.

Running 3 jobs in the processer logs seemed to be surviving. Soon after adding a 4th thread one of the large global logs was shrunk, 3.5mb down to 2.27kb.

I paused the processor and checked the raw logs for the 4 active threads. Nothing there seems strange, they all finished scraping, then checked the results against processed logs and started sending DM's.
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Duplicate DM Actions

Post by Alexiom »

Tested overnight with just a single processor thread. Works absolutely fine, all global logs steadily increasing in size. However, it's far too slow to possibly send DM's across all accounts.

@Martin, as far as I can tell some kind of lock occurs when multiple threads try to update the logs simultaneously which results in them being destroyed.
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Duplicate DM Actions

Post by Alexiom »

One more update for now. To check if I had some kind of corrupted installation or data I tested this on another instance of TwitterDub I have that runs only about 70 accounts in a different niche. Exactly the same issue when I ran DM's across all accounts in parallel, existing global logs were shrunk/deleted during the process.
User avatar
martin@rootjazz
Site Admin
Posts: 34359
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Duplicate DM Actions

Post by martin@rootjazz »

ok found the thread from your email, looking into now
User avatar
martin@rootjazz
Site Admin
Posts: 34359
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Duplicate DM Actions

Post by martin@rootjazz »

Alexiom wrote: Sun May 28, 2023 5:58 pm Hi Martin, I've done a ton of testing and recorded everything. Sorry for the massive reply, but hope the info is helpful:

1) Deleted existing raw logs

2) Confirmed there are two existing global DM logs, the original, and the one that got added when I ran "Convert Shared to Global"

3) Checked size of both log files (DM_Logs = 78kb and auto = 49kb)

4) Had 5 accts send 5 DM's each: 5 custom search results, 5 max actions, unique items selected, 1 job per acct.

This ran much faster than expected, I confirmed that all filters were set as usual.
No actions were skipped, and no previously processed items were found.

5) Checked global log sizes: Now DM_Logs = 80kb and auto = 51kb

6) Added new global log to all accts (now 3 each)

7) Ran same DM action on same 5 accts.

8) Checked global log sizes, now DM_Logs = 82kb, auto = 53kb, DM_Logs2 = 3kb (all growing)

9) Checked how many previously processed jobs were loaded in all raw logs for both runs:

3832 -> 3992
4077 -> 4237
3959 -> 4119
3878 -> 4038
4116 -> 4264 (This acct had previously been returning 150k+ processed items)

These numbers all seem very small, I've sent many thousands of DM's across all accts, but much less than 4k on each acct.

10) Submitted logs: 31806

11) Since the number of previously processed items seems very low, I ran "Convert Shared to Global" again across my 810 active accounts and this time got a global log of 3.5mb!!! Now all accts have 4 global logs.

12) For a 3rd time, ran the same DM process sending 5 DM's from the same 5 accts.

This pulled in WAYYY more "previous db logs"

3832 -> 3992 -> 85353
4077 -> 4237 -> 85598
3959 -> 4119 -> 85480
3878 -> 4038 -> 85399
4116 -> 4264 -> 85353

All jobs are still completed without finding a duplicate processed item.

Unlike the first time I converted shared to global, the new global log did not shrink when I ran the jobs. Last time I got a similar size file, about 3.5 MB, then sent my DM's, and the global log had shrunk to a few KB, like all of the previously processed items were blown away.

I've been running 1 job per acct, so 800 + jobs, each job sends just a few DM's but repeats 5 times. I run 150+ threads in the processor. I'm wondering if so many parallel jobs all trying to update the same log file could corrupt it? Maybe I should use "Share Across Accounts" to run fewer parallel jobs?

13) Submitted logs again: 99119

11) Emailed you 15x raw logs and before and after snapshots of the 4 global logs.

Hope this is useful. Thanks again.

Something isn't right here, sorry I should have said before, but I will need to see your datebase and analyse it. Maybe there has been database corruption, or the functions haven't run correctly.

Can you submit your database please

TOOLS > CREATE DB BACKUP AND SEND TO SUPPORT

then send your DB ID - the first 4 numbers is sufficient (displayed after successful uploading of logs)



Regards,
Martin
User avatar
martin@rootjazz
Site Admin
Posts: 34359
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Duplicate DM Actions

Post by martin@rootjazz »

Alexiom wrote: Sun May 28, 2023 10:06 pm Hi Martin,

Quick update, things were running fine until I tried to run 20 parallel threads (1 acct per thread sending DM's). Within a few minutes something happened and shrunk the newest log file down to 22.7 KB (from 3541 KB).

So possibly there is an issue with parallel threads clashing...
ok..... if file corrupted, perhaps it was backed up and a new one created... I cannot think how it was wiped another way. AS above, I need your database for review
User avatar
martin@rootjazz
Site Admin
Posts: 34359
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Duplicate DM Actions

Post by martin@rootjazz »

Alexiom wrote: Tue May 30, 2023 12:12 pm One more update for now. To check if I had some kind of corrupted installation or data I tested this on another instance of TwitterDub I have that runs only about 70 accounts in a different niche. Exactly the same issue when I ran DM's across all accounts in parallel, existing global logs were shrunk/deleted during the process.
No, it definitely sounds like a bug. let me check the new logs.

But if you can confirm the actions.

5 processing threads
5 DM actions.
How many accounts in the action? Can you explain what they were doing exactly, so I can try and recreate. If it happens all the time within 5 minutes I should be able to recreate fairly easily... I hope
User avatar
martin@rootjazz
Site Admin
Posts: 34359
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Duplicate DM Actions

Post by martin@rootjazz »

Alexiom wrote: Tue May 30, 2023 12:12 pm One more update for now. To check if I had some kind of corrupted installation or data I tested this on another instance of TwitterDub I have that runs only about 70 accounts in a different niche. Exactly the same issue when I ran DM's across all accounts in parallel, existing global logs were shrunk/deleted during the process.
if could send the database from this instance, it will probably be easier to see / confirm what happening

Can you submit your database please

TOOLS > CREATE DB BACKUP AND SEND TO SUPPORT

then send your DB ID - the first 4 numbers is sufficient (displayed after successful uploading of logs)



Regards,
Martin
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Duplicate DM Actions

Post by Alexiom »

martin@rootjazz wrote: Tue May 30, 2023 10:44 pm
Alexiom wrote: Tue May 30, 2023 12:12 pm One more update for now. To check if I had some kind of corrupted installation or data I tested this on another instance of TwitterDub I have that runs only about 70 accounts in a different niche. Exactly the same issue when I ran DM's across all accounts in parallel, existing global logs were shrunk/deleted during the process.
if could send the database from this instance, it will probably be easier to see / confirm what happening

Can you submit your database please

TOOLS > CREATE DB BACKUP AND SEND TO SUPPORT

then send your DB ID - the first 4 numbers is sufficient (displayed after successful uploading of logs)



Regards,
Martin
Hi Martin, this is done, Database ID: database_backup_3.4.1.664_2023-05-30.56180432_
Post Reply