Cleaning out and minimizing logs. Application currently crushing 8 core VM.

Support / help / discussion forum for twitter bot
Post Reply
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Cleaning out and minimizing logs. Application currently crushing 8 core VM.

Post by Alexiom »

Hi Team,

Can anyone advise on the best way to ensure logs are shared across accounts without CRUSHING the application?

I'm currently running 350 accounts and it's almost unusable on an 8 core VM - CPU is locked at 99% whenever TwitterDub is doing anything. Hangs and crashes constantly, and I want to scale bigger. The only thing I can think of is way too many log files. I've already removed Chrome sessions folder which was about 80gb but did not improve performance.

For sharing actions, I've done all 3 methods, merged all logs from accounts, then assigned a global log to all accounts. And when I run actions, I run them in batches of 25-50 accounts and share actions across the whole batch.

Q1: Just confirming that If all accounts are assigned the same global log, will they never duplicate actions? eg DM the same Twitter user or like the same tweet. So merging acct logs and sharing jobs across accounts should not be necessary?
Q2: Would it be better to not run batches with actions shared across the batch, and just have hundreds (or thousands) of jobs running in the processor with suitable pauses?
Q3: I need to flush out the logs and get as clean a start as possible. I want to delete everything I can from the AppData folder except for accounts and SearchDM logs. Is there an easy way to find these specifically? Or is there a way to ensure all actions from all accounts are rolled into the global log and just back that up before blowing away everything else? I'm concerned that the GlobalLog.bin file is one of the smallest log files so probably doesn't contain everything I need...

Any other suggestions on wiping unnecessary logs, and minimizing them in the future would be greatly appreciated. Or if there is somewhere else I should be looking to solve my performance issue, please let me know.

Kind regards
User avatar
martin@rootjazz
Site Admin
Posts: 34376
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.

Post by martin@rootjazz »

Alexiom wrote: Wed May 10, 2023 3:32 pm
For sharing actions, I've done all 3 methods, merged all logs from accounts,
Might be an issue, you have duplicated all processed logs across all accounts.
The only thing I can think of is way too many log files
yes, processed_items database files is probably too big

Hi
Q1: Just confirming that If all accounts are assigned the same global log, will they never duplicate actions? eg DM the same Twitter user or like the same tweet. So merging acct logs and sharing jobs across accounts should not be necessary?
correct
Q2: Would it be better to not run batches with actions shared across the batch, and just have hundreds (or thousands) of jobs running in the processor with suitable pauses?
Shouldn't make much difference
Q3: I need to flush out the logs and get as clean a start as possible. I want to delete everything I can from the AppData folder except for accounts and SearchDM logs. Is there an easy way to find these specifically? Or is there a way to ensure all actions from all accounts are rolled into the global log and just back that up before blowing away everything else? I'm concerned that the GlobalLog.bin file is one of the smallest log files so probably doesn't contain everything I need...
there is a CLEAN function, I don't remember it though. Let me get back to you
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.

Post by Alexiom »

Q1: Just confirming that If all accounts are assigned the same global log, will they never duplicate actions? eg DM the same Twitter user or like the same tweet. So merging acct logs and sharing jobs across accounts should not be necessary?
correct
Thanks Martin,

From what I've read I need to assign the global log on every action tab that I want to be included (follow, like, DM etc). As it only applies to the currently active action tab, is this correct? Can I use the same log name/ID on each tab, or will they overwrite each other?
User avatar
martin@rootjazz
Site Admin
Posts: 34376
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.

Post by martin@rootjazz »

Next update will include an easier (and hopefully better) clean / de-dupe processed logs to global routine


The next update will include this feature. I shall let you know when it is ready.



Regards,
Martin
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.

Post by Alexiom »

Ok great, thanks Martin.

I tried a few things, converted all logs to a new global log for like/follow/DM, then saved them, blew away the whole log files folder and just added these 3 log files back in. But this was still super heavy, I think they are full of duplicates since I merged all account logs.

In the end just started with fresh logs, I'll get over a few DM duplicates.

However, the system is still grinding hard, and takes several minutes to load up application and it's constantly hanging on a now 8 core VM. Is there anywhere else I can look to clean up to improve performance?

Or should I just back up account details, wipe app-data, and re-install?

Kind regards,
User avatar
martin@rootjazz
Site Admin
Posts: 34376
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.

Post by martin@rootjazz »

Alexiom wrote: Wed May 10, 2023 9:00 pm
Or should I just back up account details, wipe app-data, and re-install?
the processed_items folder to delete is:

%appdata%\rootjazz\twitterdub\database\dbloggeddata\

that contains all the records of who you have followed / liked / DM etc etc


if you want a clean start, you can backup the folder:


%appdata%\rootjazz\twitterdub\database\dbaccount\

delete
%appdata%\rootjazz\twitterdub\database

then readd the backed up
%appdata%\rootjazz\twitterdub\database\dbaccount


Or you can wait for the update, see if it helps clean things, if yes, great, if not, then try the above?
User avatar
martin@rootjazz
Site Admin
Posts: 34376
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.

Post by martin@rootjazz »

for all accounts:

TOOLS > SHARED TO GLOBAL PROCESSED LOGS

or for specific accounts:

select / check accounts > right click > global processed >shared to global processed (all action tabs)
Alexiom
Posts: 62
Joined: Thu Mar 30, 2023 7:16 pm

Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.

Post by Alexiom »

Thanks Martin.

Cleaned out logs and that didn't seem to help, then I backed up accounts and completely wiped appdata and reinstalled the app. This helped for a bit then started to lag again.

I eventually worked out that it was Windows Defender. Likely trying to scan every log file as it was rewritten. Adding an exception to windows defender to not scan the rootjazz appdata folder made a dramatic difference. So that seems to be the issue, running fairly smoothly now.
User avatar
martin@rootjazz
Site Admin
Posts: 34376
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.

Post by martin@rootjazz »

Alexiom wrote: Sat May 13, 2023 8:23 pm

I eventually worked out that it was Windows Defender. Likely trying to scan every log file as it was rewritten. Adding an exception to windows defender to not scan the rootjazz appdata folder made a dramatic difference. So that seems to be the issue, running fairly smoothly now.
thanks for the information.

What OS are you running? Is it a server OS? Windows server?
Post Reply