Hi Team,
Can anyone advise on the best way to ensure logs are shared across accounts without CRUSHING the application?
I'm currently running 350 accounts and it's almost unusable on an 8 core VM - CPU is locked at 99% whenever TwitterDub is doing anything. Hangs and crashes constantly, and I want to scale bigger. The only thing I can think of is way too many log files. I've already removed Chrome sessions folder which was about 80gb but did not improve performance.
For sharing actions, I've done all 3 methods, merged all logs from accounts, then assigned a global log to all accounts. And when I run actions, I run them in batches of 25-50 accounts and share actions across the whole batch.
Q1: Just confirming that If all accounts are assigned the same global log, will they never duplicate actions? eg DM the same Twitter user or like the same tweet. So merging acct logs and sharing jobs across accounts should not be necessary?
Q2: Would it be better to not run batches with actions shared across the batch, and just have hundreds (or thousands) of jobs running in the processor with suitable pauses?
Q3: I need to flush out the logs and get as clean a start as possible. I want to delete everything I can from the AppData folder except for accounts and SearchDM logs. Is there an easy way to find these specifically? Or is there a way to ensure all actions from all accounts are rolled into the global log and just back that up before blowing away everything else? I'm concerned that the GlobalLog.bin file is one of the smallest log files so probably doesn't contain everything I need...
Any other suggestions on wiping unnecessary logs, and minimizing them in the future would be greatly appreciated. Or if there is somewhere else I should be looking to solve my performance issue, please let me know.
Kind regards
Cleaning out and minimizing logs. Application currently crushing 8 core VM.
- martin@rootjazz
- Site Admin
- Posts: 34376
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.
Might be an issue, you have duplicated all processed logs across all accounts.
yes, processed_items database files is probably too bigThe only thing I can think of is way too many log files
Hi
correctQ1: Just confirming that If all accounts are assigned the same global log, will they never duplicate actions? eg DM the same Twitter user or like the same tweet. So merging acct logs and sharing jobs across accounts should not be necessary?
Shouldn't make much differenceQ2: Would it be better to not run batches with actions shared across the batch, and just have hundreds (or thousands) of jobs running in the processor with suitable pauses?
there is a CLEAN function, I don't remember it though. Let me get back to youQ3: I need to flush out the logs and get as clean a start as possible. I want to delete everything I can from the AppData folder except for accounts and SearchDM logs. Is there an easy way to find these specifically? Or is there a way to ensure all actions from all accounts are rolled into the global log and just back that up before blowing away everything else? I'm concerned that the GlobalLog.bin file is one of the smallest log files so probably doesn't contain everything I need...
Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.
Thanks Martin,correctQ1: Just confirming that If all accounts are assigned the same global log, will they never duplicate actions? eg DM the same Twitter user or like the same tweet. So merging acct logs and sharing jobs across accounts should not be necessary?
From what I've read I need to assign the global log on every action tab that I want to be included (follow, like, DM etc). As it only applies to the currently active action tab, is this correct? Can I use the same log name/ID on each tab, or will they overwrite each other?
- martin@rootjazz
- Site Admin
- Posts: 34376
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.
Next update will include an easier (and hopefully better) clean / de-dupe processed logs to global routine
The next update will include this feature. I shall let you know when it is ready.
Regards,
Martin
The next update will include this feature. I shall let you know when it is ready.
Regards,
Martin
Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.
Ok great, thanks Martin.
I tried a few things, converted all logs to a new global log for like/follow/DM, then saved them, blew away the whole log files folder and just added these 3 log files back in. But this was still super heavy, I think they are full of duplicates since I merged all account logs.
In the end just started with fresh logs, I'll get over a few DM duplicates.
However, the system is still grinding hard, and takes several minutes to load up application and it's constantly hanging on a now 8 core VM. Is there anywhere else I can look to clean up to improve performance?
Or should I just back up account details, wipe app-data, and re-install?
Kind regards,
I tried a few things, converted all logs to a new global log for like/follow/DM, then saved them, blew away the whole log files folder and just added these 3 log files back in. But this was still super heavy, I think they are full of duplicates since I merged all account logs.
In the end just started with fresh logs, I'll get over a few DM duplicates.
However, the system is still grinding hard, and takes several minutes to load up application and it's constantly hanging on a now 8 core VM. Is there anywhere else I can look to clean up to improve performance?
Or should I just back up account details, wipe app-data, and re-install?
Kind regards,
- martin@rootjazz
- Site Admin
- Posts: 34376
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.
the processed_items folder to delete is:
%appdata%\rootjazz\twitterdub\database\dbloggeddata\
that contains all the records of who you have followed / liked / DM etc etc
if you want a clean start, you can backup the folder:
%appdata%\rootjazz\twitterdub\database\dbaccount\
delete
%appdata%\rootjazz\twitterdub\database
then readd the backed up
%appdata%\rootjazz\twitterdub\database\dbaccount
Or you can wait for the update, see if it helps clean things, if yes, great, if not, then try the above?
- martin@rootjazz
- Site Admin
- Posts: 34376
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
- martin@rootjazz
- Site Admin
- Posts: 34376
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.
for all accounts:
TOOLS > SHARED TO GLOBAL PROCESSED LOGS
or for specific accounts:
select / check accounts > right click > global processed >shared to global processed (all action tabs)
TOOLS > SHARED TO GLOBAL PROCESSED LOGS
or for specific accounts:
select / check accounts > right click > global processed >shared to global processed (all action tabs)
Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.
Thanks Martin.
Cleaned out logs and that didn't seem to help, then I backed up accounts and completely wiped appdata and reinstalled the app. This helped for a bit then started to lag again.
I eventually worked out that it was Windows Defender. Likely trying to scan every log file as it was rewritten. Adding an exception to windows defender to not scan the rootjazz appdata folder made a dramatic difference. So that seems to be the issue, running fairly smoothly now.
Cleaned out logs and that didn't seem to help, then I backed up accounts and completely wiped appdata and reinstalled the app. This helped for a bit then started to lag again.
I eventually worked out that it was Windows Defender. Likely trying to scan every log file as it was rewritten. Adding an exception to windows defender to not scan the rootjazz appdata folder made a dramatic difference. So that seems to be the issue, running fairly smoothly now.
- martin@rootjazz
- Site Admin
- Posts: 34376
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Cleaning out and minimizing logs. Application currently crushing 8 core VM.
thanks for the information.Alexiom wrote: ↑Sat May 13, 2023 8:23 pm
I eventually worked out that it was Windows Defender. Likely trying to scan every log file as it was rewritten. Adding an exception to windows defender to not scan the rootjazz appdata folder made a dramatic difference. So that seems to be the issue, running fairly smoothly now.
What OS are you running? Is it a server OS? Windows server?