Cleaning out and minimizing logs. Application currently crushing 8 core VM.

[Emin]

Hi, Martin
after adding this function, it will take 5-7 hours to make 30-35 likes
https://imgur.com/SxPWgWS
https://imgur.com/ncBMOiV

[martin@rootjazz]

Hi, Martin
after adding this function, it will take 5-7 hours to make 30-35 likes
https://imgur.com/SxPWgWS
https://imgur.com/ncBMOiV

Please in future start a new thread for your issue, as it's not related to this thread.

As to the issue, no idea with logs


Can you send raw logs please:
right click the action > RAW LOGS > copy and send me the contents of the file that opens.

Code: Select all

support[at]rootjazz[dot]com

And
HELP > LOGS > SUBMIT

then send your logs ID - the numbers are sufficient (displayed after successful uploading of logs)


For future reference, please, read through this post and submit your issue accordingly. Not everything will be relevant to your issue, but have a read as it gives an idea of the information you can provide to help me help you as quickly as possible without delays in having to request additional information

https://rootjazz.com/forum/viewtopic.php?f=23&t=1634



Regards,
Martin

[Emin]

my problem is related to this post
I didn't have this problem before adding this feature.
id_logs 38562

[Alexiom]

I think it's related to this thread:

https://rootjazz.com/forum/viewtopic.ph ... 0&start=20

Martin was able to resolve an issue of global logs being destroyed while multiple threads were processing. Now the global logs stay intact and duplicate actions are correctly identified.

However, I'm seeing the same issue as you, multi-treaded performance has tanked. If I run 10 threads, each thread takes nearly 10 times as long to process.

In the meantime, I guess you can roll back to an earlier version...

Martin, can you assist in troubleshooting this?

[Emin]

yes, if I run only 1 thread, then everything is fine, but the more I run tasks, the longer the process takes

[Alexiom]

Did some testing. With my configuration, running the DM process, the average time for a job to run doesn't take too much of a hit from 1-10 threads. But at 20 threads average job takes twice as long to run, so throughput is the same as just 10 threads. But I'd love to run 200 threads

[martin@rootjazz]

my problem is related to this post
I didn't have this problem before adding this feature.
id_logs 38562

ok so you updated and the issue started.


Your logs are strange, something is looping loading the global log... But that wasn't changed in this update, just hardened the concurrency code in accessing the global file.

Cannot think why it's taking 10x as long...

[martin@rootjazz]

Did some testing. With my configuration, running the DM process, the average time for a job to run doesn't take too much of a hit from 1-10 threads. But at 20 threads average job takes twice as long to run, so throughput is the same as just 10 threads. But I'd love to run 200 threads

I'm guessing it was you I told in the email, I'll need to have a think and completely change how global logs are accessed, as the current method is not fit for purpose at scale.

I have an idea how it can be done, but will need to code it from scratch. Please bear with me (and give me frequent nudges to hurry up and get it done)

[Emin]

I only do like,dm and dm watch followers,quite a long time, more than a year, and this has never happened ... already 300 accounts banned

[martin@rootjazz]

ok so I had some time to get it done. Should be much better now (if the issue is what I thought it was).


I'm thinking the same technique could be used to on all processed_logs, not just global logs, but would increase memory usage considerably