100% CPU load

[axisar]

Its not about new actions.
Is old stable actions which already working through 100+ subversions of software, just one day start consume lot of disk.
I just increase number of videos in watched folder, but looks like its also not a reason.

i have 5 threads and 15 actions to watch folder

22:35:23: Watching folder: E:\tmp\twtvids
22:51:50: Random order

take a look how much time it takes to load folder content, here is a problem

another example
05:19:01: Watching folder: E:\tmp\twtvids
05:38:50: Random order

post and ffmpeg works just fine and quick

from another account
23:14:18: Watching folder: E:\tmp\twtvids
23:30:27: Random order

i just wonder why software tries to open ALL files instead of just read folder content?

[axisar]

I cannot actually remember how TD handles detected watch files, let me check

Any update regarding this?

[martin@rootjazz]

I cannot actually remember how TD handles detected watch files, let me check

Any update regarding this?

Sorry I've been away, didn't have time before I left.

Just checked, all working. CPU load normal.
video detected
processed
remove from watch directory.


If you can post the logs as requested before, I Can look further into what happens for you

[martin@rootjazz]

Its not about new actions.
Is old stable actions which already working through 100+ subversions of software, just one day start consume lot of disk.
I just increase number of videos in watched folder, but looks like its also not a reason.

i have 5 threads and 15 actions to watch folder

22:35:23: Watching folder: E:\tmp\twtvids
22:51:50: Random order

take a look how much time it takes to load folder content, here is a problem

another example
05:19:01: Watching folder: E:\tmp\twtvids
05:38:50: Random order

post and ffmpeg works just fine and quick

from another account
23:14:18: Watching folder: E:\tmp\twtvids
23:30:27: Random order

i just wonder why software tries to open ALL files instead of just read folder content?

the issue is on your machine

Program just loops files, finds the contents and logs it. Is performing this in a split second with 1 video or 100 videos in folder for me.


Unless you are adding millions of files to the folder, no folder loop is taking 20 minutes, unless the machine is being throttled, which if you are on some kind of VPS it might be (free AWS / google / azure tiers for example).


but try the usual:
restart app
restart machine

Create new test action watch 1 folder. Add 1 video. How long?
Add 10 videos, how long?
Add 100 videos. How long?

etc etc

But the program is not doing anything different.

It asks the operating system for a list of files.
Files are checked if the extension is a video extension, if so, it's logged and added to a list.

[axisar]

the issue is on your machine

Program just loops files, finds the contents and logs it. Is performing this in a split second with 1 video or 100 videos in folder for me.

It asks the operating system for a list of files.
Files are checked if the extension is a video extension, if so, it's logged and added to a list.

only folder content and extension? dir and filter by *.mp4? or maybe dir *.mp4? and this takes 2-20minutes?
why this happened? -> https://i.imgur.com/PAWWD6N.png
Why system proccess reading all these files one by one?

02:37:54: Watching folder: E:\tmp\twtvids2
02:39:06: Random order

^^ 100 files, looks bad

02:44:10: Watching folder: E:\tmp\twtvids2
02:49:13: Random order

^^ 500 files, awful

[axisar]

Sad what i cant test it with old versions, because of that incompatibility of actions between versions, just dont want to mess with import/export... already spent lot of time for just investigate what actually happens and tests
i bet issue comes from that fix with actions/version incompatibility

Unless you are adding millions of files to the folder, no folder loop is taking 20 minutes, unless the machine is being throttled, which if you are on some kind of VPS it might be (free AWS / google / azure tiers for example).

NOTHING CHANGED ON MY SIDE. Its non free vps, 2 cores and 4GB Ram and it was very suitable for at least a year, but after some update i facing this stupid issue now
Also just found old folder with 5000 files and td worked fine watching this folder on SAME vps but on older version

[axisar]

ID 16993

[martin@rootjazz]

Also just found old folder with 5000 files and td worked fine watching this folder on SAME vps but on older version

Doesn't make sense, this code has not changed.

Let me check your logs and see if there are any clues there. If not, I'll need to get you a debugging build put together so can try and identify exactly where the issue is.

Unfortunately I doubt that will be middle of next week, as am flying back home Saturday and it will take about 26 hours. Then I don't actually get to my house a day after that.

[martin@rootjazz]

ok, I see your issue I think. No idea why the issue has just occurred, this code has always been there.

For your setup, of watching a large folder of files but only wanting to select 1, it's very inefficient. The idea was the program would be posting all contents, on each iteration, that way, the efficiency is ok, but it's not working for your case.

I'll make some improvements

[martin@rootjazz]

https://rootjazz.com/twitterdub/updatetesting.html

try this, should fix it



Regards,
Martin