CPU overload if follow by list

[alexliff]

Hi Martin !
I hope you're well.

This is not a bug/problem but maybe there is something we can do about it

If I follow using "followers of" (twitter.com/user) it works perfectly (unique is on). I run 6 - 8 accounts and the program use 5% - 60% CPU.
If I follow using "from list" (list of user profiles "twitter.com/user" not user IDs), and "unique" is on the program eats 100% CPU.
I can't run any process or use the PC.

If unique is off it works just fine.

List consists of ~18000 users. I follow ~200 per 24 hours.

When the CPU is overloaded the program does this:

Code: Select all

Starting: 28.02.2017 08:00 
Search: c:\BestNewMusician_NEW.txt
Num: 200
SearchType: -1
Unique: True

Processing: c:\BestNewMusician_NEW.txt max: 200
Loaded from list: 18467 items to process
Items loaded to process: 18467
Remove already processed items
account: ******************** load previously processed items
Removing already process items of: ********************
Loading previous db logs: 979c0a9f-8edf-4993-aa13-38b091070afa_Follow
Loaded previous db logs: 35390 within 24hrs: 0
Checking results: 18467 against already processed list: 35390 of: ********************
Remaining logs to check: 18400
Remaining logs to check: 18300
Remaining logs to check: 18200
Remaining logs to check: 18100
Remaining logs to check: 18000
Remaining logs to check: 17900
Remaining logs to check: 17800
Remaining logs to check: 17700
Remaining logs to check: 17600
Remaining logs to check: 17500
Remaining logs to check: 17400
Remaining logs to check: 17300
Remaining logs to check: 17200
Remaining logs to check: 17100
Remaining logs to check: 17000
Remaining logs to check: 16900
Remaining logs to check: 16800
Remaining logs to check: 16700
Remaining logs to check: 16600
Remaining logs to check: 16500
Remaining logs to check: 16400
Remaining logs to check: 16300
Remaining logs to check: 16200
Remaining logs to check: 16100
Remaining logs to check: 16000
Remaining logs to check: 15900
Remaining logs to check: 15800
Remaining logs to check: 15700
Remaining logs to check: 15600
Remaining logs to check: 15500
Remaining logs to check: 15400
Remaining logs to check: 15300
Remaining logs to check: 15200
Remaining logs to check: 15100
Remaining logs to check: 15000
Remaining logs to check: 14900
Remaining logs to check: 14800
Remaining logs to check: 14700
Remaining logs to check: 14600
Remaining logs to check: 14500
Remaining logs to check: 14400
Remaining logs to check: 14300
Remaining logs to check: 14200
Remaining logs to check: 14100
Remaining logs to check: 14000
Remaining logs to check: 13900
Remaining logs to check: 13800
Remaining logs to check: 13700
Remaining logs to check: 13600
Remaining logs to check: 13500
Remaining logs to check: 13400
Remaining logs to check: 13300
Remaining logs to check: 13200
Remaining logs to check: 13100
Remaining logs to check: 13000
Remaining logs to check: 12900
Remaining logs to check: 12800
Remaining logs to check: 12700
Remaining logs to check: 12600
Remaining logs to check: 12500
Remaining logs to check: 12400
Remaining logs to check: 12300
Remaining logs to check: 12200
Remaining logs to check: 12100
Remaining logs to check: 12000
Remaining logs to check: 11900
Remaining logs to check: 11800
Remaining logs to check: 11700
Remaining logs to check: 11600
Remaining logs to check: 11500
Remaining logs to check: 11400
Remaining logs to check: 11300
Remaining logs to check: 11200
Remaining logs to check: 11100
Remaining logs to check: 11000
Remaining logs to check: 10900
and so on :)))

As I understand the program checks already processed items. When this process is finished CPU comes to live again

I can't even run another account, because if I try to navigate the program it falls to "not responding".
I tested it on VPS and my home PC.

VPS (windows server 2008 R2 Enterprise, QEMU Virtual CPU (64) 3.5 GHz, 2GB RAM)
falls to "not responding" immediately after I start process and after a hour being not responding, keeps working (I think when the checking process is finished).

On my home PC (win 7, i7, 16GB ram). I can see processing log (I posted it previously), program doesn't fall to not responding, but the CPU is 100% too and I can't use PC or run another process because the program becomes really unstable.


I'm sorry for huge post hope you will understand me

All the best!

[martin@rootjazz]

Using all CPU *shouldn't* be an issue. It just means the program utilising all available resources to perform something as quickly as possible.

However, this SHOULD NOT impact other programs. The OS scheduler should decide what resources are available to use to keep everything working, then anything access can be used to maximum potential. If the resources are required for something else, then it should claim them back.

note the word *should*, although it sounds like something is not working correctly there


Let me see what I can find out

[martin@rootjazz]

How many accounts are you using?


I have made some speed improvements, although I don't think it will help the CPU issue, but next update you can test and let me know

[martin@rootjazz]

https://rootjazz.com/twitterdub/updatetesting.html

please try this, might be better, might now. But let me know

[alexliff]

omg ! Now it is so fast!
This checking process lasts just a second )) It has been one - two hours till today and I couldn't run more then one account because of the program was not responding during this process (CPU was overloaded).

Now CPU rests and I can run all my accounts simultaneously like if I uncheck "unique"!

You're truly an artist of programing!

Thank you so much !

[martin@rootjazz]

omg ! Now it is so fast!
This checking process lasts just a second )) It has been one - two hours till today and I couldn't run more then one account because of the program was not responding during this process (CPU was overloaded).

Now CPU rests and I can run all my accounts simultaneously like if I uncheck "unique"!

You're truly an artist of programing!

Thank you so much !

really? wasn't expecting it to be that much faster, although I suppose it could if you had large previously processed logs.

Keep an eye on things and let me know if you have any more problems



Regards,
Martin

[alexliff]

really? wasn't expecting it to be that much faster, although I suppose it could if you had large previously processed logs.

Yes, it helped alot and I have a huge previously processed logs (around 100k items)
I even recorded a video "before and after" so, you can see what I was writing about. It works really fast now.

Here is the video:
https://drive.google.com/open?id=0B2L62 ... G9sajkxUWM

Only thing, I don't know does it really check previous items now (see the video)

[martin@rootjazz]

Can you submit logs rather than a video so I can follow the log lines in the code


HELP > LOGS > SUBMIT

then send your logs ID - the first 4 numbers is sufficient (displayed after successful uploading of logs)

[alexliff]

Hey Martin !
I'm sorry, I always forget, first of all, you need logs

As I wrote before the program works super fast now. Only thing, I don't know does it really check previous items now

I ran follow process by list with the same settings twice:

Settings:
- follow by list (list is ±18000 user urls),
- pause 10 - 25,
- follow 10 users,
- unique is on.

1. Before update. (Really slow, CPU is overloaded). It lasted around an hour.
LOGS 92191

2. After update . (Fast and perfect ). Action ran for 3min.
LOGS 45881

Let me know if you need any additional information.
Thanks !

[martin@rootjazz]

your uploaded logs didn't upload correctly so I cannot see them. Can you upload again please

HELP > LOGS > SUBMIT

then send your logs ID - the first 4 numbers is sufficient (displayed after successful uploading of logs)