Process Item Checks for Non-Unique Items - Functionality Improvements

[martin@rootjazz]

https://rootjazz.com/twitterdub/updatetesting.html

tried to make some improvements to the areas I think it may be. But I'm not sure I fully understand your issue, so it probably doesn't fix it - but maybe

[Rev]

Ok I understand. You just want to be 100% sure all the list has actually been followed, so you may re-process the list multiple times.

I just want to run through a list that could have already processed items in it while not experiencing long pause timers for the said already processed items when they are caught.

As both of us mentioned, a partial solution for this would be the "Unique:True" option. But it has its own downfalls for the operation that I'm trying to do at the moment.

Can you explain this. Are you saying they are NOT detected as already followed? Or just not attempted?

Select a list of 100 items
Select account(s)
specify 100 items

all items in list are processed (should be). What is / isn't happening for you?


LEt me read through the thread again as maybe I've missed it

They are NOT detected as already followed and still being attempted/processed as if they weren't followed already.

Take a look at the logs of a "follow" process with "Unique:False":

Code: Select all

Items loaded to process: 1
Not unique process only - do NOT remove already processed items
*** Pre Process Setup ***
Pre process items: shuffle accounts
Loop items: 1 with: 1 threads
*** Process Items: 1 with: 1 threads ***
Process: 1/1 : https://twitter.com/ACCOUNT-TO-FOLLOW
Selected account: MY-ACCOUNT
Use account: MY-ACCOUNT
MY-ACCOUNT Loaded previous db logs: Id: LOGS_Follow Data: 0 Ids: 0 Processed: 0 24: 0 Today: 0 GlobalCount: 0
0/1
Global Daily limit: 0
Processing: https://twitter.com/ACCOUNT-TO-FOLLOW with user: MY-ACCOUNT
Using acc: MY-ACCOUNT on: https://twitter.com/ACCOUNT-TO-FOLLOW
Get User: https://twitter.com/ACCOUNT-TO-FOLLOW with: MY-ACCOUNT
UserDetails: https://twitter.com/ACCOUNT-TO-FOLLOW with: MY-ACCOUNT  Proxy: PROXY-DETAILS 
200
https://twitter.com/i/api/graphql/...
found: @ACCOUNT-TO-FOLLOW
* SUCCESS: UserDetails: https://twitter.com/ACCOUNT-TO-FOLLOW with: MY-ACCOUNT  Proxy: PROXY-DETAILS 
follow: https://twitter.com/ACCOUNT-TO-FOLLOW with: MY-ACCOUNT  Proxy: PROXY-DETAILS 
* SUCCESS: follow: https://twitter.com/ACCOUNT-TO-FOLLOW with: MY-ACCOUNT  Proxy: PROXY-DETAILS 
MY-ACCOUNT processed: 1 items this action
* SUCCESS (1): https://twitter.com/ACCOUNT-TO-FOLLOW user: MY-ACCOUNT
Stopping as got enough...
Pausing for: 10
Paused for: 10 seconds
All items: 1
Remaining items: 0

*********
Processed: 1/1
Failed: 0
*********

Even though I redacted any personal information for public sharing and/or personal privacy reasons, the issue for the "follow" functionality is still easily noticeable.

The "https://twitter.com/ACCOUNT-TO-FOLLOW" was already been followed by "MY-ACCOUNT" but the process/software still attempted and "successfully" processed it as if the followed account weren't followed before.

It's clear that the "follow" function either fails to or never checks for the already processed items before attempting to process them.

Also, if you really do NOT want to pause on a skip item,. I can add a config value for this so there is not pause (not recommended but if you want I can do it)

In my case, I believe these changes/additions would be enough to solve my problems:

1. Adding a special/separate "random pause timer" feature/option just for the already processed items.
2. Ensuring the "Unique:True" option to never miss/evade any of the list items and deem them as "remaining". In short, processing all items in one sitting.
AND/OR
Ensuring proper checks for already followed items/accounts for the "follow" function while "Unique:False" is active.

https://rootjazz.com/twitterdub/updatetesting.html

tried to make some improvements to the areas I think it may be. But I'm not sure I fully understand your issue, so it probably doesn't fix it - but maybe

Thanks for the update. I will take a look in a bit and report my findings.

[martin@rootjazz]

They are NOT detected as already followed and still being attempted/processed as if they weren't followed already.

Set unique true. Why are you setting the option to false, when you DO want unique?

Although I can see the point, unique or not, if they are followed they are not going to be followed again.

[martin@rootjazz]

ok next update regardless of unique will test if already followed




Regards,
Martin

[Rev]

Set unique true. Why are you setting the option to false, when you DO want unique?

Although I can see the point, unique or not, if they are followed they are not going to be followed again.

As I mentioned multiple times before, the "Unique:True" option is really unreliable for my use-case. It skips many list items and calls them "remaining" at the end of the process. What I want is to process entirety of a list in one go. So, that's why I had to switch back to "Unique:False" to at least ensure the integrity of what is being processed.

P.S. I try my best to keep my databases and logs corruption free. I even tried using the "Unique:True" option with a fresh config (as if I'm installing the software to a new machine), the results were the same.

[martin@rootjazz]

Set unique true. Why are you setting the option to false, when you DO want unique?

Although I can see the point, unique or not, if they are followed they are not going to be followed again.

As I mentioned multiple times before, the "Unique:True" option is really unreliable for my use-case.

Doesn't mean it's going to work better if you don't use it though. hopefully those other issues have been fixed, but you do need to use the "unique" option if you want unique results.

So, that's why I had to switch back to "Unique:False" to at least ensure the integrity of what is being processed.

Then unique checks are not made (in previous versions.

Next update will make a "is_already_following" check now regardless

[Rev]

Doesn't mean it's going to work better if you don't use it though. hopefully those other issues have been fixed, but you do need to use the "unique" option if you want unique results.

I totally agree with you but, for some unknown reason, the "Unique:False" option gets me better results. And, it will even be better with the upcoming updates you have mentioned.

I will take a deeper look once my workflow is less crowded.

Then unique checks are not made (in previous versions.

Next update will make a "is_already_following" check now regardless

That's great!

Could you also introduce an option for removal of pause-timers for items that are deemed already processed before? The ability to specifically change the pause-timers for said already-processed items would even be greater than a straight out removal option.

[martin@rootjazz]

try this, it will check if used is already followed with / without unique option

https://rootjazz.com/twitterdub/updatetesting.html

Also, I think I fixed the "remaining" issue (not sure if it was this or previous release)


As always let me know if it works better / does / does not resolve your issue.



Regards,
Martin

[Rev]

try this, it will check if used is already followed with / without unique option

https://rootjazz.com/twitterdub/updatetesting.html

Also, I think I fixed the "remaining" issue (not sure if it was this or previous release)


As always let me know if it works better / does / does not resolve your issue.



Regards,
Martin

I just ran my main operation and it's all good now. My follow processes with "Unique:False" literally take 3 times less the time (with the same pause-timers) to complete their work. As the program isn't fiddling around with already processed items anymore, I believe this small change also enhanced accounts' security against random suspensions as well.

A report on "Unique:True" is also coming in a couple of days. I haven't yet tried it but I applied the option to my main processes now. Lets hope that pesky "remaining" issue is also got fixed, fingers crossed

All in all, thanks for the update!

[martin@rootjazz]