Process Item Checks for Non-Unique Items - Functionality Improvements

[Rev]

What I mean by the subject-title is, If we could have "processability" checks for the items without the "Unique item" check-box enabled to be processed just like the ones that are getting processed with the "Unique item" check-box enabled. Rather than having any database entries and/or logging just like items processed by "Unique item" check-box enabled, processes that are non-unique (not using the "Unique item" option) should have the functionality to check their list-entries/search-terms for engagements that are already been applied (or not) once before the time that software proceeds to "mistakenly" re-process them.

I know my wording for this matter is kinda confusing but, here is a simple example:

Lets say we have made 2 different "retweet" processes. Both processes has 10 different items (tweet links) that are already been retweeted before (with or without the help of the software) to process with the "Random Pause" timer of 10 seconds. The list of items (tweet links in this case) that is going to be used for both of the processes are exactly the same. The only difference between these 2 processes is, one having the "Unique item" check-box enabled and the other is not. Here is how they actually perform (at the current time - v3.564):

Process #1 - "Unique item" check-box is enabled:

[Processing Sequence]
Checks the list of item entries -> qualifies them if they are processed before -> ends the process with a "success" handle as all the items in the list is already been processed (retweeted) before

[Conclusion]
1. Applies no "Random Pause" timer & takes little to no time to complete.
2. Has a tendency to miss/evade some of the items and categorize them as "remaining", which is caused by the miss-match of database entries and such (according to other threads/posts in this forum).
3. Completes its process with mostly missing/unprocessed items and still calls the result handle a "success".

Process #2 - "Unique item" check-box is disabled:

[Processing Sequence]
Starts processing a random list entry/item without checking if it's already been retweeted before -> somehow processes & retweets the already retweeted item -> applies "Random Pause" timer to the process -> repeats this exact cycle till all the other 9 items are processed

[Conclusion]
1. Processes the already retweeted items and applies "Random Pause" timer per item processed in the list, which results in way longer processing time-frames (100 seconds in total for this example) than the first process.
2. Processes every single item in the list successfully and without any misses.
3. The whole process takes even longer to finish with a higher "Random Pause" timer and risking accounts integrity by re-processing the already retweeted items.

These examples are not theoretical. This is what you get by processing items with the mentioned parameters and options. This happens with all types of processes (retweets, likes, follows, etc.).

In conclusion, I know that I can just use the "Unique item" option for most of my processes (which I already tried for a long time) but, as mentioned above, the "Unique item" option is prone to errors which are fatal to what I'm trying to achieve. I can't tell if this whole thing is actually intended or a major bug but, all I'm suggesting is having some of these checks for the non-unique processing-types to ensure that the software doesn't process already processed items/entries which were processed by other means (without the help of the software).

[martin@rootjazz]

What I mean by the subject-title is, If we could have "processability" checks for the items without the "Unique item" check-box enabled to be processed just like the ones that are getting processed with the "Unique item" check-box enabled. Rather than having any database entries and/or logging just like items processed by "Unique item" check-box enabled, processes that are non-unique (not using the "Unique item" option) should have the functionality to check their list-entries/search-terms for engagements that are already been applied (or not) once before the time that software proceeds to "mistakenly" re-process them.

what?

I know my wording for this matter is kinda confusing but, here is a simple example:

I've re-read twice and I have no idea what you are referring to lol


So it's something to do with processing unique items. Unique items, will check the database during the SEARCH stage, these items are not included in the search results. Outside of TD tweets WILL be returned during the search, but I thought the program performed a check on the tweet_details endpoint before retweeting to make sure it was not retweeted (unless you specified to unretweet and retweet)

[Rev]

I know my wording for this matter is kinda confusing but, here is a simple example:

I've re-read twice and I have no idea what you are referring to lol


So it's something to do with processing unique items. Unique items, will check the database during the SEARCH stage, these items are not included in the search results. Outside of TD tweets WILL be returned during the search, but I thought the program performed a check on the tweet_details endpoint before retweeting to make sure it was not retweeted (unless you specified to unretweet and retweet)

I thought I was making my point more clear by giving an example but, I guess I should have included a tl;dr as well haha. It's actually quite simple but hard to explain on paper.

What I mean by all of this is, processes created without the "Unique item" check-box enabled doesn't perform the mandatory "tweet_details" endpoint checks to ensure that the tweet is already processed or not. This behaviour is apparent for most if not all the processing-types, such as retweeting, liking and following.

So, without the mandatory "tweet_details" endpoint checks, these processes fail to ensure if the given item is processed or not, with or without the help of the software. This causes the software to re-process the already processed items and call them a "success", while applying pause timers as if these processed items weren't processed before or were processable.

As my work machine is occupied at the moment, I will be sharing logs about this in a bit. That will definitely make this whole situation more understandable.

P.S. I have meant all this ONLY for processes created without the "Unique item" option enabled. Just to clear any confusion.

[martin@rootjazz]

What I mean by all of this is, processes created without the "Unique item" check-box enabled doesn't perform the mandatory "tweet_details" endpoint checks to ensure that the tweet is already processed or not. This behaviour is apparent for most if not all the processing-types, such as retweeting, liking and following.

strange, let me check on that for you.

Quick question, you are not using FAST MODE are you by any chance?

[Rev]

strange, let me check on that for you.

Quick question, you are not using FAST MODE are you by any chance?

I see the "fast mode" option you have mentioned but, I can't confirm if it's enabled or not without pressing the "turn on / off" context menu item (I guess). But I have never changed or enabled this option before. I guess it's disabled by default.

Edit:
I just checked and it wasn't/isn't enabled (no orange tick-box near the context-menu item).

[Rev]

Important Update:
I now able to test things out for this matter and just found out that the "retweet" function is working as intended. My apologies for the misinformation in that case. However, the actual issue is mainly happening with the "follow" function/processes. Also, the "like" function is kinda faulty as well. It kinda works as intended and can be able differentiate the tweets if they were processed/liked before but, it still applies pause timers even though it finds if the tweets were already processed/liked or not.

Simply put:
Retweets = Working as intended, no need for a fix.
Likes = Working as intended but, applies unnecessary pause timers (which isn't the case for the "retweet" function").
Follows = NOT working as intended, dismisses the fact that the accounts/profiles were followed before, applies unnecessary pause timers (which isn't the case for the "retweet" function").

Here are my test results:
Logs for a FOLLOWING process: logs_58287
Logs for a LIKING process: logs_29873

P.S. "Unique" option was NOT applied to any of the submitted/shared processes.

[martin@rootjazz]

it still applies pause timers even though it finds if the tweets were already processed/liked or not.

Yes because there was effort to work out if it was already processed. The risk being, if you have already processed everything before, without a delay program would blast through them all and hammer twitter server.

You shouldn't really be trying to reprocess items you have already done outside of the app.

Retweets = Working as intended, no need for a fix.

I'll add the delay for retweet

[Rev]

Yes because there was effort to work out if it was already processed. The risk being, if you have already processed everything before, without a delay program would blast through them all and hammer twitter server.

You shouldn't really be trying to reprocess items you have already done outside of the app.

My bad then. I thought the program first scrapes all the information from the given tweet links (etc.) and acts upon them later at once. Good thing that I notified about something missing from the "retweet" function without even understanding it correctly, haha.

Nevertheless, I'm definitely sure something is up with the "follow" function when it is processed without the "Unique item" option being on/enabled. Could you please take a look at my logs and/or test it by yourself when possible? This is really crucial for the work I'm trying to do.

About re-processing the same items, what I'm trying to achieve precisely requires this, at least for the accounts following part. You see, I got to follow bunch of accounts without any misses which are scraped by hand/manually. As I have hundreds of active accounts, I can't really order what accounts they already have followed before or not. It's highly possible that even one of my accounts to miss or haven't followed one or more of the required accounts to be followed in a hundreds bunch. I guess that the "Unique:True" option is probably intended for such cases but, it's really unreliable in this/my case and, as mentioned in my previous posts, it tends to evade/miss many items in a list and categorize these misses under the "remaining" tag/title at the end of the process. Anyways, lets not get off-track from the main topic and complicate things for now. A new topic/thread for another day, haha.

I really appreciate the work you put in & the daily support you provide. I hope I'm not being a burden with all the things I come up with

[martin@rootjazz]

Yes because there was effort to work out if it was already processed. The risk being, if you have already processed everything before, without a delay program would blast through them all and hammer twitter server.

You shouldn't really be trying to reprocess items you have already done outside of the app.

My bad then. I thought the program first scrapes all the information from the given tweet links (etc.) and acts upon them later at once. Good thing that I notified about something missing from the "retweet" function without even understanding it correctly, haha.

No it will happen just before the action, that way it can check the status just before it (likes / follows / retweets) if it did it at the start, by the time it got round to the actioning part, the status may be out of date

Nevertheless, I'm definitely sure something is up with the "follow" function when it is processed without the "Unique item" option being on/enabled. Could you please take a look at my logs and/or test it by yourself when possible? This is really crucial for the work I'm trying to do.

sorry, what is the issue with follow?

You see, I got to follow bunch of accounts without any misses which are scraped by hand/manually. As I have hundreds of active accounts, I can't really order what accounts they already have followed before or not.

Ok I understand. You just want to be 100% sure all the list has actually been followed, so you may re-process the list multiple times.

I guess that the "Unique:True" option is probably intended for such cases

yes exactly.

but, it's really unreliable in this/my case and, as mentioned in my previous posts, it tends to evade/miss many items in a list and categorize these misses under the "remaining" tag/title at the end of the process.

Can you explain this. Are you saying they are NOT detected as already followed? Or just not attempted?

Select a list of 100 items
Select account(s)
specify 100 items

all items in list are processed (should be). What is / isn't happening for you?


LEt me read through the thread again as maybe I've missed it

[martin@rootjazz]

Also, if you really do NOT want to pause on a skip item,. I can add a config value for this so there is not pause (not recommended but if you want I can do it)