Follow process

[shaily]

Hi Martin,
I think there is an opportunity for improvement.... Look at the case below:
We have 10,000 profiles in a list to follow which has to run once a day to follow 100 profiles. I see that everyday, twitterdub runs the entire 10000 profiles. Though, it should not run 100 profiles on day two, which it ran on day one... or even better, if Twitterdub finds out on a day that there were 50 profiles who were already processed earlier, it should mark them in the database for that list and not run for that list again. Atleast for that list.
Just a thought.
Hopefully, I am making sense.
Cheers,
Shaily

[martin@rootjazz]

The program shouldn't run the same profiles again (unless you have told it to).

Are you sure you are not seeing the logs saying "this item has been processed before, it's being ignored" until it has the 100 items it needs for this run, then processes them. That is what people usually see when reporting the program is performing duplicate items when it isn't.

But without logs, I cannot be sure what is going on



Regards,
Martin

[shaily]

What you have explained is the way it works...

The problem is:
If it has to follow 10000 profiles from a list with 100 each day, then on 90th day, it will check all the records it has performed check on previous 89 days, by checking it against the database. This increases the time significantly.
It may take only 5 minutes on Day 1 but it will be a couple of hours on 50th day.... just because it still has to check for all the profile, even though with the database itself.

If you had a marker for that list, then it should start from the next line item from the list, checking only the remaining profiles in the list. This will reduce the time to execute each time.

If not, I will explain with an example.

Hope that makes sense.

Cheers,
Shaily

[martin@rootjazz]

What you have explained is the way it works...

The problem is:
If it has to follow 10000 profiles from a list with 100 each day, then on 90th day, it will check all the records it has performed check on previous 89 days, by checking it against the database. This increases the time significantly.
It may take only 5 minutes on Day 1 but it will be a couple of hours on 50th day.... just because it still has to check for all the profile, even though with the database itself.

That isn't how it "should" work. The checking should be near instantaneous. It should just fly by then, few seconds at most. Sounds like something is going wrong, or is not optimised.

Can you submit logs showing this:

HELP > LOGS > SUBMIT

then send your logs ID - the numbers are sufficient (displayed after successful uploading of logs)

If you had a marker for that list, then it should start from the next line item from the list, checking only the remaining profiles in the list. This will reduce the time to execute each time.

In fact, when working with a list / filepath, the program will save out a new list of the non-processed items and use that new list so it shouldn't even check the processed items. Although there could be ways to setup the action that this doesn't happen.

But can you explain in more detail how you build / run this action. Then submit logs showing the action taking along time to pass the processed items.

I Can then look into why it isn't optimised, as I said above, it should blast through processed items (if exist in the list) so you don't notice any difference (5minutes and 5 seconds rather than 5 minutes kind of thing. That's if it isn't updating the filepath anywwhere.




Regards,
Martin