Checking for already processed items

Ask any support / help / issues / problem or question related to TumblingJazz
jrichards
Posts: 378
Joined: Tue Sep 22, 2015 11:58 pm

Checking for already processed items

Post by jrichards »

Hi Martin,

do you think it would be possible to somehow speed up the Like function? Because when you use older accounts with thousands of likes, the checking for already processed items is slowing the action significantly. (I use batch of 50 accounts with 20 threads for 1 shared action)

I'm not sure, if this is correct, but I have the impression, that the Instadub has different approach to this and it perform the checks for already processed items much faster.

Thanks.
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Checking for already processed items

Post by martin@rootjazz »

let me check that tumjazz is using a hash rather than a list for the check, as that can give vast speed improvements
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Checking for already processed items

Post by martin@rootjazz »

no, tumjazz is using the fastest check.

submit your logs please so I Can see the times you are referring to and where the delays are coming from
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Checking for already processed items

Post by martin@rootjazz »

forgot:
HELP > LOGS > SUBMIT

then send your logs ID - the first 4 numbers is sufficient (displayed after successful uploading of logs)
jrichards
Posts: 378
Joined: Tue Sep 22, 2015 11:58 pm

Re: Checking for already processed items

Post by jrichards »

Sent the logs - ID: 24220

See the first likeshared action, it took 4h and 12m to complete ... when the accounts were new, it took about hour to complete.

I hope you'll find the way to speed it up ;)
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Checking for already processed items

Post by martin@rootjazz »

am sure can improve that, that is terrible
jrichards
Posts: 378
Joined: Tue Sep 22, 2015 11:58 pm

Re: Checking for already processed items

Post by jrichards »

That would be awesome, I'm looking forward to it :)
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Checking for already processed items

Post by martin@rootjazz »

sorry, seems I misunderstood your issue.

4 hours to post 12159 likes is one like every 1.8 seconds

You are passing in profiles, so the program needs to scrape a post, that takes time, if you want it faster, pass in posts

and as the processed logs get bigger, it will take longer. you just added 12k items which next time need to be checked against.


I thought you were saying that was a 4 hour delay at the start of the action for it to get going, not that the whole action took 4 hours.

tbh I think 4 hours for 12k likes, at 1 a second is good going. It isn't getting better than that. And I don't think it should. Faster is not always better
jrichards
Posts: 378
Joined: Tue Sep 22, 2015 11:58 pm

Re: Checking for already processed items

Post by jrichards »

The real problem is, that 1 month ago, it took 1 hour to complete with the same setup. Now it takes 4 hours and if the accounts last, it will be probably more and more every day.

The reason why it takes longer is the comparing of the already processed items. That's why I asked you, if there's some room for improvement. For example, I think Instadub is comparing the array of already processed items with items in the source file before the action starts and it seems to be much faster.
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Checking for already processed items

Post by martin@rootjazz »

It shouldn't make a difference.

each item must be checked against merged processed records.

merge logs built

if check each item at point of processing, or at beginning, each item must be checked so is same time.


The delays may be caused by additional pauses I have put into the program due to issues with things going too fast and tumblr getting more strict.
Post Reply