Checking for already processed items
Checking for already processed items
Hi Martin,
do you think it would be possible to somehow speed up the Like function? Because when you use older accounts with thousands of likes, the checking for already processed items is slowing the action significantly. (I use batch of 50 accounts with 20 threads for 1 shared action)
I'm not sure, if this is correct, but I have the impression, that the Instadub has different approach to this and it perform the checks for already processed items much faster.
Thanks.
do you think it would be possible to somehow speed up the Like function? Because when you use older accounts with thousands of likes, the checking for already processed items is slowing the action significantly. (I use batch of 50 accounts with 20 threads for 1 shared action)
I'm not sure, if this is correct, but I have the impression, that the Instadub has different approach to this and it perform the checks for already processed items much faster.
Thanks.
- martin@rootjazz
- Site Admin
- Posts: 34360
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Checking for already processed items
let me check that tumjazz is using a hash rather than a list for the check, as that can give vast speed improvements
- martin@rootjazz
- Site Admin
- Posts: 34360
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Checking for already processed items
no, tumjazz is using the fastest check.
submit your logs please so I Can see the times you are referring to and where the delays are coming from
submit your logs please so I Can see the times you are referring to and where the delays are coming from
- martin@rootjazz
- Site Admin
- Posts: 34360
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Checking for already processed items
forgot:
HELP > LOGS > SUBMIT
then send your logs ID - the first 4 numbers is sufficient (displayed after successful uploading of logs)
HELP > LOGS > SUBMIT
then send your logs ID - the first 4 numbers is sufficient (displayed after successful uploading of logs)
Re: Checking for already processed items
Sent the logs - ID: 24220
See the first likeshared action, it took 4h and 12m to complete ... when the accounts were new, it took about hour to complete.
I hope you'll find the way to speed it up
See the first likeshared action, it took 4h and 12m to complete ... when the accounts were new, it took about hour to complete.
I hope you'll find the way to speed it up
- martin@rootjazz
- Site Admin
- Posts: 34360
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Checking for already processed items
am sure can improve that, that is terrible
Re: Checking for already processed items
That would be awesome, I'm looking forward to it
- martin@rootjazz
- Site Admin
- Posts: 34360
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Checking for already processed items
sorry, seems I misunderstood your issue.
4 hours to post 12159 likes is one like every 1.8 seconds
You are passing in profiles, so the program needs to scrape a post, that takes time, if you want it faster, pass in posts
and as the processed logs get bigger, it will take longer. you just added 12k items which next time need to be checked against.
I thought you were saying that was a 4 hour delay at the start of the action for it to get going, not that the whole action took 4 hours.
tbh I think 4 hours for 12k likes, at 1 a second is good going. It isn't getting better than that. And I don't think it should. Faster is not always better
4 hours to post 12159 likes is one like every 1.8 seconds
You are passing in profiles, so the program needs to scrape a post, that takes time, if you want it faster, pass in posts
and as the processed logs get bigger, it will take longer. you just added 12k items which next time need to be checked against.
I thought you were saying that was a 4 hour delay at the start of the action for it to get going, not that the whole action took 4 hours.
tbh I think 4 hours for 12k likes, at 1 a second is good going. It isn't getting better than that. And I don't think it should. Faster is not always better
Re: Checking for already processed items
The real problem is, that 1 month ago, it took 1 hour to complete with the same setup. Now it takes 4 hours and if the accounts last, it will be probably more and more every day.
The reason why it takes longer is the comparing of the already processed items. That's why I asked you, if there's some room for improvement. For example, I think Instadub is comparing the array of already processed items with items in the source file before the action starts and it seems to be much faster.
The reason why it takes longer is the comparing of the already processed items. That's why I asked you, if there's some room for improvement. For example, I think Instadub is comparing the array of already processed items with items in the source file before the action starts and it seems to be much faster.
- martin@rootjazz
- Site Admin
- Posts: 34360
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Checking for already processed items
It shouldn't make a difference.
each item must be checked against merged processed records.
merge logs built
if check each item at point of processing, or at beginning, each item must be checked so is same time.
The delays may be caused by additional pauses I have put into the program due to issues with things going too fast and tumblr getting more strict.
each item must be checked against merged processed records.
merge logs built
if check each item at point of processing, or at beginning, each item must be checked so is same time.
The delays may be caused by additional pauses I have put into the program due to issues with things going too fast and tumblr getting more strict.