Long pauses during follow processes

Support / help / discussion forum for twitter bot
Post Reply
lombard
Posts: 42
Joined: Sun Aug 09, 2015 1:39 pm

Long pauses during follow processes

Post by lombard »

Hi Martin,

Have been using twitterdub for a few days now, just getting used to it.

Everything seems to work well, but when performing a follow operation there seem to often be very long pauses (5min - 20min) in the process. If I watch the log as it progresses, it just stays on the last task as if frozen, no message to say what is happening. Eventually the process starts again but a follow operation can take several hours this way (if say I am following 500 accounts). Do you have any idea why this would happen?

Just FYI, it makes no difference whether I use a proxy or not. Issue occurs with or without.

Have submitted logs for a very simple search/follow of 100 users, to illustrate. As you will see there are 2 long pauses in the process. Logs ID is 49910.
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Long pauses during follow processes

Post by martin@rootjazz »

The program will attempt to detect rate limits and wait for the limiting to pass before carrying on.


IF you are trying to follow 500 users without a pause, I imagine that is the issue
lombard
Posts: 42
Joined: Sun Aug 09, 2015 1:39 pm

Re: Long pauses during follow processes

Post by lombard »

Thanks for the info. A couple of follow up questions if that's OK.

1. Can you expand a bit on what rate limits, how they work and how the programme deals with them? Apologies if this is somewhere else in the documentation. If not, think would be useful for other users to know a bit more about them too.

2. Ok so 500 at once without a pause is too many from what you are saying. However, the follow action I sent you in the logs I submitted was only 100 users and it had 2 pauses of around 13 mins each during the processing. Is this expected behaviour?


James
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Long pauses during follow processes

Post by martin@rootjazz »

https://dev.twitter.com/rest/public/rate-limiting


Limits are per account. So a lot depends on what you are doing, what you have been doing and can be effected by threading / instances etc.
1. Can you expand a bit on what rate limits, how they work and how the programme deals with them?
Limits are when you have done too much too quick. Twitter responds with a "woah slow down" the program *TRIES* to detect this and force a pause to stop you hammering twitter when you are limited and stop you feeling their wrath. Ideally the program should log out on the pauses, but this is relying on a base library which has some issues in pushing a notification of ratelimits. I am waiting on the devs to fix this so it can be better reported in Twitterdub.
Post Reply