Issues with "Stream" + A Few Questions

Support / help / discussion forum for twitter bot
User avatar
martin@rootjazz
Site Admin
Posts: 34387
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Issues with "Stream" + A Few Questions

Post by martin@rootjazz »

Random opinion question, but would you suggest having a separate aged twitter account to run scrapes on?
Depends what you are doing. But as long as you don't abuse twitter / twitter users, for scraping it doesn't matter IN MY EXPERIENCE.

That is all I can say. Perhaps others have had a different experience. As with most things you have to take everything you read with a pinch of salt. What works for others may not work for you. Get in the habit of testing testing testing, reviewing amending then testing some more
KaedeSkirata
Posts: 5
Joined: Wed Nov 25, 2015 2:39 pm

Re: Issues with "Stream" + A Few Questions

Post by KaedeSkirata »

All right - I'm back again! Merry Christmas!

Question 1: I'm getting API rate limit exceed errors from my Twitter Watch Stream (error 88 FYI) - I'm filtering the results of a VERY high-volume keyword by removing users who: a) are default avatar b) follow me c) are private. Then I filter out users with less than 100 followers, and then language target it to English, english and en (I re-iterate it because I'm not 100% clear on how the program executes it... was getting errors and profiles skipped over because english =/= English). This narrows a 1000 tweet an hour hashtag down to probably 50, but I can't follow any of them, as this API rate limit keeps dinging for me.

Question 1A: What information is actually pushed out by Twitter's streaming API? I was under the impression that all of the "filters" available under the watch tab filters were data points pushed out by Twitter, and thus didn't count against my API limits. Are there certain data points that necessitate a client-based API call? I'd like to avoid using those.

Question 2: I'm not clear on the difference between the "wait X time between handling tweets" option that's available under the global Watch Stream menu, the time-locks available on each individual action for each individual keyword, and whether or not "actioning" an item is processed before or after filtering that item.

E.G: My watch stream is set up to monitor #puppies, @cats, and #Christmas - three separate keywords.
#puppies is a high volume keyword that I want to narrow down to a laser-targeted group of filtered people to follow - I want to follow and like one heavily filtered and qualified user per minute. This is the one that gives me trouble.
@cats is a low-volume keyword that I want to immediately like and follow the users of each tweet - no filter necessary. (for the record, this works if someone @mentions cats, then the stream is triggered, right?)
#christmas is just e-mail and profile scraping

I set up my watch stream as such, but as far as I can tell, the bot just stores all the recent tweets in a queue, and when the 60 second wait time is up, "actions" an item by pulling the first queued tweet up and analyzing it. If it doesn't meet my highly granular filter, it gets thrown out and I miss that follow opportunity - except it's more like 48/50 times that I'll not follow someone based on a filter. This is only an issue because #puppies is a high-enough volume keyword that if the user filtering was handled prior to queueing the tweet for "actioning" (like and follow in this case) I could conceivably follow one new highly active account per minute until Twitter decides to sit me the eff down for automated activity.

Question 3: Seriously, I must be stupid, but I really don't understand the "wait between handing tweets" option.


Thanks for all your help! No rush on the reply - it's the holidays!
User avatar
martin@rootjazz
Site Admin
Posts: 34387
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Issues with "Stream" + A Few Questions

Post by martin@rootjazz »

KaedeSkirata wrote:All right - I'm back again! Merry Christmas!

Question 1: I'm getting API rate limit exceed errors from my Twitter Watch Stream (error 88 FYI) - I'm filtering the results of a VERY high-volume keyword by removing users who: a) are default avatar b) follow me c) are private. Then I filter out users with less than 100 followers, and then language target it to English, english and en (I re-iterate it because I'm not 100% clear on how the program executes it... was getting errors and profiles skipped over because english =/= English). This narrows a 1000 tweet an hour hashtag down to probably 50, but I can't follow any of them, as this API rate limit keeps dinging for me.
Depending on how you filter, there could be additional calls made. Some data is sent with the notification from twitter, some isn't and will require a call. For high traffic KWs, this could mean hitting the limits
because english =/= English).
I will fix that. En != english, but english should equal English.

Question 1A: What information is actually pushed out by Twitter's streaming API? I was under the impression that all of the "filters" available under the watch tab filters were data points pushed out by Twitter, and thus didn't count against my API limits. Are there certain data points that necessitate a client-based API call? I'd like to avoid using those.
I will have to check. The notification from twitter is not limited, so will be recieved
Question 2: I'm not clear on the difference between the "wait X time between handling tweets" option that's available under the global Watch Stream menu, the time-locks available on each individual action for each individual keyword, and whether or not "actioning" an item is processed before or after filtering that item.
you can specify to only process items once every X minutes. Anything that comes after until the pause is completed is ignored.

This mean you can specify to follow an account once every 2 minutes, so not to hammer the twitter servers so hard

I set up my watch stream as such, but as far as I can tell, the bot just stores all the recent tweets in a queue, and when the 60 second wait time is up, "actions" an item by pulling the first queued tweet up and analyzing it. If it doesn't meet my highly granular filter, it gets thrown out and I miss that follow opportunity - except it's more like 48/50 times that I'll not follow someone based on a filter. This is only an issue because #puppies is a high-enough volume keyword that if the user filtering was handled prior to queueing the tweet for "actioning" (like and follow in this case) I could conceivably follow one new highly active account per minute until Twitter decides to sit me the eff down for automated activity.
when you action an item, the pause is in place. Anything that comes in after is ignored for actioning (it WILL be scraped / stored if specified). Items are not put in a queue.

tweet 1 receved: meets filter
pause
tweet 2 receved: ignored
tweet 3 receved: ignored
tweet 4 receved: ignored
tweet 5 receved: ignored
pause ends
tweet 6 receved: fails filter
tweet 7 receved: fails filter
tweet 8 receved: fails filter
tweet 10 receved: meets filter
pause
...
.


so perhaps tweet 2 and 3 would have met your filter, but were ignored as you were pausing.

Question 3: Seriously, I must be stupid, but I really don't understand the "wait between handing tweets" option.
If you mean on the watch for kws tab (not setup actions / kw) it is global and applies across the board - across all keywords / actions. If you want more control, use the action / kw specific pauses
Post Reply