URGENT - Following stuck at checking filter active`

[ron]

If can confirm this, I can take a look at what twitter returns for the search by default and see if you can filter on that - thus not requiring any additional calls and should *fix* this.

I am requesting a total of 200 user follows per process for 100 users, it's being shared across 100 users with 2 follow per user in a process..and it repeats every 10 minutes. And I have confirmed, searches go a lot smoother when i don't put in the active filter, so yes having that as a separate API call is a big problem. Also maybe you are taking the initial few accounts to search everytime when sharing across a big list instead of taking accounts randomly so the accounts at the top are not overused ?
Looks like check filter -> followings also has the same issue of separate api call
FYI I am generally searching for generic keyword groups like [sexy,hot] etc. Please fix this asap, check my logs at the end of this post on what happened on a recent search with active filter which gives ratelimiting errors, same old jazz, additionaly doing a new custom search every 10 followers is the biggest bug ever

For a single account, then it should work fine (although for a strict filter / poor search it may still get limited)

If I run on individual accounts, how do i maintain unique follows per account in a group ? Makes no sense


Also please remove the waiting time of 796 seconds for rate limiting and move to a new account to search instead.

Are you running a large shared across accounts action requesting a large number of results to share?

How many results are you requesting?

Checking last active, does require API calls, for a large result set this will just hit the limits and it may not be the best way for you to find user to follow. For a single account, then it should work fine (although for a strict filter / poor search it may still get limited)


If can confirm this, I can take a look at what twitter returns for the search by default and see if you can filter on that - thus not requiring any additional calls and should *fix* this.

Failing that, another way is to just scrape a huge list and follow from list with filter. The filter will then happen WITH the actioning account at the point of actioning, as opposed with the account selected at random for the search

I AM SHARING BETWEEN 100 ACCOUNTS, WHAT'S THE POINT OF RATE LIMITING FOR 824 SECONDS ? CANT YOU JUST ADD CODE TO CYCLE TO A NEW USER AND RATE LIMIT ONLY WHEN ALL THE USERS HAVE BEEN CYCLED THROUGH AND TESTED ??
WHY NOT LEAVE THAT RATE LIMITED ACCOUNT ALONE FOR A WHILE ?

Just don't get a lot of logic on the code level.

* FAILED: Term: 684233588716498945 98734895 : Couldn't get user from value: 684233588716498945
Results of search: CreatorOfTweet with: 684233588716498945
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [4] 2880991049
Perform custom search: #chain/total: 5/5
Start search: FollowersOf with: 2880991049
Get TwitterUser: 2880991049 using: Ashley79662
Got user: 2880991049 username: Mulher Maravilha #followers: 23516
Got followerIds: 10
Checking Filter: LastActive
Get TwitterUser: 3626240657 using: Elana93930
* FAILED check active: lastActive: 6 > 83
* IGNORE: active date
* IGNORE: Filtered: 3626240657
Checking Filter: LastActive
Get TwitterUser: 2382853022 using: Elana93930
* FAILED check active: lastActive: 6 > 83
* IGNORE: active date
* IGNORE: Filtered: 2382853022
Checking Filter: LastActive
Get TwitterUser: 4719465197 using: Elana93930
* FAILED check active: lastActive: 6 > 83
* IGNORE: active date
* IGNORE: Filtered: 4719465197
Checking Filter: LastActive
Get TwitterUser: 3174379150 using: Elana93930
Completed Total Found: 10
Checking Filter: LastActive
Get TwitterUser: 2631103725 using: Elana93930
Completed Total Found: 10
Checking Filter: LastActive
Get TwitterUser: 4429403974 using: Elana93930
Completed Total Found: 10
Checking Filter: LastActive
Get TwitterUser: 2610102413 using: Elana93930
Completed Total Found: 10
Checking Filter: LastActive
Get TwitterUser: 2153308654 using: Elana93930
Completed Total Found: 10
Checking Filter: LastActive
Get TwitterUser: 4713952343 using: Elana93930
Completed Total Found: 10
Checking Filter: LastActive
Get TwitterUser: 3292364502 using: Elana93930
Completed Total Found: 10
Results of search: FollowersOf with: 2880991049
End of chain: Store results: 7
StoreResults: checking: 7
Checking Filter: LastActive
Completed Total Found: 10
Checking Filter: LastActive
Completed Total Found: 10
Checking Filter: LastActive
Completed Total Found: 10
Checking Filter: LastActive
Completed Total Found: 10
Checking Filter: LastActive
Completed Total Found: 10
Checking Filter: LastActive
Completed Total Found: 10
Checking Filter: LastActive
Completed Total Found: 10
Stored: 7
Total Results so far: 12
Perform chained search: [3] 684230039504863232
Perform custom search: #chain/total: 4/5
Start search: CreatorOfTweet with: 684230039504863232
Checking Filter: LastActive
Get TwitterUser: 684230039504863232 using: Elana93930
* FAILED to get user: 684230039504863232
* ERROR: Not Found - The URI requested is invalid or the resource requested, such as a user, does not exists. Also returned when the requested format is not supported by the requested method.
404
https://api.twitter.com/1.1/users/show. ... 9504863232

34 - Sorry, that page does not exist.

* FAILED: Term: 684230039504863232 98734895 : Couldn't get user from value: 684230039504863232
Results of search: CreatorOfTweet with: 684230039504863232
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [4] 3127971605
Perform custom search: #chain/total: 5/5
Start search: FollowersOf with: 3127971605
Get TwitterUser: 3127971605 using: Ashley79662
Got user: 3127971605 username: Ray #followers: 129
Got followerIds: 10
Checking Filter: LastActive
Get TwitterUser: 4134403932 using: Elana93930
Completed Total Found: 10
Checking Filter: LastActive
Get TwitterUser: 3637334955 using: Elana93930
* WAITING: rate limited: Ready in: 824 seconds
* WAITING: rate limited: Ready in: 824 seconds
* WAITING: rate limited: Ready in: 824 seconds

[ron]

Any updates ?

[qqg1112]

I have the same problem

[martin@rootjazz]

when i don't put in the active filter, so yes having that as a separate API call is a big problem.

It is always going to require an API call (And in some cases multiple) to find out when the user last did something

As a temporary solution, if you want active users and using generic high traffic keywords, then just do a recent search with no active filter.

Also maybe you are taking the initial few accounts to search everytime when sharing across a big list instead of taking accounts randomly so the accounts at the top are not overused ?

Should be randomised when action first runs, on a repeat though, not 100% sure it will be re-randomised, will ensure
Updated the custom search so it will now swap accounts for each part of the search and for each filter check

additionaly doing a new custom search every 10 followers is the biggest bug ever

This is by design and required for some styles of search. However will cause issues for different styles of searches which should be overriding it. Will run some checks

[ron]

It is always going to require an API call (And in some cases multiple) to find out when the user last did something
Updated the custom search so it will now swap accounts for each part of the search and for each filter check

Please bear in mind that i am running 100 accounts for every custom search using share across accounts. I think 100 is the maximum anyone would run per instance.
So please randomize use of accounts accordingly, to avoid rate limits. Right now, the second i put in filter as active, even for untouched accounts or old accounts, i get rate limited in just 50 searched results.

additionaly doing a new custom search every 10 followers is the biggest bug ever

This is by design and required for some styles of search. However will cause issues for different styles of searches which should be overriding it. Will run some checks

Well the end result is mostly going to be followers/followings of some user -- so why not just complete that list of followers for searching and then move to next atleast for that broad case.
Maybe we can initiate a new top level search in the next process but atleast in the current one, a new search every 10 followers when we already see that the current user we are scraping has 70k followers might be bad since it leads to more api calls. Solving that could fix a lot of things including the token deauthorization bug maybe

Also i have a question, when i do a mixed search, will I get a new object everytime or is it the same tweet - how is mix defined in the algo, is it a random selection from a search on tweets?

[martin@rootjazz]

changing accounts during search / filter

https://rootjazz.com/twitterdub/updatetesting.html

[martin@rootjazz]

Also i have a question, when i do a mixed search, will I get a new object everytime or is it the same tweet - how is mix defined in the algo, is it a random selection from a search on tweets?

twitter decides, the program tells twitter
give me popular tweets
or
give me recent tweets
or
give me a mixed selection of recent / popular.

[martin@rootjazz]

Well the end result is mostly going to be followers/followings of some user -- so why not just complete that list of followers for searching and then move to next atleast for that broad case.

Very quickly as have to go out. A custom search could have many steps, if that is the case, it would not want to pull everything for every result for every step - would kill the limits.

However, got a small step custom search, it was / is meant to change the number of results per step accordingly. But from what you say this is not working correctly

Maybe we can initiate a new top level search in the next process but atleast in the current one, a new search every 10 followers when we already see that the current user we are scraping has 70k followers might be bad since it leads to more api calls. Solving that could fix a lot of things including the token deauthorization bug maybe

Granted, however there are cases where other methods of extracting could be wanted. Fix one issue, create another.

Ultimately, what I think is required is the custom search will need to allow you to specify the results of the step


so lets say
tweet search
creator
followers of
latest tweets


if a total number of 100 results are wanted, there are numerous ways to do this

get 10 tweets = 10 creators
get 10 followers from each creator = 100 followers
get latest tweet
(or 10 most recent) = 1000 shuffle / 10


however, if the goal is purely find 100 users, the easiest way to step through without hitting limits
max tweets to return in a single call = 100
start on tweet 1 - get creator
get max results in a single call, say 100
get latest tweet

done

however, all the followers came from the first creator. This may be ok for some, but not others.

Other issues with the latter is run it again and you scrape the same followers from the same creator, ok so we ignore some, but we then loop through all followers of that user, unless a limit is set.


Unfortunately there is no magic number that works as all things to all people. The only way is a custom max results per step, which will complicate it for those who don't need it, but for those who do, should sort out any issues

[ron]

Okay, a temporary solution would be change the limit from 10 to 30 to reduce api calls.
And make filter checking and user retrieving in one single api call if possible because whenever I use filters, I am hit with rate limits 95% of the time, so filters is basically of no use unless some changes are made.
Also I am fearing
P.S - Checked with updates on fresh 100 SHARED accounts - rate limited in just 2 seconds of trying to follow with the check filter thing on for active
Pasting the log, follow is still the same yet, please fix filter on high priority ! Also it is not even switching accounts when rate limited, just waiting for last 20 minutes

Custom Search: [bra,sexy,porn] using 100 accounts: Max: 101
Processing: [bra,sexy,porn] max: 101
*** Pre Search Setup***
*** Performing Search ***
Is multi search
Extract multi terms
term: porn
term: sexy
term: bra
multi search terms: bra, sexy, porn
Max: 101: PerTerm max: 33
Searching: bra
custom search builder
Loading previous db logs: 0852caaa-61fd-4136-af40-2ac56f32f2bc_Follow
Loaded previous db logs: 0 within 24hrs: 0
Setup custom search controller
Custom search run: search: bra
Perform custom search: #chain/total: 1/3
Force initial result set to 100 to ensure enough results from first chain search to check subsequent results of chains
Start search: Tweet Search (mixed) with: bra
Searching: bra type: mixed
Found: 100
Got: 100 from tweet search
Unique: 96 from tweet search
Results of search: Tweet Search (mixed) with: bra
Got results from chain: 96 feed into next chained custom search item
Perform chained search: [2] 103012173
Perform custom search: #chain/total: 3/3
Start search: FollowersOf with: 103012173
Get TwitterUser: 103012173 using: Mandy68790
Got user: 103012173 username: CHARLI XCX #followers: 1719837
Got followerIds: 10
Checking Filter: LastActive
Get TwitterUser: 4226498765 using: Kara7179
Get TwitterUser: 4226498765 using: Kara7179
Got Timeline: null: of: DCsteelers12
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 4226498765
Checking Filter: LastActive
Get TwitterUser: 3624029957 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 3624029957
Checking Filter: LastActive
Get TwitterUser: 4720520774 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 4720520774
Checking Filter: LastActive
Get TwitterUser: 2222167581 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 2222167581
Checking Filter: LastActive
Get TwitterUser: 3241805825 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 3241805825
Checking Filter: LastActive
Get TwitterUser: 481968395 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 481968395
Checking Filter: LastActive
Get TwitterUser: 2413081069 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 2413081069
Checking Filter: LastActive
Get TwitterUser: 4728715121 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 4728715121
Checking Filter: LastActive
Get TwitterUser: 4720562623 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 4720562623
Checking Filter: LastActive
Get TwitterUser: 476128964 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 476128964
Results of search: FollowersOf with: 103012173
End of chain: Store results: 0
StoreResults: checking: 0
Stored: 0
Total Results so far: 0
Perform chained search: [2] 87818409
Perform custom search: #chain/total: 3/3
Start search: FollowersOf with: 87818409
Get TwitterUser: 87818409 using: Andrea67033
Got user: 87818409 username: The Guardian #followers: 4886004
Got followerIds: 10
Checking Filter: LastActive
Get TwitterUser: 4728855988 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 4728855988
Checking Filter: LastActive
Get TwitterUser: 4728766095 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 4728766095
Checking Filter: LastActive
Get TwitterUser: 107825641 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 107825641
Checking Filter: LastActive
Get TwitterUser: 4720559120 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 4720559120
Checking Filter: LastActive
Get TwitterUser: 440453612 using: Kara7179
Completed Total Found: 0
* FAILED check active: lastActive: 6 > 47
* IGNORE: active date
* IGNORE: Filtered: 440453612
Checking Filter: LastActive
Get TwitterUser: 4728729256 using: Kara7179
* WAITING: rate limited: Ready in: 865 seconds
* WAITING: rate limited: Ready in: 865 seconds
* WAITING: rate limited: Ready in: 865 seconds
* WAITING: rate limited: Ready in: 865 seconds

Well the end result is mostly going to be followers/followings of some user -- so why not just complete that list of followers for searching and then move to next atleast for that broad case.

Very quickly as have to go out. A custom search could have many steps, if that is the case, it would not want to pull everything for every result for every step - would kill the limits.

However, got a small step custom search, it was / is meant to change the number of results per step accordingly. But from what you say this is not working correctly

Maybe we can initiate a new top level search in the next process but atleast in the current one, a new search every 10 followers when we already see that the current user we are scraping has 70k followers might be bad since it leads to more api calls. Solving that could fix a lot of things including the token deauthorization bug maybe

Granted, however there are cases where other methods of extracting could be wanted. Fix one issue, create another.

Ultimately, what I think is required is the custom search will need to allow you to specify the results of the step


so lets say
tweet search
creator
followers of
latest tweets


if a total number of 100 results are wanted, there are numerous ways to do this

get 10 tweets = 10 creators
get 10 followers from each creator = 100 followers
get latest tweet
(or 10 most recent) = 1000 shuffle / 10


however, if the goal is purely find 100 users, the easiest way to step through without hitting limits
max tweets to return in a single call = 100
start on tweet 1 - get creator
get max results in a single call, say 100
get latest tweet

done

however, all the followers came from the first creator. This may be ok for some, but not others.

Other issues with the latter is run it again and you scrape the same followers from the same creator, ok so we ignore some, but we then loop through all followers of that user, unless a limit is set.


Unfortunately there is no magic number that works as all things to all people. The only way is a custom max results per step, which will complicate it for those who don't need it, but for those who do, should sort out any issues

[ron]

Edited the same answer, please update