Scrape more then 70k followers

Support / help / discussion forum for twitter bot
Post Reply
alexliff
Posts: 48
Joined: Sun Oct 04, 2015 6:22 pm

Scrape more then 70k followers

Post by alexliff »

Hello Martin!
I hope you're well.

ver: 1.2.8.0

I've been trying to scrape followers from account which has 300k+ followers.
I set up "Max scrape results" to 300k in settings (all other settings are by default).

The program scrapes only 74k - 79k ... I tried different accounts/settings.
If I set up "Max scrape results" to 1k, 2k, 3k... it does exactly 1k, 2k, 3k.... So it works but doesn't want to cross 79k :)

I did it before (about 3 month ago) , everything was ok (unfortunately I don't remember the version of the program) .

I will be grateful for any help you can provide.

Thanks !
User avatar
martin@rootjazz
Site Admin
Posts: 34388
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape more then 70k followers

Post by martin@rootjazz »

nothing has changed on this side, so it could be twitter just not giving the full information.


But I'd need to see your logs. IIRC scrapes can continue from where they were - or used to and would pause with regards to limits.


Who are you trying to scrape and I can test here
alexliff
Posts: 48
Joined: Sun Oct 04, 2015 6:22 pm

Re: Scrape more then 70k followers

Post by alexliff »

Hi Martin ! Merry Christmas :))
I'm sorry for the delay with the answer and thanks for your time helping us.

I'm pretty sure this is a Twitter issue but it will be awesome to get any advise from you.

This time I was trying to get "followers of" from https://twitter.com/pianoguys (got only 74k from 120+k).

I've submitted logs :
Logs_27411
V 1.2.8.0

Here is my last action

Code: Select all

Perform custom search
Setup custom search controller
One search stage detected, setting default per item per stage value to max: 310000
Custom search run: search: https://twitter.com/pianoguys
Perform custom search: #chain/total: 1/1 using: *******************
Start search: FollowersOf with: https://twitter.com/pianoguys using: *******************
Get TwitterUser: pianoguys using: *******************
Got user: https://twitter.com/pianoguys username: ThePianoGuys #followers: 127160
Got Followers Ids: 74997
Results of search: FollowersOf with: https://twitter.com/pianoguys
Handle results: 74997 nextstep: 1/1
End of chain: Store results: 74997
StoreResults: checking: 74997
Stored: 74997
Total Results so far: 74997
Results: 74997
Saving to: C:\Users\Sergey\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercompianoguys_2016-12-27.txt
User avatar
martin@rootjazz
Site Admin
Posts: 34388
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape more then 70k followers

Post by martin@rootjazz »

when I test my logs are different.

How exactly do you build the search, do you request IDS profile urls details?

the custom search is just FOLLOWERS OF

you ensure the max results on the scraper tab to be 0
and the max results on the settings tab to be 99999999999 (or similar)

any other settings?
User avatar
martin@rootjazz
Site Admin
Posts: 34388
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape more then 70k followers

Post by martin@rootjazz »

I got 127,210 when testing.

But my logs are different, I hit rate limits, forced pause for 15 minutes and it carries on.


scrape tab
max results: 0
custom search: followers of
results: IDs

Settings tab: max results: 999999999
Post Reply