Hello Martin!
I hope you're well.
ver: 1.2.8.0
I've been trying to scrape followers from account which has 300k+ followers.
I set up "Max scrape results" to 300k in settings (all other settings are by default).
The program scrapes only 74k - 79k ... I tried different accounts/settings.
If I set up "Max scrape results" to 1k, 2k, 3k... it does exactly 1k, 2k, 3k.... So it works but doesn't want to cross 79k
I did it before (about 3 month ago) , everything was ok (unfortunately I don't remember the version of the program) .
I will be grateful for any help you can provide.
Thanks !
Scrape more then 70k followers
- martin@rootjazz
- Site Admin
- Posts: 34388
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scrape more then 70k followers
nothing has changed on this side, so it could be twitter just not giving the full information.
But I'd need to see your logs. IIRC scrapes can continue from where they were - or used to and would pause with regards to limits.
Who are you trying to scrape and I can test here
But I'd need to see your logs. IIRC scrapes can continue from where they were - or used to and would pause with regards to limits.
Who are you trying to scrape and I can test here
Re: Scrape more then 70k followers
Hi Martin ! Merry Christmas )
I'm sorry for the delay with the answer and thanks for your time helping us.
I'm pretty sure this is a Twitter issue but it will be awesome to get any advise from you.
This time I was trying to get "followers of" from https://twitter.com/pianoguys (got only 74k from 120+k).
I've submitted logs :
Logs_27411
V 1.2.8.0
Here is my last action
I'm sorry for the delay with the answer and thanks for your time helping us.
I'm pretty sure this is a Twitter issue but it will be awesome to get any advise from you.
This time I was trying to get "followers of" from https://twitter.com/pianoguys (got only 74k from 120+k).
I've submitted logs :
Logs_27411
V 1.2.8.0
Here is my last action
Code: Select all
Perform custom search
Setup custom search controller
One search stage detected, setting default per item per stage value to max: 310000
Custom search run: search: https://twitter.com/pianoguys
Perform custom search: #chain/total: 1/1 using: *******************
Start search: FollowersOf with: https://twitter.com/pianoguys using: *******************
Get TwitterUser: pianoguys using: *******************
Got user: https://twitter.com/pianoguys username: ThePianoGuys #followers: 127160
Got Followers Ids: 74997
Results of search: FollowersOf with: https://twitter.com/pianoguys
Handle results: 74997 nextstep: 1/1
End of chain: Store results: 74997
StoreResults: checking: 74997
Stored: 74997
Total Results so far: 74997
Results: 74997
Saving to: C:\Users\Sergey\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercompianoguys_2016-12-27.txt
- martin@rootjazz
- Site Admin
- Posts: 34388
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scrape more then 70k followers
when I test my logs are different.
How exactly do you build the search, do you request IDS profile urls details?
the custom search is just FOLLOWERS OF
you ensure the max results on the scraper tab to be 0
and the max results on the settings tab to be 99999999999 (or similar)
any other settings?
How exactly do you build the search, do you request IDS profile urls details?
the custom search is just FOLLOWERS OF
you ensure the max results on the scraper tab to be 0
and the max results on the settings tab to be 99999999999 (or similar)
any other settings?
- martin@rootjazz
- Site Admin
- Posts: 34388
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scrape more then 70k followers
I got 127,210 when testing.
But my logs are different, I hit rate limits, forced pause for 15 minutes and it carries on.
scrape tab
max results: 0
custom search: followers of
results: IDs
Settings tab: max results: 999999999
But my logs are different, I hit rate limits, forced pause for 15 minutes and it carries on.
scrape tab
max results: 0
custom search: followers of
results: IDs
Settings tab: max results: 999999999