Can't scrape followings (followers too)

[Bartekef]

Hey
So I just tried to scrape followings of my profile, the action starts but it seems like it doesn't work. Returns me:

Starting: 19/04/2017 20:48 PM
Perform custom search
Max Results Wanted: 5000
Setup custom search controller
One search stage detected, setting default per item per stage value to max: 5000
Custom search run: search: https://twitter.com/xxxxx
Perform custom search: #chain/total: 1/1 using: xxxxx
Start search: FollowingsOf with: https://twitter.com/xxxxxxx using: xxxxx
Get TwitterUser: xxxxxxx using: xxxxx
Got user: https://twitter.com/xxxxxx username: xxxxxx #followings: 5354


And stays still. Nothing moves. Just like it stuck. Funny thing, I tride 3 or 4 times the same with scrapping followers, and it also stucks every time but even tho I have .txt file of IDs in saved_data folder (even tho the action still is stuck, and 'processing', never completed)

Logs: 41647

[martin@rootjazz]

I think you are just not waiting enough.

twitter scrape followers grabs as many as possible in a single pull, then has to translate the IDs to profileurls / details look at the logs:

08:23:11: Got followings Ids: 5000
08:39:52: Results of search: FollowingsOf with: https://twitter.com/wronabeats

so it grabs 5000 IDS, it then has to translate that to your output.

look at the times, it took 16 minutes, but it did complete.


Just wait. Or if you are just scraping to feed the results into Twitterdub, then return IDs as it will be quicker and TD can work with IDs (in fact works better with IDs). Profile URLs as result type is just if you want to use it manually / human readable

[Bartekef]

Ayeee got you
Thanks