Scraping user details lists the twitter shortened link only
Scraping user details lists the twitter shortened link only
When I scrape details of a user, instead of the visible web link, it scrapes only twitter shortened link. Pls can we add the visible links too on scraping. Also it would be nice if we can scrape details extra too. For follower count and tweets
-
- Posts: 2317
- Joined: Tue Feb 02, 2016 1:58 pm
Re: Scraping user details lists the twitter shortened link only
Can you tell me how you are doing your this?
How are you setting up the search? A screnshot or instructions would be appreciated along with the issue.
As long as i knew the bot works through the twitter api and perhaps that is the reason why the urls are returned as twitter shortened [t.co] style. I am requesting admin to havea look on this and for details extra returning. Please be patient. Also, if you provide the scraping instructions then it would be more easy to find the solution for you.
Regards
Re: Scraping user details lists the twitter shortened link only
I just searched for followers of a profile and selected to scrape details. The ones which has websites in them scraped the short link and not what was displayed
- martin@rootjazz
- Site Admin
- Posts: 34360
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scraping user details lists the twitter shortened link only
Let me see how the data is specified. The default is how you are getting it, but maybe twitter has another property with the raw link.
If you want, I can dump all the data as received in JSON format so you can load it into a db (maybe excel I don't know if excel can take JSON). Otherwise you need a custom formatter to put it in CSV and that will take me longer todo
If you want, I can dump all the data as received in JSON format so you can load it into a db (maybe excel I don't know if excel can take JSON). Otherwise you need a custom formatter to put it in CSV and that will take me longer todo
- martin@rootjazz
- Site Admin
- Posts: 34360
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scraping user details lists the twitter shortened link only
next update will parse out the correct URL and include followers / followings / status count in output
- martin@rootjazz
- Site Admin
- Posts: 34360
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scraping user details lists the twitter shortened link only
This update is still passing the twitter short link instead of real link.
-
- Posts: 2317
- Joined: Tue Feb 02, 2016 1:58 pm
Re: Scraping user details lists the twitter shortened link only
The Page https://rootjazz.com/twitterdub/updatetesting.html has no link to download version 1.415 . Please wait for the link fix. I have reported this to martin.
Regards
Re: Scraping user details lists the twitter shortened link only
oh i didnt notice that. thanks
I edited the link to get latest version and its working great. Thanks a lot martin
I edited the link to get latest version and its working great. Thanks a lot martin
-
- Posts: 2317
- Joined: Tue Feb 02, 2016 1:58 pm