Scraping user details lists the twitter shortened link only

Support / help / discussion forum for twitter bot
Post Reply
rsbmanish
Posts: 419
Joined: Wed Jun 28, 2017 9:56 am

Scraping user details lists the twitter shortened link only

Post by rsbmanish »

When I scrape details of a user, instead of the visible web link, it scrapes only twitter shortened link. Pls can we add the visible links too on scraping. Also it would be nice if we can scrape details extra too. For follower count and tweets
support@rootjazz
Posts: 2317
Joined: Tue Feb 02, 2016 1:58 pm

Re: Scraping user details lists the twitter shortened link only

Post by support@rootjazz »

rsbmanish wrote: Sat Mar 17, 2018 2:03 pm When I scrape details of a user, instead of the visible web link, it scrapes only twitter shortened link. Pls can we add the visible links too on scraping. Also it would be nice if we can scrape details extra too. For follower count and tweets
Can you tell me how you are doing your this?
How are you setting up the search? A screnshot or instructions would be appreciated along with the issue.

As long as i knew the bot works through the twitter api and perhaps that is the reason why the urls are returned as twitter shortened [t.co] style. I am requesting admin to havea look on this and for details extra returning. Please be patient. Also, if you provide the scraping instructions then it would be more easy to find the solution for you.



Regards
rsbmanish
Posts: 419
Joined: Wed Jun 28, 2017 9:56 am

Re: Scraping user details lists the twitter shortened link only

Post by rsbmanish »

I just searched for followers of a profile and selected to scrape details. The ones which has websites in them scraped the short link and not what was displayed
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping user details lists the twitter shortened link only

Post by martin@rootjazz »

Let me see how the data is specified. The default is how you are getting it, but maybe twitter has another property with the raw link.

If you want, I can dump all the data as received in JSON format so you can load it into a db (maybe excel I don't know if excel can take JSON). Otherwise you need a custom formatter to put it in CSV and that will take me longer todo
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping user details lists the twitter shortened link only

Post by martin@rootjazz »

next update will parse out the correct URL and include followers / followings / status count in output
rsbmanish
Posts: 419
Joined: Wed Jun 28, 2017 9:56 am

Re: Scraping user details lists the twitter shortened link only

Post by rsbmanish »

This update is still passing the twitter short link instead of real link.
support@rootjazz
Posts: 2317
Joined: Tue Feb 02, 2016 1:58 pm

Re: Scraping user details lists the twitter shortened link only

Post by support@rootjazz »

rsbmanish wrote: Wed Mar 21, 2018 10:06 pm This update is still passing the twitter short link instead of real link.
The Page https://rootjazz.com/twitterdub/updatetesting.html has no link to download version 1.415 . Please wait for the link fix. I have reported this to martin.


Regards
rsbmanish
Posts: 419
Joined: Wed Jun 28, 2017 9:56 am

Re: Scraping user details lists the twitter shortened link only

Post by rsbmanish »

oh i didnt notice that. thanks
I edited the link to get latest version and its working great. Thanks a lot martin
support@rootjazz
Posts: 2317
Joined: Tue Feb 02, 2016 1:58 pm

Re: Scraping user details lists the twitter shortened link only

Post by support@rootjazz »

rsbmanish wrote: Thu Mar 22, 2018 8:26 am oh i didnt notice that. thanks
I edited the link to get latest version and its working great. Thanks a lot martin
Hope the version worked according to your requirements. :)


Regards
Post Reply