Scrape Tweet Images/Links?

Support / help / discussion forum for twitter bot
Post Reply
kevshTO
Posts: 101
Joined: Tue Oct 24, 2017 3:09 pm

Scrape Tweet Images/Links?

Post by kevshTO »

Hi,

I've been playing around with the scrape feature and haven't yet been able to figure out if it can download images included in tweets, or at least include the link in the created tweet file to the attached image. Is this possible?

Secondly, scraped tweets don't seem to include links contained in tweets either?

Lastly, is it possible to set the location of the file for scraped tweets besides the default?

Thanks for the help
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Tweet Images/Links?

Post by martin@rootjazz »

kevshTO wrote: Mon Jan 28, 2019 6:49 pm Hi,

I've been playing around with the scrape feature and haven't yet been able to figure out if it can download images included in tweets, or at least include the link in the created tweet file to the attached image. Is this possible?
No not currently possible. I will make a note to look at it. Remind me next week sometime if you don't hear from me
Secondly, scraped tweets don't seem to include links contained in tweets either?
Can you send a sample. A Tweet Url that was scraped, and how it was saved. Probably it is due to how the tweet is formatted in the response from twitter, maybe this is correct, maybe not and it needs some post scrape processing to format, but it will depend on where / how the link is specified for that tweet
Lastly, is it possible to set the location of the file for scraped tweets besides the default?

Thanks for the help
No, this is not possible
kevshTO
Posts: 101
Joined: Tue Oct 24, 2017 3:09 pm

Re: Scrape Tweet Images/Links?

Post by kevshTO »

Thanks for looking into this.

I'll run a few scrapes on tweets with links and see whether not including links is the norm or perhaps it was limited to a few based on format as you mentioned.
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Tweet Images/Links?

Post by martin@rootjazz »

do let me know as soon as you have some samples of it not working



Regards,
Martin
kevshTO
Posts: 101
Joined: Tue Oct 24, 2017 3:09 pm

Re: Scrape Tweet Images/Links?

Post by kevshTO »

Log ID: 4195

Okay, here's one that failed. From the output file in TwitterDub it looks like it messed up the Tweet URL by including the text in the actual URL (Error shown: Couldn't extract tweet ID: https://twitter.com/Find out what the definition...). At the very end of that string are two URLs, so one is likely an external link from the tweet, the other the URL of the attached image. So it does appear to be grabbing the URL at least.

The steps to create this (Scraper tab obviously):
- Max Results = 7 (just for testing)
- Scrape Tweets form/section:
- Custom Search = Recent Tweets of
- Scrape Tweets = (ID of the user)
- ID/URLs/Details = Details checked
- Click the Scrape Body button

I should note that in that test the output file saved in my local folder included a list of tweet IDs. I understand that the process failed, but shouldn't the output to the file be the actual tweet body/bodies? Is there something I'm doing wrong in the setup?

Thanks for the help.
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Tweet Images/Links?

Post by martin@rootjazz »

can you include the output saved file

and let me know the URL that was scraped and output wrongly
kevshTO
Posts: 101
Joined: Tue Oct 24, 2017 3:09 pm

Re: Scrape Tweet Images/Links?

Post by kevshTO »

Thanks for your reply. How do I include the output saved file?
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Tweet Images/Links?

Post by martin@rootjazz »

email it (sorry I should have said before)

Code: Select all

support[at]rootjazz[dot]com
Or just tell who was scraped who gave teh bad output and I can repeat it
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Tweet Images/Links?

Post by martin@rootjazz »

Post Reply