Can twitterdub track trending topics, then scrape and retweet?

Support / help / discussion forum for twitter bot
lsberic
Posts: 75
Joined: Mon Aug 21, 2017 7:19 pm

Can twitterdub track trending topics, then scrape and retweet?

Post by lsberic »

Hi Rootjazz

With twitterdub I am wondering if the program can spot topics/keywords if they begin trending and then run a few automated commands or would I need to use twitterdub with another program (ala scrapebox) to get this working?

Here are my needs.

Twitterdub would need to spot certain keywords and/or hashtags when they begin trending and retweet the trending topic and scrape any links to see if the webpages have twitter tweet functions on them and tweet it out.

So for example, if the keyword black friday started trending could Twitterdub automatically see which accounts were tweeting? And let's say it's a store like Macy’s or Amazon, could twitterdub then scrape the website the tweet links to, look for a twitter retweet function and compose a tweet and send it out? Or could twitterdub simply retweet the Amazon/Macy's tweet?

Or would I need scrapebox do the scraping part and have twitterdub compose the tweet and retweet?

Thanks
User avatar
martin@rootjazz
Site Admin
Posts: 34388
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Can twitterdub track trending topics, then scrape and retweet?

Post by martin@rootjazz »

You can access trending tags on a per country basis using the trend hash token.

As for "spotting" trends, the program doesn't. But if you can define what you mean in steps, the program could be made to do it. But there needs to be some instructions on what you mean by "spot trending".
lsberic
Posts: 75
Joined: Mon Aug 21, 2017 7:19 pm

Re: Can twitterdub track trending topics, then scrape and retweet?

Post by lsberic »

martin@rootjazz wrote: Mon Nov 18, 2019 12:44 pm You can access trending tags on a per country basis using the trend hash token.
How does this access trending tags and can I define which tags to look for?
martin@rootjazz wrote: Mon Nov 18, 2019 12:44 pm As for "spotting" trends, the program doesn't. But if you can define what you mean in steps, the program could be made to do it. But there needs to be some instructions on what you mean by "spot trending".
I ran into a program recently that could "spot" when a tag is trending by using what it called a twitter stream agent. That combined with a peak detector, it looks at the twitter stream constantly for when a term begins to "peak" or trend. Then it spits out an email to the user. In this case I would want twitterdub to perform an action as described in the op.

Here is a link to a video here that explains it better: https://vimeo.com/tectonic/huginn-introduction. Take a look at about 2;09 in.
User avatar
martin@rootjazz
Site Admin
Posts: 34388
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Can twitterdub track trending topics, then scrape and retweet?

Post by martin@rootjazz »

lsberic wrote: Mon Nov 18, 2019 7:07 pm
martin@rootjazz wrote: Mon Nov 18, 2019 12:44 pm You can access trending tags on a per country basis using the trend hash token.
How does this access trending tags and can I define which tags to look for?
Log into your twitter account in teh website, top right on the home page (I think) it shows "trending hashtags for your region".

There is nothing to specify. You can access those trending tags for your country (as specified in settings).
martin@rootjazz wrote: Mon Nov 18, 2019 12:44 pm As for "spotting" trends, the program doesn't. But if you can define what you mean in steps, the program could be made to do it. But there needs to be some instructions on what you mean by "spot trending".
I ran into a program recently that could "spot" when a tag is trending by using what it called a twitter stream agent. That combined with a peak detector, it looks at the twitter stream constantly for when a term begins to "peak" or trend. Then it spits out an email to the user. In this case I would want twitterdub to perform an action as described in the op.

Here is a link to a video here that explains it better: https://vimeo.com/tectonic/huginn-introduction. Take a look at about 2;09 in.
It is using the twitter API stream endpoint. TD does NOT use the API so cannot perform this. Plus, unless the streaming endpoint has changed since I looked at it, I would doubt it is actually detecting trends, just returning what it finds on the stream connected to based on some rules set by the program. As I said, it needs to be defined exactly to have the program do it.

watch this endpoint
when a word is mention (n) times in (y) minutes, mark it as trending. But is that actually trending, or just something spammed out?

For actual trends, use the trending tags. Twitter is telling you these are currently trending.
lsberic
Posts: 75
Joined: Mon Aug 21, 2017 7:19 pm

Re: Can twitterdub track trending topics, then scrape and retweet?

Post by lsberic »

martin@rootjazz wrote: Mon Nov 18, 2019 9:06 pm
For actual trends, use the trending tags. Twitter is telling you these are currently trending.
Ok, I'll take a look at it. I assume a tag can be a keyword. Once that's discovered, can twitterdub perform an action like:
  • send out a tweet regarding that keyword or tag.
  • retweet tweets that contain the keyword or tag.
  • scrape tweet that contain the keyword/tag and look for links to webpages, then find the tweet function on the webpage and send out a tweet?
Thanks
User avatar
martin@rootjazz
Site Admin
Posts: 34388
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Can twitterdub track trending topics, then scrape and retweet?

Post by martin@rootjazz »

lsberic wrote: Tue Nov 19, 2019 4:03 pm
martin@rootjazz wrote: Mon Nov 18, 2019 9:06 pm
For actual trends, use the trending tags. Twitter is telling you these are currently trending.
Ok, I'll take a look at it. I assume a tag can be a keyword. Once that's discovered, can twitterdub perform an action like:
  • send out a tweet regarding that keyword or tag.
  • retweet tweets that contain the keyword or tag.
yes, just include the token in your tweet text. The form instructions should list it

Code: Select all

#trend_hash#
scrape tweet that contain the keyword/tag and look for links to webpages, then find the tweet function on the webpage and send out a tweet?[/list]
So you want to scrape tweets that contain the #trend_hash#
then scrape links contained (if contained) in that tweet
then??? I get confused here, you wrote " then find the tweet function on the webpage"

So go to the link contained in the tweet and find the tweet function on that webpage? That does not make sense (to me anyway)

Eitherway, no the program cannot extract URLs from tweets and perform anything based on that webpage, so I guess whatever it is you want to do the program cannot do it
lsberic
Posts: 75
Joined: Mon Aug 21, 2017 7:19 pm

Re: Can twitterdub track trending topics, then scrape and retweet?

Post by lsberic »

martin@rootjazz wrote: Wed Nov 20, 2019 1:43 pm So you want to scrape tweets that contain the #trend_hash#
then scrape links contained (if contained) in that tweet
then??? I get confused here, you wrote " then find the tweet function on the webpage"

So go to the link contained in the tweet and find the tweet function on that webpage? That does not make sense (to me anyway)
Hey Martin,

Thanks for indulging me in this thread.

So, what I was referring to is a function similar to scrapebox which has a social plugin. With that plugin, scrapebox can look at a set of links and find the social urls on a given webpage.

That function is displayed here. https://youtu.be/S8RxlzdBAjU
martin@rootjazz wrote: Wed Nov 20, 2019 1:43 pm
Eitherway, no the program cannot extract URLs from tweets and perform anything based on that webpage, so I guess whatever it is you want to do the program cannot do it
I'm not sure why a program could not scrape the links in a tweet, but I don't know a lot about how twitter links are formulated on the backend.

But just so I am clear, using the twitter trends tag, I can automate twitterdub to monitor when keyword is trending and set it to tweet a new message or like or retweet an existing twitter post automatically?
User avatar
martin@rootjazz
Site Admin
Posts: 34388
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Can twitterdub track trending topics, then scrape and retweet?

Post by martin@rootjazz »

lsberic wrote: Wed Nov 20, 2019 5:11 pm
martin@rootjazz wrote: Wed Nov 20, 2019 1:43 pm So you want to scrape tweets that contain the #trend_hash#
then scrape links contained (if contained) in that tweet
then??? I get confused here, you wrote " then find the tweet function on the webpage"

So go to the link contained in the tweet and find the tweet function on that webpage? That does not make sense (to me anyway)
Hey Martin,

Thanks for indulging me in this thread.

So, what I was referring to is a function similar to scrapebox which has a social plugin. With that plugin, scrapebox can look at a set of links and find the social urls on a given webpage.

That function is displayed here. https://youtu.be/S8RxlzdBAjU
martin@rootjazz wrote: Wed Nov 20, 2019 1:43 pm
Eitherway, no the program cannot extract URLs from tweets and perform anything based on that webpage, so I guess whatever it is you want to do the program cannot do it
I'm not sure why a program could not scrape the links in a tweet, but I don't know a lot about how twitter links are formulated on the backend.
I meant, it could if i coded it to do it. But I haven't so it cannot. Not that it is impossible.

But I don't see the point, why do you want scrape social URLs contained in a webpage that was containined in a tweet.
But just so I am clear, using the twitter trends tag, I can automate twitterdub to monitor when keyword is trending and set it to tweet a new message or like or retweet an existing twitter post automatically?
yes you can use the token #trend_hash# as input to your searches.

So you can scrape recent tweets containing #trend_hash# and like them
lsberic
Posts: 75
Joined: Mon Aug 21, 2017 7:19 pm

Re: Can twitterdub track trending topics, then scrape and retweet?

Post by lsberic »

martin@rootjazz wrote: Thu Nov 21, 2019 12:28 am
I meant, it could if i coded it to do it. But I haven't so it cannot. Not that it is impossible.

But I don't see the point, why do you want scrape social URLs contained in a webpage that was containined in a tweet.

Hi Martin,

Thanks for the clarification.

But the main thing is to be able grab trends sorted by keywords and then being able to retweet useful information in an automated fashion. If twitterdub cannot scrape web pages linked in that tweet, that's fine.

I was hoping there was a function like scrapebox social plugin so that if I had that additional data I could make use of it.
User avatar
martin@rootjazz
Site Admin
Posts: 34388
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Can twitterdub track trending topics, then scrape and retweet?

Post by martin@rootjazz »

Sorry for not understanding, but I don't understand what you are trying to do
But the main thing is to be able grab trends sorted by keywords and then being able to retweet useful information in an automated fashion.
So you want to
1) get a list of trending topics
2) manually create tweets for each trending hashtag
3) schedule those curated tweets to post out that day?
If twitterdub cannot scrape web pages linked in that tweet, that's fine.

I was hoping there was a function like scrapebox social plugin so that if I had that additional data I could make use of it.
Again, maybe I just don't understand you correctly. But I cannot see any benefit to
a) scraping tweets
b) checking for web links
c) scraping weblinks and looking for links back to social sites.

I cannot think of any use that would be. If I am going to add a feature, I need to understand it and agree that it will be beneficial to multiple users or can be promoted as a unique feature that could drum up more sales.
Post Reply