Scrape Help

Support / help / discussion forum for twitter bot
Post Reply
shaily
Posts: 95
Joined: Mon Aug 12, 2019 2:30 am

Scrape Help

Post by shaily »

Hi Martin,
I want to get help on two areas of scraping:

1. How to get tweets of an individual between a particular date range? I have gone to advanced twitter search option and created a query:
URL: https://twitter.com/search?q=(from%3Aso ... 2020-12-01
Search String: (from:soundcloud) until:2020-12-31 since:2020-12-01

and then I went to Twitterdub Scrape Tweet section and tried to paste both the url and the search string but it wanted me to select custom search button and provide a search option. Then I picked Tweet URL but even then it didn't work.

Can you please advise two things. To run this query:
a. Do I put URL or Search String in the tweet scrape option?
b. What should be the custom search option?

2. The first line on the scrape tab is: Scrape Profiles containing Emails. How does this work?? Do we need to put in Profile URL / Profile URLs in a file? If yes, how do we find the profiles with email address? Can you please help us on how progress that functionality.


Really appreciate your help.

Cheers,
Shaily
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Help

Post by martin@rootjazz »

User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Help

Post by martin@rootjazz »

Can you please advise two things. To run this query:
a. Do I put URL or Search String in the tweet scrape option?
Just the search params explained in the link above
b. What should be the custom search option?
TWEET SEARCH
shaily
Posts: 95
Joined: Mon Aug 12, 2019 2:30 am

Re: Scrape Help

Post by shaily »

Hi Martin,
I did exactly that... and that's what I explained above... and it didn't work.
Cheers,
Shaily
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Help

Post by martin@rootjazz »

Logs showing attempt please

HELP > LOGS > SUBMIT

then send your logs ID - the numbers are sufficient (displayed after successful uploading of logs)
cyclop
Posts: 15
Joined: Thu May 12, 2022 11:48 am

Re: Scrape Help

Post by cyclop »

martin@rootjazz wrote: Mon Feb 22, 2021 10:03 pm Logs showing attempt please

HELP > LOGS > SUBMIT

then send your logs ID - the numbers are sufficient (displayed after successful uploading of logs)
Hi Martin! Is there a tutorial how to use Scraper function? I can't find anything except the link that you posted above..
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Help

Post by martin@rootjazz »

cyclop wrote: Fri May 13, 2022 1:11 pm
martin@rootjazz wrote: Mon Feb 22, 2021 10:03 pm Logs showing attempt please

HELP > LOGS > SUBMIT

then send your logs ID - the numbers are sufficient (displayed after successful uploading of logs)
Hi Martin! Is there a tutorial how to use Scraper function? I can't find anything except the link that you posted above..

Twitterdub custom search tutorial
- Explains how to build a search and specify how many results per step (optional)

https://www.youtube.com/watch?v=D1W2KnNciPI



Getting Started with Twitterdub (new for 2022)
https://www.youtube.com/watch?v=rVfGB1_ ... swlBFQBn8J

All new tutorials: (new for 2022)
https://www.youtube.com/watch?v=rVfGB1_ ... rQItYJO4bU




I do still have a list of tutorials to make, so can add any new tutorials you want to the list. Until then, explain what you are trying to do, what you don't understand, what is confusing and I can do my best to explain



Regards,
Martin
cyclop
Posts: 15
Joined: Thu May 12, 2022 11:48 am

Re: Scrape Help

Post by cyclop »

martin@rootjazz wrote: Mon Feb 22, 2021 10:03 pm I do still have a list of tutorials to make, so can add any new tutorials you want to the list. Until then, explain what you are trying to do, what you don't understand, what is confusing and I can do my best to explain
Regards,
Martin
Thanks! I have looked those but haven't understood it completely yet.. Sorry if these questions are silly.

1) If i have a list of tweets urls. How can i scrape their bodies? (I always got url in the scrape file.. Trying it in "Scrape Tweets" fields)

2) How can I scrape last 10 tweet bodies of a profile?
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape Help

Post by martin@rootjazz »

cyclop wrote: Fri May 13, 2022 4:52 pm
martin@rootjazz wrote: Mon Feb 22, 2021 10:03 pm I do still have a list of tutorials to make, so can add any new tutorials you want to the list. Until then, explain what you are trying to do, what you don't understand, what is confusing and I can do my best to explain
Regards,
Martin
Thanks! I have looked those but haven't understood it completely yet.. Sorry if these questions are silly.

1) If i have a list of tweets urls. How can i scrape their bodies? (I always got url in the scrape file.. Trying it in "Scrape Tweets" fields)
SCRAPE TAB
SCRAPE TWEETS
check DETAILS, it will get all the details in a CSV, one of those will be the tweet body

With your list of tweet URLS in a file, one per line

CUSTOM SEARCH
input: your_file.txt
STEPS:
ID URL OF TWEET (this also takes a file)

2) How can I scrape last 10 tweet bodies of a profile?
Assuming you have a list of profiles in a file, one per line

STEPS:
ID URL PROFILE
USER TWEETS (10)

Then same as above. SCRAPE TAB > SCRAPE TWEETS > OUTPUT: DETAILS



Regards,
Martin
Post Reply