Hi martin,
I'm using TD to scrape views for tweets for hundred of accounts, it takes time to complete while there is no option for threads there, can you make one for us?
thnk...
threads for scraping tweets
- martin@rootjazz
- Site Admin
- Posts: 35035
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: threads for scraping tweets
Searches need to proceed sequential as the next page relies on the ID contained in the previous page.
But if you can describe your search exactly there might be some ways to speed it up
But if you can describe your search exactly there might be some ways to speed it up
Re: threads for scraping tweets
first: A user id to url.martin@rootjazz wrote: ↑Sun May 05, 2024 9:41 pm Searches need to proceed sequential as the next page relies on the ID contained in the previous page.
But if you can describe your search exactly there might be some ways to speed it up
second: user tweets.
I just want to scrap views of the tweets of my accounts.
thanks.
- martin@rootjazz
- Site Admin
- Posts: 35035
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: threads for scraping tweets
No way to thread thisjoj.j2111 wrote: ↑Thu May 09, 2024 10:55 pmfirst: A user id to url.martin@rootjazz wrote: ↑Sun May 05, 2024 9:41 pm Searches need to proceed sequential as the next page relies on the ID contained in the previous page.
But if you can describe your search exactly there might be some ways to speed it up
second: user tweets.
I just want to scrap views of the tweets of my accounts.
thanks.
Program scrapes the user_tweets end points, then gets the details for each results (which should be cached and not require an additional call
The next page of results ID comes from the previous page. So it has to be sequential.
Best you can is bulk create searches, one per account to scrape. Do this by entering a file path for your search input, where each line is a new profile to scrape.
change your search:
USER_TWEETS
do not use USER ID URL as this will load the file path in.
Select the accounts to perform the searches.
Create the action. You will create 1 action per line in your file. These can then be threaded on the processing tab
I'd recommend you limit your results to 100 or 500 or whatever per search. I guess you don't need the views on older tweets? So setting a limit to ensure you only scrape the data you really want, rather than scraping it needlessly
Regards,
Martin
Re: threads for scraping tweets
it works perfect, thanks,
but is there is a way to merge all result in one file?
but is there is a way to merge all result in one file?
Re: threads for scraping tweets
I found a way to do that using MS Excel:
https://www.myonlinetraininghub.com/pow ... m-a-folder
Thank you Martin
https://www.myonlinetraininghub.com/pow ... m-a-folder
Thank you Martin
- martin@rootjazz
- Site Admin
- Posts: 35035
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: threads for scraping tweets
You will then need to merge the resulting files: http://stackoverflow.com/questions/6764 ... o-one-file
Code: Select all
Load Dos / CMD
CD c:\path\to\
copy *.txt merged.txt
- martin@rootjazz
- Site Admin
- Posts: 35035
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: threads for scraping tweets
joj.j2111 wrote: ↑Fri May 10, 2024 9:57 pm I found a way to do that using MS Excel:
https://www.myonlinetraininghub.com/pow ... m-a-folder