threads for scraping tweets

[joj.j2111]

Hi martin,

I'm using TD to scrape views for tweets for hundred of accounts, it takes time to complete while there is no option for threads there, can you make one for us?

thnk...

[martin@rootjazz]

Searches need to proceed sequential as the next page relies on the ID contained in the previous page.

But if you can describe your search exactly there might be some ways to speed it up

[joj.j2111]

Searches need to proceed sequential as the next page relies on the ID contained in the previous page.

But if you can describe your search exactly there might be some ways to speed it up

first: A user id to url.
second: user tweets.

I just want to scrap views of the tweets of my accounts.

thanks.

[martin@rootjazz]

Searches need to proceed sequential as the next page relies on the ID contained in the previous page.

But if you can describe your search exactly there might be some ways to speed it up

first: A user id to url.
second: user tweets.

I just want to scrap views of the tweets of my accounts.

thanks.

No way to thread this

Program scrapes the user_tweets end points, then gets the details for each results (which should be cached and not require an additional call

The next page of results ID comes from the previous page. So it has to be sequential.


Best you can is bulk create searches, one per account to scrape. Do this by entering a file path for your search input, where each line is a new profile to scrape.

change your search:

USER_TWEETS

do not use USER ID URL as this will load the file path in.

Select the accounts to perform the searches.
Create the action. You will create 1 action per line in your file. These can then be threaded on the processing tab

I'd recommend you limit your results to 100 or 500 or whatever per search. I guess you don't need the views on older tweets? So setting a limit to ensure you only scrape the data you really want, rather than scraping it needlessly




Regards,
Martin

[joj.j2111]

it works perfect, thanks,

but is there is a way to merge all result in one file?

[joj.j2111]

I found a way to do that using MS Excel:

https://www.myonlinetraininghub.com/pow ... m-a-folder

Thank you Martin

it works perfect, thanks,

but is there is a way to merge all result in one file?

[martin@rootjazz]

it works perfect, thanks,

but is there is a way to merge all result in one file?

You will then need to merge the resulting files: http://stackoverflow.com/questions/6764 ... o-one-file

Code: Select all

Load Dos / CMD
CD c:\path\to\
copy *.txt merged.txt

You would then want to remove duplicate items from the file, for this I would recommend notepad++ with the TextFX extension

[martin@rootjazz]

I found a way to do that using MS Excel:

https://www.myonlinetraininghub.com/pow ... m-a-folder