Scraping Profile Under Twitter Search (people) and (recent)

Support / help / discussion forum for twitter bot
papamax
Posts: 55
Joined: Sat Nov 26, 2022 6:20 pm

Scraping Profile Under Twitter Search (people) and (recent)

Post by papamax »

I have been trying without success to scrape profiles from the People search on twitter like below:

Image

Isn't there a way to do that? I can't find any suitable option on Custom Search, nothing that will give a Profile URL as direct output for a search.

----

Additionally, a second issue is when I try to scrape profiles using tweet search(recent / video / img) using custom search to output creator of Tweet like below:

Image

Here is what I have on the scraper page, note that there is no user filter and the account performing the search is open for sensitive content and even posts it.

Image

It gives pretty much no results back when there is literally hundreds available:

Image

I have seen this same issue multiple times before as well when trying to scrape tweets from some user... When they have 1000's of tweets in the account and the search also shows endless numbers if I scroll down the scraper give me back 40-50 items when I am requesting 500 for example.

Needless to say I believe a scraper tutorial is in order, tool has been in the market for a long time and there isn't a single one. I believe that would help a lot of users and decrease the questions you answer here.

Thanks
User avatar
martin@rootjazz
Site Admin
Posts: 34038
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping Profile Under Twitter Search (people) and (recent)

Post by martin@rootjazz »

I don't think PEOPLE is supported actually

what you are doing is scraping the RECENT tab, and getting the profiles that posted the tweets


As for why it isn't working, don't know will need to test. I won't lie, I am massively backlogged at the moment due to kids on school holidays and I'm on my own.


You can look at the ADVANCED SEARCH help page:
viewtopic.php?f=30&t=6985&p=66649#p66649

and try and figure out why it isn't working yourself, or wait for me to get round to it (will be a few days, sorry)
papamax
Posts: 55
Joined: Sat Nov 26, 2022 6:20 pm

Re: Scraping Profile Under Twitter Search (people) and (recent)

Post by papamax »

Any updates on the issue of scraping profiles using tweet search(recent) using custom search to output creator of Tweet like explained above?

I have tried figuring it out and testing a lot of different settings, it is always the same issue... I think it has to do with NSFW account. It generally works with regular accounts.

Thanks
User avatar
martin@rootjazz
Site Admin
Posts: 34038
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping Profile Under Twitter Search (people) and (recent)

Post by martin@rootjazz »

papamax wrote: Tue Mar 14, 2023 7:46 am Any updates on the issue of scraping profiles using tweet search(recent) using custom search to output creator of Tweet like explained above?
working fine here
Start search: Tweet Search (recent) with: fitness min_faves:50 -filter:replies(100) using: <user>
Search: fitness min_faves:50 -filter:replies with: <user>: max100 return: RETURN_URLS
tweets: 24
Search pages: 1
Paused for 4secs
Results: 24
tweets: 23
Search pages: 2
Paused for 4secs
Results: 47
tweets: 24
Search pages: 3
Paused for 5secs
Results: 71

STEPS
tweet search (recent)
creator

input: fitness min_faves:50 -filter:replies

I guess it still doesn't work for you? try a different account maybe? What happens then?
papamax
Posts: 55
Joined: Sat Nov 26, 2022 6:20 pm

Re: Scraping Profile Under Twitter Search (people) and (recent)

Post by papamax »

I figured something out here...

Most of my issues were using searches related to NSFW terms.

Here is what fixed is, you may want to add it to the software if the user selects sensitive content on filter:

Image

then

Image

and

Image

The HIDE SENSITIVE tick box on Search Settings MUST be unchecked for it to work... And it is checked by default! Even if the account is NSFW set and even if the time line is enabled to Sensitive content... Search settings are a separate setting.

Thanks
User avatar
martin@rootjazz
Site Admin
Posts: 34038
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping Profile Under Twitter Search (people) and (recent)

Post by martin@rootjazz »

thanks for the info. I thought the site settings NSFW changed that... Let me check on that and see why it isn't
tonyton99
Posts: 6
Joined: Sat Nov 18, 2023 11:56 am

Re: Scraping Profile Under Twitter Search (people) and (recent)

Post by tonyton99 »

How can make it automatically for bulk accounts by (Twitter dub - Bot) pls ?
How can i make all accounts IN Search section
"changes to buttons OFF (hide sensitive) ?
There is a place to click in bot for filtre that section in The BOT ?

(Bcs i looking about it in tutorial i can't find)
User avatar
martin@rootjazz
Site Admin
Posts: 34038
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping Profile Under Twitter Search (people) and (recent)

Post by martin@rootjazz »

tonyton99 wrote: Thu Nov 30, 2023 9:22 pm How can make it automatically for bulk accounts by (Twitter dub - Bot) pls ?
Sorry I do not understand your question can you explain please
How can i make all accounts IN Search section
On the SEARCH tab in the ACCOUNTS SELECTION form, bulk select accounts, by using shift / cntrl keys for normal multiple selection
"changes to buttons OFF (hide sensitive) ?
UPDATE ACCOUNTS tab

There is a place to click in bot for filtre that section in The BOT ?
Again, sorry I don't understand the question can you explain please
tonyton99
Posts: 6
Joined: Sat Nov 18, 2023 11:56 am

Re: Scraping Profile Under Twitter Search (people) and (recent)

Post by tonyton99 »

i mean I'm interested in (nsfw post) and in twitter can't see that (nsfw posts) "Only" if you make FILTRE by (search) settings, for me i do it manually and that take time and bored when i do it one by one , so i want know if there is an option how can i select all accounts i login in twitterdub and make all filter accounts in (Hide sensitive) as pictures below..

Image

Image

Image
User avatar
martin@rootjazz
Site Admin
Posts: 34038
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping Profile Under Twitter Search (people) and (recent)

Post by martin@rootjazz »

ok seems Twitter have changed how this work. Will get it updated
Post Reply