Scrape only users with their DM open?

Support / help / discussion forum for twitter bot
eddielp123
Posts: 510
Joined: Tue Feb 11, 2020 2:53 pm

Scrape only users with their DM open?

Post by eddielp123 »

Hey, when performing a filtered user search, whether it is purely for scraping purposes or to follow these users, is there any filter that allows me to only scrape "messageable" users and filter out anybody who isn't? Meaning that they have their DMs open and I will be able to message them even if they didn't follow back?
eddielp123
Posts: 510
Joined: Tue Feb 11, 2020 2:53 pm

Re: Scrape only users with their DM open?

Post by eddielp123 »

I also just saw in the changelog
3.226 13/05/2021
DM action will test for userdetails endpoint to confirm profile can receive DM before attempting
so it seems possible that there is a "can dm" filter I'm just wondering if its possible to be able to include that as a filter option in the User Filter settings for scraping, following, DMing etc.
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape only users with their DM open?

Post by martin@rootjazz »

eddielp123 wrote: Mon Jun 21, 2021 1:11 pm I also just saw in the changelog
3.226 13/05/2021
DM action will test for userdetails endpoint to confirm profile can receive DM before attempting
so it seems possible that there is a "can dm" filter I'm just wondering if its possible to be able to include that as a filter option in the User Filter settings for scraping, following, DMing etc.
Makes sense and is a two minute update. I'll add it for next update and let you know when ready



Regards,
Martin
eddielp123
Posts: 510
Joined: Tue Feb 11, 2020 2:53 pm

Re: Scrape only users with their DM open?

Post by eddielp123 »

martin@rootjazz wrote: Mon Jun 21, 2021 2:25 pm
eddielp123 wrote: Mon Jun 21, 2021 1:11 pm I also just saw in the changelog
3.226 13/05/2021
DM action will test for userdetails endpoint to confirm profile can receive DM before attempting
so it seems possible that there is a "can dm" filter I'm just wondering if its possible to be able to include that as a filter option in the User Filter settings for scraping, following, DMing etc.
Makes sense and is a two minute update. I'll add it for next update and let you know when ready



Regards,
Martin
amazing!!
eddielp123
Posts: 510
Joined: Tue Feb 11, 2020 2:53 pm

Re: Scrape only users with their DM open?

Post by eddielp123 »

just tried it out, it doesnt seem to work, it will just skip/ignore users even if they're dm-able.

i submitted logs with a test run - all people that are being scraped here are 100% dm-able (and of course in the setting i had the "Can DM" enabled)
but none of them are being scraped and instead itll say * IGNORE: not allowed tro DM

log id 53886
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape only users with their DM open?

Post by martin@rootjazz »

let me test, code looks correct, maybe the property is not being set by twitter...
eddielp123
Posts: 510
Joined: Tue Feb 11, 2020 2:53 pm

Re: Scrape only users with their DM open?

Post by eddielp123 »

martin@rootjazz wrote: Tue Jun 22, 2021 2:33 pm let me test, code looks correct, maybe the property is not being set by twitter...
I tried it out in the follow tool, scrape tool, dm tool, always the same issue unfortunately :(
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape only users with their DM open?

Post by martin@rootjazz »

ok, seems like the CAN_DM option only appears on a full user_details endpoint scrape, the program was scraping a less user object and not detecting the property wasn't set.

The next update will fix this. I shall let you know when it is ready.



Regards,
Martin
eddielp123
Posts: 510
Joined: Tue Feb 11, 2020 2:53 pm

Re: Scrape only users with their DM open?

Post by eddielp123 »

martin@rootjazz wrote: Tue Jun 22, 2021 5:38 pm ok, seems like the CAN_DM option only appears on a full user_details endpoint scrape, the program was scraping a less user object and not detecting the property wasn't set.

The next update will fix this. I shall let you know when it is ready.



Regards,
Martin
perfect, thank you so much.
Post Reply