Page 1 of 2

Scrape only users with their DM open?

Posted: Mon Jun 21, 2021 12:32 pm
by eddielp123
Hey, when performing a filtered user search, whether it is purely for scraping purposes or to follow these users, is there any filter that allows me to only scrape "messageable" users and filter out anybody who isn't? Meaning that they have their DMs open and I will be able to message them even if they didn't follow back?

Re: Scrape only users with their DM open?

Posted: Mon Jun 21, 2021 1:11 pm
by eddielp123
I also just saw in the changelog
3.226 13/05/2021
DM action will test for userdetails endpoint to confirm profile can receive DM before attempting
so it seems possible that there is a "can dm" filter I'm just wondering if its possible to be able to include that as a filter option in the User Filter settings for scraping, following, DMing etc.

Re: Scrape only users with their DM open?

Posted: Mon Jun 21, 2021 2:25 pm
by martin@rootjazz
eddielp123 wrote: Mon Jun 21, 2021 1:11 pm I also just saw in the changelog
3.226 13/05/2021
DM action will test for userdetails endpoint to confirm profile can receive DM before attempting
so it seems possible that there is a "can dm" filter I'm just wondering if its possible to be able to include that as a filter option in the User Filter settings for scraping, following, DMing etc.
Makes sense and is a two minute update. I'll add it for next update and let you know when ready



Regards,
Martin

Re: Scrape only users with their DM open?

Posted: Mon Jun 21, 2021 4:30 pm
by eddielp123
martin@rootjazz wrote: Mon Jun 21, 2021 2:25 pm
eddielp123 wrote: Mon Jun 21, 2021 1:11 pm I also just saw in the changelog
3.226 13/05/2021
DM action will test for userdetails endpoint to confirm profile can receive DM before attempting
so it seems possible that there is a "can dm" filter I'm just wondering if its possible to be able to include that as a filter option in the User Filter settings for scraping, following, DMing etc.
Makes sense and is a two minute update. I'll add it for next update and let you know when ready



Regards,
Martin
amazing!!

Re: Scrape only users with their DM open?

Posted: Mon Jun 21, 2021 5:58 pm
by martin@rootjazz

Re: Scrape only users with their DM open?

Posted: Mon Jun 21, 2021 8:41 pm
by eddielp123
just tried it out, it doesnt seem to work, it will just skip/ignore users even if they're dm-able.

i submitted logs with a test run - all people that are being scraped here are 100% dm-able (and of course in the setting i had the "Can DM" enabled)
but none of them are being scraped and instead itll say * IGNORE: not allowed tro DM

log id 53886

Re: Scrape only users with their DM open?

Posted: Tue Jun 22, 2021 2:33 pm
by martin@rootjazz
let me test, code looks correct, maybe the property is not being set by twitter...

Re: Scrape only users with their DM open?

Posted: Tue Jun 22, 2021 3:17 pm
by eddielp123
martin@rootjazz wrote: Tue Jun 22, 2021 2:33 pm let me test, code looks correct, maybe the property is not being set by twitter...
I tried it out in the follow tool, scrape tool, dm tool, always the same issue unfortunately :(

Re: Scrape only users with their DM open?

Posted: Tue Jun 22, 2021 5:38 pm
by martin@rootjazz
ok, seems like the CAN_DM option only appears on a full user_details endpoint scrape, the program was scraping a less user object and not detecting the property wasn't set.

The next update will fix this. I shall let you know when it is ready.



Regards,
Martin

Re: Scrape only users with their DM open?

Posted: Tue Jun 22, 2021 7:39 pm
by eddielp123
martin@rootjazz wrote: Tue Jun 22, 2021 5:38 pm ok, seems like the CAN_DM option only appears on a full user_details endpoint scrape, the program was scraping a less user object and not detecting the property wasn't set.

The next update will fix this. I shall let you know when it is ready.



Regards,
Martin
perfect, thank you so much.