Filter out people in a specific list from scrape action

Support / help / discussion forum for twitter bot
Post Reply
eddielp123
Posts: 510
Joined: Tue Feb 11, 2020 2:53 pm

Filter out people in a specific list from scrape action

Post by eddielp123 »

Hey, so I'm doing scrape actions across multiple accounts with different search settings, and while I do have the

"Global unique results for search regardless of account / filters used"

checked, it will still give duplicate results across different search terms / settings.

The only filter I have enabled is "Can DM".

Now my issue is that each of the accounts has a follow limit of 5000. So once that limit is reached i will have to unfollow everybody and start over again, and I'm afraid that I will scrape many people that I have scraped already.

My idea was to put all of them in a list, so that I can add the filter "Exclude if user is in list" to make sure they don't get scraped, as I assume there is no Global Unique Results across different search terms in the scraping tool.

I hope this is somewhat easily understandable. Feel free to let me know if you want me to explain a little better
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Filter out people in a specific list from scrape action

Post by martin@rootjazz »

eddielp123 wrote: Fri Jul 23, 2021 6:32 pm Hey, so I'm doing scrape actions across multiple accounts with different search settings, and while I do have the

"Global unique results for search regardless of account / filters used"

checked, it will still give duplicate results across different search terms / settings.
Yes, the unique options is per search

so profile_a_followers can return the same results as profile_b_followings. But if you run profile_a_followers search multiple times you won't get the same results


as I assume there is no Global Unique Results across different search terms in the scraping tool.
That's correct. What you want to do, isn't really possible at this time. I suppose you could "try" the IGNORE words filter and enter the username of each previous result BUT you will need to use the username token:
username:<input>

As the ignore words function is only for BIO at this time (next update will allow for bio + username however).

But again, I am not sure how this is going to work if you try and paste thousands and thousands of results in.

So it isn't really possible in all honesty
eddielp123
Posts: 510
Joined: Tue Feb 11, 2020 2:53 pm

Re: Filter out people in a specific list from scrape action

Post by eddielp123 »

martin@rootjazz wrote: Fri Jul 23, 2021 9:37 pm
Yes, the unique options is per search

so profile_a_followers can return the same results as profile_b_followings. But if you run profile_a_followers search multiple times you won't get the same results
Yup exactly and that's the issue because I mostly have to scrape multiple different filters to get to the amount of results that i desire :(

martin@rootjazz wrote: Fri Jul 23, 2021 9:37 pm What you want to do, isn't really possible at this time. I suppose you could "try" the IGNORE words filter and enter the username of each previous result BUT you will need to use the username token:
username:<input>

As the ignore words function is only for BIO at this time (next update will allow for bio + username however).

But again, I am not sure how this is going to work if you try and paste thousands and thousands of results in.

So it isn't really possible in all honesty
Well it would definitely be a start and I could definitely test it out. But then again I don't know if it would be an API call each time that it checks for the ignore usernames or if it would be a bot internal process like in the other "IGNORE" processes.

Is it in no way possible to do something such as the "global processed logs" that work in other tools such as the FOLLOWING tool but for the SCRAPING tool?

Thanks :)
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Filter out people in a specific list from scrape action

Post by martin@rootjazz »

eddielp123 wrote: Fri Jul 23, 2021 10:40 pm
Is it in no way possible to do something such as the "global processed logs" that work in other tools such as the FOLLOWING tool but for the SCRAPING tool?

Thanks :)
Your suggestion has been noted and added to the feature suggestions list. Depending on the demand for the change / feature will alter how quickly it gets implemented.
Post Reply