Filter Results

Support / help / discussion forum for twitter bot
shaner4042
Posts: 86
Joined: Sun Sep 04, 2016 5:25 pm

Filter Results

Post by shaner4042 »

Hey,

Just wondering if in the scraper tab it would be possible to add some sort of feature like, "filter URL results", where we could link to a file path of previously scraped URL's, and filter those results further.

I understand that there is already a user filter option, but I find that many searches with filters are too exhaustive for any one account and end up exceeding the limitations before a significant amount of results are achieved. I was wondering if this could be avoided if we could use multiple accounts to carry out different steps of the search and filter process. For example, say I scrape the followers of an account -- I get 5000 saved URL's. Then I could switch the connection account and scrape that file path of URL's for users who were active within the last 30 days. And so on and so on. Hope you understand what I'm trying to get at.
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Filter Results

Post by martin@rootjazz »

so to confirm I understand.

You want to be able to specify a file of profiles / tweets.
Then filter those using multiple (your) accounts to avoid limitations.


Is that correct?
shaner4042
Posts: 86
Joined: Sun Sep 04, 2016 5:25 pm

Re: Filter Results

Post by shaner4042 »

Yes, that basically sums it up.

I don't necessarily need to be able to set this up all in one go; If that makes it easier on your end. It would still possible to carry this out by doing an initial search/filter, then switching the connection account, specify the previously scraped file path and set new filter parameters --switch the connection account again, set new filter parameters again -- so on and so on.

I think you understand what I am trying to say. If you're willing to add this feature at some point, I'm sure you know the best way to go about it :)

Thanks for all your help Martin.
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Filter Results

Post by martin@rootjazz »

should be straight forward to add this to the scrape tab. Remind me end of the week, about Thursday
shaner4042
Posts: 86
Joined: Sun Sep 04, 2016 5:25 pm

Re: Filter Results

Post by shaner4042 »

martin@rootjazz wrote:should be straight forward to add this to the scrape tab. Remind me end of the week, about Thursday
Just giving you that reminder. 8-) Let me know if you'd like another one later down the line if you're still too busy
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Filter Results

Post by martin@rootjazz »

I should really stop asking for reminders, hoping that day will never come :( lol

Will take a look today
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Filter Results

Post by martin@rootjazz »

shaner4042
Posts: 86
Joined: Sun Sep 04, 2016 5:25 pm

Re: Filter Results

Post by shaner4042 »

Awesome Martin! Love the support.
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Filter Results

Post by martin@rootjazz »

:)
shaner4042
Posts: 86
Joined: Sun Sep 04, 2016 5:25 pm

Re: Filter Results

Post by shaner4042 »

Thanks again for adding the feature. However I am not sure it's working properly. I'll run you through what I've done.

I scraped followers of Twitch's Twitter page and got 5000 saved profile URL's. I then specified this list in the filter list feature, filtering profiles that contain 'youtube' keyword in their bio, allowing for partial match. The action completed 5 hours later, only turning up with one profile URL. I know from looking manually through the list that there are dozens of profiles that contain YouTube in their bio.

I also noticed the action was doing something a little strange as it was processing. It seems every time it found a profile that matched the filter, it would overwrite the previously found profile in the .txt file in saved data. Meaning, if I periodically closed and opened the .txt file as the filter set action was processing, every time the saved twitter URL would be different, as if it was constantly overwriting itself. I figured this may be why it only turned up with one result in the end. The processor was also constantly throwing this error message: * ERROR: handleprofile: Source array was not long enough. Check srcIndex and length, and the array's lower bounds.

I've submitted logs: 84626

Thanks!
Post Reply