New user need helps

Support / help / discussion forum for twitter bot
Danny000
Posts: 106
Joined: Mon Jan 15, 2018 12:19 am

New user need helps

Post by Danny000 » Sat Aug 11, 2018 2:51 am

Hello there,

I am an old user of TumblingJazz, and became a user of Twitterdub 2 months later. But I couldn't use TD as good as TJ due to too many problems began with API issue, and I couldn't find the answer I need from tutorial videos. so I stopped using it. Now the non-API version launch, maybe it's time for me to start over again, yet I still need help, from the basic use of TD.

Basically I need to find niche users and then follow them. I tried it myself many times and still couldn't find the right way. So would you please describe it step by step, the more detailed the better:

For example, if I am doing dating niche, need to search niche users who like sexy women, the keyword is "sexy woman", how do I perform this?

I understand this is really simple for you guys, and thanks for any help.

Danny

User avatar
martin@rootjazz
Site Admin
Posts: 17380
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: New user need helps

Post by martin@rootjazz » Sat Aug 11, 2018 3:45 pm

Danny000 wrote:
Sat Aug 11, 2018 2:51 am


Basically I need to find niche users and then follow them. I tried it myself many times and still couldn't find the right way. So would you please describe it step by step, the more detailed the better:
Can you explain what you tried and why it didn't work.

For example, if I am doing dating niche, need to search niche users who like sexy women, the keyword is "sexy woman", how do I perform this?
TWEET SEARCH. That will find users who posted a tweet with those keywords. You can do RECENT or POPULAR searches of this (same as TUMJAZZ)

You can also scrape NOTES (same as tumjazz) via custom search (tweet interactions it is called)



Same as TumJazz you can scrape followers of a particular account, you can scrape tweets of a similar niche account then scrape those you intereact with their media.

Basically if you can think of a way to find users, you can do it with CUSTOM SEARCH (which is a lot more powerful in TD than Tumjazz). IF you can explain your searches in words, I can explain how to do it in the app

Danny000
Posts: 106
Joined: Mon Jan 15, 2018 12:19 am

Re: New user need helps

Post by Danny000 » Wed Aug 15, 2018 8:16 am

Hi Martin

I finally succeeded to finish some scraping job, with simplest way - scrape followers. But the result is not good enough. I search on a specify user's followers which has about 50k, TD only scraped 5-10k and then stopped. Here I tried to do it again to submit some log, and got 0 results:

Log submitted: 35823
Next cursor: 1608821652557571885
FAILED GET: (163479) 502
http://51.15.222.94:22694/
* ERROR: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercomcloudxxx_2018-08-15.txt
Started: 15/08/2018 00:10 AM
Finished: 15/08/2018 00:15 AM
ID: 40a20631-827c-4479-995c-f920ce16c3bd
Action ran for: 0hr:5min:19s
Here are some questions:
1. How to scrape more from 10k followers from one account?
In this way, no matter how many times I tried, the best is 10k, the worst is 0.

2. As I browse manually, most twitter users account got no more then 10k followers. How to scrapes 100s of k niche users as I did in Tumblr? It's easy to do it in Tumblr because there are lots popular post got more than 100k notes. What's the right way to do it in twitter?

thanks
Danny
Last edited by Danny000 on Wed Aug 15, 2018 1:24 pm, edited 1 time in total.

User avatar
martin@rootjazz
Site Admin
Posts: 17380
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: New user need helps

Post by martin@rootjazz » Wed Aug 15, 2018 12:15 pm

Danny000 wrote:
Wed Aug 15, 2018 8:16 am
Hi Martin

I finally succeeded to finish some scraping job, with simplest way - scrape followers. But the result is not good enough. I search on a specify user's followers which has about 50k, TD only scraped 5-10k and then stopped. Here I tried to do it again to submit some log, and got 0 results:

Log submitted: 35823
Next cursor: 1608821652557571885
FAILED GET: (163479) 502
http://51.15.222.94:22694/
* ERROR: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercomcloudxxx_2018-08-15.txt
Started: 15/08/2018 00:10 AM
Finished: 15/08/2018 00:15 AM
ID: 40a20631-827c-4479-995c-f920ce16c3bd
Action ran for: 0hr:5min:19s
Why are you trying to scrape more than 50k results? Do you need that many? (possibly yes, but possibly no as well).

After your scrape, you got a 502 error, which is a server error. Not too sure why, am downloading your logs to look. But the next time you got 0, sounds like a server block for over searching. As the program is so new, I am not sure on limits for searching, so everything is all learning and understanding at the moment (hence why is best to scrape what you need and no more)
Here are some questions:
1. How to scrape more from 1 user?
What you are doing, but it will depend on amounts / limits.
In this way, no matter how many times I tried, the best is 10k, the worst is 0.
Probably best NOT to keep attempting the same thing over and over again if you have problems. This *COULD* be used as a footprint. Would a real user keep doing this? If you have a problem, best to try again once or twice then let me know, rather than keep repeating.

Hopefully your logs have more info...
2. As I browse manually, most twitter users account got no more then 10k followers. How to scrapes 100s of k niche users as I did in Tumblr? It's easy to do it in Tumblr because there are lots popular post got more than 100k notes. What's the right way to do it in twitter?

As above, I don't know. I haven't done it yet. From the sounds of it, lots of smaller searches, rather than just a single "scrape all followers of X" or "scrape all notes of Y".

Perhaps a CUSTOM SEARCH OF
FOLLOWERS OF
FOLLOWERS OF

meaning you scrape 200 follows of profileA
then from each of those 200 results, you scrape 200 followers from each

thus 200x200 = 40k

IF there is a limit per user, this will help get around it.

Danny000
Posts: 106
Joined: Mon Jan 15, 2018 12:19 am

Re: New user need helps

Post by Danny000 » Wed Aug 15, 2018 1:46 pm

Hi Martin

I actually not did the same thing repeatedly on the same account. I use one account for scraping only once, every time. The jobs stop at 5k - 10k.

I need lots of niche users because I run hundreds of accounts.

As per your suggestion,

CUSTOM SEARCH OF
FOLLOWERS OF
FOLLOWERS OF
it might scrape much more followers, but those scraped are not niche users any more.

For example, I start with a sexy girl account, its followers are mainly guys, they are what I need to scrape, but these guys' followers are not what I am looking for.

and one question off this topic:
then from each of those 200 results, you scrape 200 followers from each

thus 200x200 = 40k

IF there is a limit per user, this will help get around it.
I think it will be the same account to scrape followers of the 200 account, and that make the same account doing the same thing again and again, and again. isn't it?

-----------------------------------------

So I still need a way to scrape more niche users. Any other suggestion?

thanks
Danny

martin@rootjazz wrote:
Thu Aug 09, 2018 12:06 pm

User avatar
martin@rootjazz
Site Admin
Posts: 17380
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: New user need helps

Post by martin@rootjazz » Wed Aug 15, 2018 11:02 pm

Danny000 wrote:
Wed Aug 15, 2018 1:46 pm
Hi Martin

I actually not did the same thing repeatedly on the same account. I use one account for scraping only once, every time. The jobs stop at 5k - 10k.

I need lots of niche users because I run hundreds of accounts.
ah ok, was just checking as is not unusual for someone with one account to just want to scrape as many as possible for no reason.

What you may benefit from is a suggestion I have on the TODO list is a way to share out the searching between multiple accounts. Although no ETA on that at this moment as I want the new update to settle in first, before I start ripping parts out and "improving" (without new bugs)
As per your suggestion,

CUSTOM SEARCH OF
FOLLOWERS OF
FOLLOWERS OF
it might scrape much more followers, but those scraped are not niche users any more.
Correct, this isn't going to be a good search for you

Perhaps something similar but different.

scrape hashtag #sexygirl
followers

should be loads of images with "sexygirl" then get their followers?

Also:
hashtag
interactions (commenters / likers)

Although commenters / likers is limited as twitter won't provide that many results, but if you can find enough images might be something - but you would want popular and not recent to ensure at least some interactions
and one question off this topic:
then from each of those 200 results, you scrape 200 followers from each

thus 200x200 = 40k

IF there is a limit per user, this will help get around it.
I think it will be the same account to scrape followers of the 200 account, and that make the same account doing the same thing again and again, and again. isn't it?
yes, you need the above mentioned multi account scrape

User avatar
martin@rootjazz
Site Admin
Posts: 17380
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: New user need helps

Post by martin@rootjazz » Wed Aug 15, 2018 11:06 pm

Danny000 wrote:
Wed Aug 15, 2018 8:16 am


Next cursor: 1608821652557571885
FAILED GET: (163479) 502
http://51.15.222.94:22694/
* ERROR: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercomcloudxxx_2018-08-15.txt
Started: 15/08/2018 00:10 AM
Finished: 15/08/2018 00:15 AM
ID: 40a20631-827c-4479-995c-f920ce16c3bd
Action ran for: 0hr:5min:19s


This is caused by your proxy failing

from your logs:
502
http://51.15.222.94:22694/
<html><head><title>502 Bad Gateway</title></head>
<body><h2>502 Bad Gateway</h2><h3>Host Not Found or connection failed</h3></body></html>

Seems like there was more results, but this failed request caused the search to drop out.

Possible improvements:
* try again failed requests if proxy / network issue suspected.
* I also noticed, that if the search fails, the scraped so far items are not saved. They should really be saved to file as the scrape processes, otherwise the issue is you get to scrape 100k, network blips and they are lost - would be quite infuriating.

Notes made to look into both

Danny000
Posts: 106
Joined: Mon Jan 15, 2018 12:19 am

Re: New user need helps

Post by Danny000 » Thu Aug 16, 2018 9:06 am

..
Last edited by Danny000 on Thu Aug 16, 2018 9:27 am, edited 1 time in total.

Danny000
Posts: 106
Joined: Mon Jan 15, 2018 12:19 am

Re: New user need helps

Post by Danny000 » Thu Aug 16, 2018 9:16 am

Hi Martin

thanks for your reply. I got many thoughts here.

Firstly, as per the issue I submitted, some scraping will stop due to proxy problem, normally the task will save those already scraped users in most cases.
my question is: is it possible to find a way to carry on the scraping job? such as, restart the job automatically after a few minutes from where it stopped, or switch to another account when issue occurs.

Secondly, I now found a way to gather niche accounts.
Some niche accounts, they will promote other similar niche accounts to get paid or for exchange-promotion. Some tweet will have lots of "@xxx1 @xxx2 @xxx3", these accounts "@xxx1 @xxx2 @xxx3" are also what we need.

Image

So my thought is ------ to develop a custom search to collect these accounts: "@xxx1 @xxx2 @xxx3" mention in ones' tweet. (Of course need to remove duplicates coz I saw many).

After gathering these accounts, there are two ways to use them:

1. Assign each one (for example: @xxx1) to one of my account, and my account will do this: to follow each follower of @xxx1.
In this case, my account will do search and follow in one job task.

2. Just scrape all followers of @xxx1 @xxx2 @xxx3 ... for further use. (I prefer this.)

The scraping job need to be made sure to carry on under any circumstances (as mentioned before, if stopped, "restart the job automatically after a few minutes from where it stopped, or switch to another account when issue occurs ")

Also consider assign the scraping job in bulk.(I mean NOT assign the job one acc to one acc, but in bulk. I did it one by on in TJ, and it takes lots of time.)

By now I believe this is the best way to gather tons of niche users:
Gather accounts mentioned in ones' tweets -- collect their followers
I believe it should be simple to develop this feature if there is no such thing.

If this feature already there, would you please teach me how to set up.


Thanks
Danny
martin@rootjazz wrote:
Sat Aug 11, 2018 3:45 pm

User avatar
martin@rootjazz
Site Admin
Posts: 17380
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: New user need helps

Post by martin@rootjazz » Thu Aug 16, 2018 12:30 pm

Danny000 wrote:
Thu Aug 16, 2018 9:16 am
Hi Martin

thanks for your reply. I got many thoughts here.

Firstly, as per the issue I submitted, some scraping will stop due to proxy problem, normally the task will save those already scraped users in most cases.
my question is: is it possible to find a way to carry on the scraping job? such as, restart the job automatically after a few minutes from where it stopped, or switch to another account when issue occurs.
No, not currently possible.
Technically it would be possible to add this (have to log out the search page cursor) but it is not possible at the moment and honestly is probably not an easy add. Note made to look into (however I have notes made for 100s of improvements / features at the moment so no promises on when or if)
Secondly, I now found a way to gather niche accounts.
Some niche accounts, they will promote other similar niche accounts to get paid or for exchange-promotion. Some tweet will have lots of "@xxx1 @xxx2 @xxx3", these accounts "@xxx1 @xxx2 @xxx3" are also what we need.

Image

So my thought is ------ to develop a custom search to collect these accounts: "@xxx1 @xxx2 @xxx3" mention in ones' tweet. (Of course need to remove duplicates coz I saw many).

After gathering these accounts, there are two ways to use them:

1. Assign each one (for example: @xxx1) to one of my account, and my account will do this: to follow each follower of @xxx1.
In this case, my account will do search and follow in one job task.

2. Just scrape all followers of @xxx1 @xxx2 @xxx3 ... for further use. (I prefer this.)

The scraping job need to be made sure to carry on under any circumstances (as mentioned before, if stopped, "restart the job automatically after a few minutes from where it stopped, or switch to another account when issue occurs ")
https://twitter.com/search-advanced
Info on how to use the advanced search should be linked from the SCRAPER tab at the bottom

Also consider assign the scraping job in bulk.(I mean NOT assign the job one acc to one acc, but in bulk. I did it one by on in TJ, and it takes lots of time.)
you should be able to with text files.

For your search input, enter a text file, one each line being a new action search input.
So custom search
HASHTAG
FOLLOWERS

input: c:\path\to\text.txt
the file contains 10 lines, one tag per line

you would get 10 search actions

Post Reply