Hello there,
I am an old user of TumblingJazz, and became a user of Twitterdub 2 months later. But I couldn't use TD as good as TJ due to too many problems began with API issue, and I couldn't find the answer I need from tutorial videos. so I stopped using it. Now the non-API version launch, maybe it's time for me to start over again, yet I still need help, from the basic use of TD.
Basically I need to find niche users and then follow them. I tried it myself many times and still couldn't find the right way. So would you please describe it step by step, the more detailed the better:
For example, if I am doing dating niche, need to search niche users who like sexy women, the keyword is "sexy woman", how do I perform this?
I understand this is really simple for you guys, and thanks for any help.
Danny
New user need helps
- martin@rootjazz
- Site Admin
- Posts: 34640
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: New user need helps
Can you explain what you tried and why it didn't work.
TWEET SEARCH. That will find users who posted a tweet with those keywords. You can do RECENT or POPULAR searches of this (same as TUMJAZZ)For example, if I am doing dating niche, need to search niche users who like sexy women, the keyword is "sexy woman", how do I perform this?
You can also scrape NOTES (same as tumjazz) via custom search (tweet interactions it is called)
Same as TumJazz you can scrape followers of a particular account, you can scrape tweets of a similar niche account then scrape those you intereact with their media.
Basically if you can think of a way to find users, you can do it with CUSTOM SEARCH (which is a lot more powerful in TD than Tumjazz). IF you can explain your searches in words, I can explain how to do it in the app
Re: New user need helps
Hi Martin
I finally succeeded to finish some scraping job, with simplest way - scrape followers. But the result is not good enough. I search on a specify user's followers which has about 50k, TD only scraped 5-10k and then stopped. Here I tried to do it again to submit some log, and got 0 results:
Log submitted: 35823
1. How to scrape more from 10k followers from one account?
In this way, no matter how many times I tried, the best is 10k, the worst is 0.
2. As I browse manually, most twitter users account got no more then 10k followers. How to scrapes 100s of k niche users as I did in Tumblr? It's easy to do it in Tumblr because there are lots popular post got more than 100k notes. What's the right way to do it in twitter?
thanks
Danny
I finally succeeded to finish some scraping job, with simplest way - scrape followers. But the result is not good enough. I search on a specify user's followers which has about 50k, TD only scraped 5-10k and then stopped. Here I tried to do it again to submit some log, and got 0 results:
Log submitted: 35823
Here are some questions:Next cursor: 1608821652557571885
FAILED GET: (163479) 502
http://51.15.222.94:22694/
* ERROR: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercomcloudxxx_2018-08-15.txt
Started: 15/08/2018 00:10 AM
Finished: 15/08/2018 00:15 AM
ID: 40a20631-827c-4479-995c-f920ce16c3bd
Action ran for: 0hr:5min:19s
1. How to scrape more from 10k followers from one account?
In this way, no matter how many times I tried, the best is 10k, the worst is 0.
2. As I browse manually, most twitter users account got no more then 10k followers. How to scrapes 100s of k niche users as I did in Tumblr? It's easy to do it in Tumblr because there are lots popular post got more than 100k notes. What's the right way to do it in twitter?
thanks
Danny
Last edited by Danny000 on Wed Aug 15, 2018 1:24 pm, edited 1 time in total.
- martin@rootjazz
- Site Admin
- Posts: 34640
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: New user need helps
Why are you trying to scrape more than 50k results? Do you need that many? (possibly yes, but possibly no as well).Danny000 wrote: ↑Wed Aug 15, 2018 8:16 am Hi Martin
I finally succeeded to finish some scraping job, with simplest way - scrape followers. But the result is not good enough. I search on a specify user's followers which has about 50k, TD only scraped 5-10k and then stopped. Here I tried to do it again to submit some log, and got 0 results:
Log submitted: 35823
Next cursor: 1608821652557571885
FAILED GET: (163479) 502
http://51.15.222.94:22694/
* ERROR: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercomcloudxxx_2018-08-15.txt
Started: 15/08/2018 00:10 AM
Finished: 15/08/2018 00:15 AM
ID: 40a20631-827c-4479-995c-f920ce16c3bd
Action ran for: 0hr:5min:19s
After your scrape, you got a 502 error, which is a server error. Not too sure why, am downloading your logs to look. But the next time you got 0, sounds like a server block for over searching. As the program is so new, I am not sure on limits for searching, so everything is all learning and understanding at the moment (hence why is best to scrape what you need and no more)
What you are doing, but it will depend on amounts / limits.Here are some questions:
1. How to scrape more from 1 user?
Probably best NOT to keep attempting the same thing over and over again if you have problems. This *COULD* be used as a footprint. Would a real user keep doing this? If you have a problem, best to try again once or twice then let me know, rather than keep repeating.In this way, no matter how many times I tried, the best is 10k, the worst is 0.
Hopefully your logs have more info...
2. As I browse manually, most twitter users account got no more then 10k followers. How to scrapes 100s of k niche users as I did in Tumblr? It's easy to do it in Tumblr because there are lots popular post got more than 100k notes. What's the right way to do it in twitter?
As above, I don't know. I haven't done it yet. From the sounds of it, lots of smaller searches, rather than just a single "scrape all followers of X" or "scrape all notes of Y".
Perhaps a CUSTOM SEARCH OF
FOLLOWERS OF
FOLLOWERS OF
meaning you scrape 200 follows of profileA
then from each of those 200 results, you scrape 200 followers from each
thus 200x200 = 40k
IF there is a limit per user, this will help get around it.
Re: New user need helps
Hi Martin
I actually not did the same thing repeatedly on the same account. I use one account for scraping only once, every time. The jobs stop at 5k - 10k.
I need lots of niche users because I run hundreds of accounts.
As per your suggestion,
CUSTOM SEARCH OF
For example, I start with a sexy girl account, its followers are mainly guys, they are what I need to scrape, but these guys' followers are not what I am looking for.
and one question off this topic:
-----------------------------------------
So I still need a way to scrape more niche users. Any other suggestion?
thanks
Danny
I actually not did the same thing repeatedly on the same account. I use one account for scraping only once, every time. The jobs stop at 5k - 10k.
I need lots of niche users because I run hundreds of accounts.
As per your suggestion,
CUSTOM SEARCH OF
it might scrape much more followers, but those scraped are not niche users any more.FOLLOWERS OF
FOLLOWERS OF
For example, I start with a sexy girl account, its followers are mainly guys, they are what I need to scrape, but these guys' followers are not what I am looking for.
and one question off this topic:
I think it will be the same account to scrape followers of the 200 account, and that make the same account doing the same thing again and again, and again. isn't it?then from each of those 200 results, you scrape 200 followers from each
thus 200x200 = 40k
IF there is a limit per user, this will help get around it.
-----------------------------------------
So I still need a way to scrape more niche users. Any other suggestion?
thanks
Danny
- martin@rootjazz
- Site Admin
- Posts: 34640
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: New user need helps
ah ok, was just checking as is not unusual for someone with one account to just want to scrape as many as possible for no reason.
What you may benefit from is a suggestion I have on the TODO list is a way to share out the searching between multiple accounts. Although no ETA on that at this moment as I want the new update to settle in first, before I start ripping parts out and "improving" (without new bugs)
Correct, this isn't going to be a good search for youAs per your suggestion,
CUSTOM SEARCH OF
it might scrape much more followers, but those scraped are not niche users any more.FOLLOWERS OF
FOLLOWERS OF
Perhaps something similar but different.
scrape hashtag #sexygirl
followers
should be loads of images with "sexygirl" then get their followers?
Also:
hashtag
interactions (commenters / likers)
Although commenters / likers is limited as twitter won't provide that many results, but if you can find enough images might be something - but you would want popular and not recent to ensure at least some interactions
yes, you need the above mentioned multi account scrapeand one question off this topic:
I think it will be the same account to scrape followers of the 200 account, and that make the same account doing the same thing again and again, and again. isn't it?then from each of those 200 results, you scrape 200 followers from each
thus 200x200 = 40k
IF there is a limit per user, this will help get around it.
- martin@rootjazz
- Site Admin
- Posts: 34640
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: New user need helps
Danny000 wrote: ↑Wed Aug 15, 2018 8:16 am
Next cursor: 1608821652557571885
FAILED GET: (163479) 502
http://51.15.222.94:22694/
* ERROR: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercomcloudxxx_2018-08-15.txt
Started: 15/08/2018 00:10 AM
Finished: 15/08/2018 00:15 AM
ID: 40a20631-827c-4479-995c-f920ce16c3bd
Action ran for: 0hr:5min:19s
This is caused by your proxy failing
from your logs:
502
http://51.15.222.94:22694/
<html><head><title>502 Bad Gateway</title></head>
<body><h2>502 Bad Gateway</h2><h3>Host Not Found or connection failed</h3></body></html>
Seems like there was more results, but this failed request caused the search to drop out.
Possible improvements:
* try again failed requests if proxy / network issue suspected.
* I also noticed, that if the search fails, the scraped so far items are not saved. They should really be saved to file as the scrape processes, otherwise the issue is you get to scrape 100k, network blips and they are lost - would be quite infuriating.
Notes made to look into both
Re: New user need helps
..
Last edited by Danny000 on Thu Aug 16, 2018 9:27 am, edited 1 time in total.
Re: New user need helps
Hi Martin
thanks for your reply. I got many thoughts here.
Firstly, as per the issue I submitted, some scraping will stop due to proxy problem, normally the task will save those already scraped users in most cases.
my question is: is it possible to find a way to carry on the scraping job? such as, restart the job automatically after a few minutes from where it stopped, or switch to another account when issue occurs.
Secondly, I now found a way to gather niche accounts.
Some niche accounts, they will promote other similar niche accounts to get paid or for exchange-promotion. Some tweet will have lots of "@xxx1 @xxx2 @xxx3", these accounts "@xxx1 @xxx2 @xxx3" are also what we need.
So my thought is ------ to develop a custom search to collect these accounts: "@xxx1 @xxx2 @xxx3" mention in ones' tweet. (Of course need to remove duplicates coz I saw many).
After gathering these accounts, there are two ways to use them:
1. Assign each one (for example: @xxx1) to one of my account, and my account will do this: to follow each follower of @xxx1.
In this case, my account will do search and follow in one job task.
2. Just scrape all followers of @xxx1 @xxx2 @xxx3 ... for further use. (I prefer this.)
The scraping job need to be made sure to carry on under any circumstances (as mentioned before, if stopped, "restart the job automatically after a few minutes from where it stopped, or switch to another account when issue occurs ")
Also consider assign the scraping job in bulk.(I mean NOT assign the job one acc to one acc, but in bulk. I did it one by on in TJ, and it takes lots of time.)
By now I believe this is the best way to gather tons of niche users:
If this feature already there, would you please teach me how to set up.
Thanks
Danny
thanks for your reply. I got many thoughts here.
Firstly, as per the issue I submitted, some scraping will stop due to proxy problem, normally the task will save those already scraped users in most cases.
my question is: is it possible to find a way to carry on the scraping job? such as, restart the job automatically after a few minutes from where it stopped, or switch to another account when issue occurs.
Secondly, I now found a way to gather niche accounts.
Some niche accounts, they will promote other similar niche accounts to get paid or for exchange-promotion. Some tweet will have lots of "@xxx1 @xxx2 @xxx3", these accounts "@xxx1 @xxx2 @xxx3" are also what we need.
So my thought is ------ to develop a custom search to collect these accounts: "@xxx1 @xxx2 @xxx3" mention in ones' tweet. (Of course need to remove duplicates coz I saw many).
After gathering these accounts, there are two ways to use them:
1. Assign each one (for example: @xxx1) to one of my account, and my account will do this: to follow each follower of @xxx1.
In this case, my account will do search and follow in one job task.
2. Just scrape all followers of @xxx1 @xxx2 @xxx3 ... for further use. (I prefer this.)
The scraping job need to be made sure to carry on under any circumstances (as mentioned before, if stopped, "restart the job automatically after a few minutes from where it stopped, or switch to another account when issue occurs ")
Also consider assign the scraping job in bulk.(I mean NOT assign the job one acc to one acc, but in bulk. I did it one by on in TJ, and it takes lots of time.)
By now I believe this is the best way to gather tons of niche users:
I believe it should be simple to develop this feature if there is no such thing.Gather accounts mentioned in ones' tweets -- collect their followers
If this feature already there, would you please teach me how to set up.
Thanks
Danny
- martin@rootjazz
- Site Admin
- Posts: 34640
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: New user need helps
No, not currently possible.Danny000 wrote: ↑Thu Aug 16, 2018 9:16 am Hi Martin
thanks for your reply. I got many thoughts here.
Firstly, as per the issue I submitted, some scraping will stop due to proxy problem, normally the task will save those already scraped users in most cases.
my question is: is it possible to find a way to carry on the scraping job? such as, restart the job automatically after a few minutes from where it stopped, or switch to another account when issue occurs.
Technically it would be possible to add this (have to log out the search page cursor) but it is not possible at the moment and honestly is probably not an easy add. Note made to look into (however I have notes made for 100s of improvements / features at the moment so no promises on when or if)
https://twitter.com/search-advancedSecondly, I now found a way to gather niche accounts.
Some niche accounts, they will promote other similar niche accounts to get paid or for exchange-promotion. Some tweet will have lots of "@xxx1 @xxx2 @xxx3", these accounts "@xxx1 @xxx2 @xxx3" are also what we need.
So my thought is ------ to develop a custom search to collect these accounts: "@xxx1 @xxx2 @xxx3" mention in ones' tweet. (Of course need to remove duplicates coz I saw many).
After gathering these accounts, there are two ways to use them:
1. Assign each one (for example: @xxx1) to one of my account, and my account will do this: to follow each follower of @xxx1.
In this case, my account will do search and follow in one job task.
2. Just scrape all followers of @xxx1 @xxx2 @xxx3 ... for further use. (I prefer this.)
The scraping job need to be made sure to carry on under any circumstances (as mentioned before, if stopped, "restart the job automatically after a few minutes from where it stopped, or switch to another account when issue occurs ")
Info on how to use the advanced search should be linked from the SCRAPER tab at the bottom
you should be able to with text files.Also consider assign the scraping job in bulk.(I mean NOT assign the job one acc to one acc, but in bulk. I did it one by on in TJ, and it takes lots of time.)
For your search input, enter a text file, one each line being a new action search input.
So custom search
HASHTAG
FOLLOWERS
input: c:\path\to\text.txt
the file contains 10 lines, one tag per line
you would get 10 search actions