Hi
I need to scrape a user list for follow/like etc, but the scraper is a lot different from TJ and kind of complicated, I can't even understand the difference of: ID, profile. and there is no documentation and i can't find tutorial video.
and if I use daily promo for 1000 accounts, that will create 3000 entries. Is TD very old? seems many tab job are the same, creating 1 entry for every account.
Question:
I need a file of users, which TD can follow from. would you please show me how to scrape niche users , such as users who guitar (or anything else).
thanks
Please tell me how to scrape users
-
- Posts: 2317
- Joined: Tue Feb 02, 2016 1:58 pm
Re: Please tell me how to scrape users
Please have a look on this thread and watch them all
TD is not OLD but TD is slightly different as TD works through API
https://rootjazz.com/forum/viewforum.php?f=32
In order to scrape followers or whatever - Please always use custom search
For example: I want to scrape followers of justinbieber
Ao, i will goto TD scraper tab
Look for scrape profiles
Input the profile url of jusatinbieber ( https://twitter.com/justinbieber)
Then click on custom search
A user url or ID
Followers of
Save
Look inot the right side and choose output type ( ID or URL ? check that )
Select an account
Hit scrape
AS TD is API based - it requests to API and waits and after few time when the API response gets ended, TD starts to read them ...... and saves it.
Hope the above information helps.
Regards
TD is not OLD but TD is slightly different as TD works through API
https://rootjazz.com/forum/viewforum.php?f=32
In order to scrape followers or whatever - Please always use custom search
For example: I want to scrape followers of justinbieber
Ao, i will goto TD scraper tab
Look for scrape profiles
Input the profile url of jusatinbieber ( https://twitter.com/justinbieber)
Then click on custom search
A user url or ID
Followers of
Save
Look inot the right side and choose output type ( ID or URL ? check that )
Select an account
Hit scrape
AS TD is API based - it requests to API and waits and after few time when the API response gets ended, TD starts to read them ...... and saves it.
Hope the above information helps.
Regards
Re: Please tell me how to scrape users
FYI, that return an empty file. Here is the realtime log:support@rootjazz wrote: ↑Mon May 28, 2018 5:11 am
For example: I want to scrape followers of justinbieber
Ao, i will goto TD scraper tab
Look for scrape profiles
Input the profile url of jusatinbieber ( https://twitter.com/justinbieber)
Then click on custom search
A user url or ID
Followers of
Save
Look inot the right side and choose output type ( ID or URL ? check that )
Select an account
Hit scrape
Starting: 30/05/2018 02:44 AM
Perform custom search
Max Results Wanted: 5000
Setup custom search controller
Custom search run: search: https://twitter.com/justinbieber
Perform custom search: #chain/total: 1/2 using: Adams4283Ryan
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Results of search: A User Id or Url with: https://twitter.com/justinbieber
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/justinbieber
Perform custom search: #chain/total: 2/2 using: Adams4283Ryan
Start search: FollowersOf with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Get TwitterUser: justinbieber using: Adams4283Ryan
Got user: https://twitter.com/justinbieber username: Justin Bieber #followers: 106419695
Got Followers Ids: 0
Results of search: FollowersOf with: https://twitter.com/justinbieber
Handle results: 0 nextstep: 2/2
End of chain: Store results: 0
Perform custom search: #chain/total: 1/2 using: Adams4283Ryan
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Results of search: A User Id or Url with: https://twitter.com/justinbieber
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/justinbieber
Perform custom search: #chain/total: 2/2 using: Adams4283Ryan
Start search: FollowersOf with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Get TwitterUser: justinbieber using: Adams4283Ryan
Got user: https://twitter.com/justinbieber username: Justin Bieber #followers: 106419702
Got Followers Ids: 0
Results of search: FollowersOf with: https://twitter.com/justinbieber
Handle results: 0 nextstep: 2/2
End of chain: Store results: 0
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercomjustinbieber_2018-05-30.txt
Started: 30/05/2018 02:44 AM
Finished: 30/05/2018 02:45 AM
ID: 25fca8ec-4039-491b-9acd-047989772889
Action ran for: 0hr:0min:30s
-
- Posts: 2317
- Joined: Tue Feb 02, 2016 1:58 pm
Re: Please tell me how to scrape users
CAn you please try with 2 -3 other accs?Danny000 wrote: ↑Tue May 29, 2018 2:47 pmFYI, that return an empty file. Here is the realtime log:support@rootjazz wrote: ↑Mon May 28, 2018 5:11 am
For example: I want to scrape followers of justinbieber
Ao, i will goto TD scraper tab
Look for scrape profiles
Input the profile url of jusatinbieber ( https://twitter.com/justinbieber)
Then click on custom search
A user url or ID
Followers of
Save
Look inot the right side and choose output type ( ID or URL ? check that )
Select an account
Hit scrape
Starting: 30/05/2018 02:44 AM
Perform custom search
Max Results Wanted: 5000
Setup custom search controller
Custom search run: search: https://twitter.com/justinbieber
Perform custom search: #chain/total: 1/2 using: Adams4283Ryan
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Results of search: A User Id or Url with: https://twitter.com/justinbieber
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/justinbieber
Perform custom search: #chain/total: 2/2 using: Adams4283Ryan
Start search: FollowersOf with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Get TwitterUser: justinbieber using: Adams4283Ryan
Got user: https://twitter.com/justinbieber username: Justin Bieber #followers: 106419695
Got Followers Ids: 0
Results of search: FollowersOf with: https://twitter.com/justinbieber
Handle results: 0 nextstep: 2/2
End of chain: Store results: 0
Perform custom search: #chain/total: 1/2 using: Adams4283Ryan
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Results of search: A User Id or Url with: https://twitter.com/justinbieber
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/justinbieber
Perform custom search: #chain/total: 2/2 using: Adams4283Ryan
Start search: FollowersOf with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Get TwitterUser: justinbieber using: Adams4283Ryan
Got user: https://twitter.com/justinbieber username: Justin Bieber #followers: 106419702
Got Followers Ids: 0
Results of search: FollowersOf with: https://twitter.com/justinbieber
Handle results: 0 nextstep: 2/2
End of chain: Store results: 0
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercomjustinbieber_2018-05-30.txt
Started: 30/05/2018 02:44 AM
Finished: 30/05/2018 02:45 AM
ID: 25fca8ec-4039-491b-9acd-047989772889
Action ran for: 0hr:0min:30s
The api did not return any value.
Can you please try?
Sometimes it happens and due to that API or depending upon accs.
Regards
Re: Please tell me how to scrape users
To make a full description of this issue, I re-install the system, re-install TD, everything start over again.
OS: windows server 2012 R2 (Fresh and clean)
TD: v1.434 (Fresh installed)
Log submitted: 60744
Only one account input, it's PVA and has all necessary API things, this account should be all good.
I use its own API thing and I did all the procedure and record here, maybe you can find something:
Setup API key, Athorize Accounts, API APP setting unlock, Fix key permission, they all succeeded, I'll skip the realtime logs here.
But the verify accounts is weird. API login is specified successful (inside the log) , but the verify accounts job was tagged Failed.
Realtime log here:
And also the custom search failed.Starting: 31/05/2018 02:07 AM
Validate account via login NancyLexxxxx sofya.grushnikova.1994@mail.ru
* SUCCESS: NancyLe99615972 sofya.grushnikova.1994@mail.ru validated successfully
FAILED VERIFY:
SUCCESS VERIFY:
http://twitter.com/NancyLexxxxx
1 accounts validated out of 1
* ERROR: The given key was not present in the dictionary.
* UNKNOWN ERROR OCCURRED - see logs for more info.
If the problem continues, please submit logs to support.
The given key was not present in the dictionary.
The given key was not present in the dictionary.
Custom search Setting:
A user url or ID
Followers of
realtime log:
I think i missed something similar to the verify accounts and cause all the problem.Starting: 31/05/2018 02:03 AM
Perform custom search
Max Results Wanted: 5000
Setup custom search controller
Custom search run: search: https://twitter.com/porn_xxx
Perform custom search: #chain/total: 1/2 using: NancyLe99615972
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Results of search: A User Id or Url with: https://twitter.com/porn_xxx
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/porn_xxx
Perform custom search: #chain/total: 2/2 using: NancyLe99615972
Start search: FollowersOf with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Get TwitterUser: porn_xxx using: NancyLe99615972
Got user: https://twitter.com/porn_xxx username: Gay sex porn videos #followers: 113227
* ERROR: TwitterErrorLog: Exception has been thrown by the target of an invocation.
* ERROR: Exception has been thrown by the target of an invocation.
Results of search: FollowersOf with: https://twitter.com/porn_xxx
Handle results: 0 nextstep: 2/2
End of chain: Store results: 0
Perform custom search: #chain/total: 1/2 using: NancyLe99615972
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Results of search: A User Id or Url with: https://twitter.com/porn_xxx
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/porn_xxx
Perform custom search: #chain/total: 2/2 using: NancyLe99615972
Start search: FollowersOf with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Get TwitterUser: porn_xxx using: NancyLe99615972
* ERROR: TwitterErrorLog: Exception has been thrown by the target of an invocation.
* ERROR: handleAResult: Term: https://twitter.com/porn_xxx Exception has been thrown by the target of an invocation.
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercompornxxx_2018-05-31_2.txt
Started: 31/05/2018 02:03 AM
Finished: 31/05/2018 02:04 AM
ID: de9ff4af-4f66-420f-82d7-139a04111edb
Action ran for: 0hr:0min:18s
Rgds.
-
- Posts: 2317
- Joined: Tue Feb 02, 2016 1:58 pm
Re: Please tell me how to scrape users
Twitters api had some issues last 24 hours of more than that.Danny000 wrote: ↑Thu May 31, 2018 9:51 amTo make a full description of this issue, I re-install the system, re-install TD, everything start over again.
OS: windows server 2012 R2 (Fresh and clean)
TD: v1.434 (Fresh installed)
Log submitted: 60744
Only one account input, it's PVA and has all necessary API things, this account should be all good.
I use its own API thing and I did all the procedure and record here, maybe you can find something:
Setup API key, Athorize Accounts, API APP setting unlock, Fix key permission, they all succeeded, I'll skip the realtime logs here.
But the verify accounts is weird. API login is specified successful (inside the log) , but the verify accounts job was tagged Failed.
Realtime log here:
And also the custom search failed.Starting: 31/05/2018 02:07 AM
Validate account via login NancyLexxxxx sofya.grushnikova.1994@mail.ru
* SUCCESS: NancyLe99615972 sofya.grushnikova.1994@mail.ru validated successfully
FAILED VERIFY:
SUCCESS VERIFY:
http://twitter.com/NancyLexxxxx
1 accounts validated out of 1
* ERROR: The given key was not present in the dictionary.
* UNKNOWN ERROR OCCURRED - see logs for more info.
If the problem continues, please submit logs to support.
The given key was not present in the dictionary.
The given key was not present in the dictionary.
Custom search Setting:
A user url or ID
Followers of
realtime log:I think i missed something similar to the verify accounts and cause all the problem.Starting: 31/05/2018 02:03 AM
Perform custom search
Max Results Wanted: 5000
Setup custom search controller
Custom search run: search: https://twitter.com/porn_xxx
Perform custom search: #chain/total: 1/2 using: NancyLe99615972
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Results of search: A User Id or Url with: https://twitter.com/porn_xxx
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/porn_xxx
Perform custom search: #chain/total: 2/2 using: NancyLe99615972
Start search: FollowersOf with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Get TwitterUser: porn_xxx using: NancyLe99615972
Got user: https://twitter.com/porn_xxx username: Gay sex porn videos #followers: 113227
* ERROR: TwitterErrorLog: Exception has been thrown by the target of an invocation.
* ERROR: Exception has been thrown by the target of an invocation.
Results of search: FollowersOf with: https://twitter.com/porn_xxx
Handle results: 0 nextstep: 2/2
End of chain: Store results: 0
Perform custom search: #chain/total: 1/2 using: NancyLe99615972
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Results of search: A User Id or Url with: https://twitter.com/porn_xxx
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/porn_xxx
Perform custom search: #chain/total: 2/2 using: NancyLe99615972
Start search: FollowersOf with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Get TwitterUser: porn_xxx using: NancyLe99615972
* ERROR: TwitterErrorLog: Exception has been thrown by the target of an invocation.
* ERROR: handleAResult: Term: https://twitter.com/porn_xxx Exception has been thrown by the target of an invocation.
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercompornxxx_2018-05-31_2.txt
Started: 31/05/2018 02:03 AM
Finished: 31/05/2018 02:04 AM
ID: de9ff4af-4f66-420f-82d7-139a04111edb
Action ran for: 0hr:0min:18s
Rgds.
Can you please check the action again?
I think the issue should be solved if you update to the latest TD version.
Regards
Re: Please tell me how to scrape users
No, verify accounts still failed.
- martin@rootjazz
- Site Admin
- Posts: 35057
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Please tell me how to scrape users
The last logs you sent were definitely the Twitter API issue
Can you regenerate me the logs so I can see the issue without this API issue. Sometimes can be hard to see your actual issue, when there are other issues as well polluting the logs.* ERROR: TwitterErrorLog: Exception has been thrown by the target of an invocation.
Please perform the following:
Open the program. Delete or pause all existing actions. If actions are already running due to autorun on startup, disable autorun at startup and restart the app. All actions must be waiting, then pause all (or delete)
Delete existing logs
HELP > LOGS > VIEW
delete all files in the folder that opens
Perform the action:
***
run a test verify issue with just 1 account to cause your issue
submit logs
***
PLEASE NOTE: Do not perform other actions at the same time. Only use one thread. Do not run any actions after this action.
Then submit logs:
HELP > LOGS > SUBMIT
and let me know the LOGS ID (just the number part is sufficient)
Regards,
Martin
Re: Please tell me how to scrape users
Log submitted again: 76352
Realtime log:
Starting: 02/06/2018 15:47 PM
Validate account via login NancyLe99615972 sofya.grushnikova.1994@mail.ru
* SUCCESS: NancyLe99615972 sofya.grushnikova.1994@mail.ru validated successfully
FAILED VERIFY:
SUCCESS VERIFY:
http://twitter.com/NancyLe99615972
1 accounts validated out of 1
* ERROR: The given key was not present in the dictionary.
* UNKNOWN ERROR OCCURRED - see logs for more info.
If the problem continues, please submit logs to support.
The given key was not present in the dictionary.
The given key was not present in the dictionary.
- martin@rootjazz
- Site Admin
- Posts: 35057
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Please tell me how to scrape users
sorry this wasn't noticed before but you on using an old version of the software (by about 3 weeks)
https://rootjazz.com/twitterdub/updatetesting.html
I have a feeling this issue has already been fixed
https://rootjazz.com/twitterdub/updatetesting.html
I have a feeling this issue has already been fixed