Please tell me how to scrape users

Support / help / discussion forum for twitter bot
Danny000
Posts: 130
Joined: Mon Jan 15, 2018 12:19 am

Please tell me how to scrape users

Post by Danny000 »

Hi

I need to scrape a user list for follow/like etc, but the scraper is a lot different from TJ and kind of complicated, I can't even understand the difference of: ID, profile. and there is no documentation and i can't find tutorial video.

and if I use daily promo for 1000 accounts, that will create 3000 entries. Is TD very old? seems many tab job are the same, creating 1 entry for every account.

Question:
I need a file of users, which TD can follow from. would you please show me how to scrape niche users , such as users who guitar (or anything else).

thanks
support@rootjazz
Posts: 2317
Joined: Tue Feb 02, 2016 1:58 pm

Re: Please tell me how to scrape users

Post by support@rootjazz »

Please have a look on this thread and watch them all
TD is not OLD but TD is slightly different as TD works through API

https://rootjazz.com/forum/viewforum.php?f=32



In order to scrape followers or whatever - Please always use custom search



For example: I want to scrape followers of justinbieber


Ao, i will goto TD scraper tab


Look for scrape profiles
Input the profile url of jusatinbieber ( https://twitter.com/justinbieber)
Then click on custom search
A user url or ID
Followers of
Save
Look inot the right side and choose output type ( ID or URL ? check that )
Select an account
Hit scrape

AS TD is API based - it requests to API and waits and after few time when the API response gets ended, TD starts to read them ...... and saves it.



Hope the above information helps.



Regards
Danny000
Posts: 130
Joined: Mon Jan 15, 2018 12:19 am

Re: Please tell me how to scrape users

Post by Danny000 »

support@rootjazz wrote: Mon May 28, 2018 5:11 am
For example: I want to scrape followers of justinbieber


Ao, i will goto TD scraper tab


Look for scrape profiles
Input the profile url of jusatinbieber ( https://twitter.com/justinbieber)
Then click on custom search
A user url or ID
Followers of
Save
Look inot the right side and choose output type ( ID or URL ? check that )
Select an account
Hit scrape

FYI, that return an empty file. Here is the realtime log:
Starting: 30/05/2018 02:44 AM
Perform custom search
Max Results Wanted: 5000
Setup custom search controller
Custom search run: search: https://twitter.com/justinbieber
Perform custom search: #chain/total: 1/2 using: Adams4283Ryan
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Results of search: A User Id or Url with: https://twitter.com/justinbieber
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/justinbieber
Perform custom search: #chain/total: 2/2 using: Adams4283Ryan
Start search: FollowersOf with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Get TwitterUser: justinbieber using: Adams4283Ryan
Got user: https://twitter.com/justinbieber username: Justin Bieber #followers: 106419695
Got Followers Ids: 0
Results of search: FollowersOf with: https://twitter.com/justinbieber
Handle results: 0 nextstep: 2/2
End of chain: Store results: 0
Perform custom search: #chain/total: 1/2 using: Adams4283Ryan
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Results of search: A User Id or Url with: https://twitter.com/justinbieber
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/justinbieber
Perform custom search: #chain/total: 2/2 using: Adams4283Ryan
Start search: FollowersOf with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Get TwitterUser: justinbieber using: Adams4283Ryan
Got user: https://twitter.com/justinbieber username: Justin Bieber #followers: 106419702
Got Followers Ids: 0
Results of search: FollowersOf with: https://twitter.com/justinbieber
Handle results: 0 nextstep: 2/2
End of chain: Store results: 0
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercomjustinbieber_2018-05-30.txt
Started: 30/05/2018 02:44 AM
Finished: 30/05/2018 02:45 AM
ID: 25fca8ec-4039-491b-9acd-047989772889
Action ran for: 0hr:0min:30s
support@rootjazz
Posts: 2317
Joined: Tue Feb 02, 2016 1:58 pm

Re: Please tell me how to scrape users

Post by support@rootjazz »

Danny000 wrote: Tue May 29, 2018 2:47 pm
support@rootjazz wrote: Mon May 28, 2018 5:11 am
For example: I want to scrape followers of justinbieber


Ao, i will goto TD scraper tab


Look for scrape profiles
Input the profile url of jusatinbieber ( https://twitter.com/justinbieber)
Then click on custom search
A user url or ID
Followers of
Save
Look inot the right side and choose output type ( ID or URL ? check that )
Select an account
Hit scrape

FYI, that return an empty file. Here is the realtime log:
Starting: 30/05/2018 02:44 AM
Perform custom search
Max Results Wanted: 5000
Setup custom search controller
Custom search run: search: https://twitter.com/justinbieber
Perform custom search: #chain/total: 1/2 using: Adams4283Ryan
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Results of search: A User Id or Url with: https://twitter.com/justinbieber
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/justinbieber
Perform custom search: #chain/total: 2/2 using: Adams4283Ryan
Start search: FollowersOf with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Get TwitterUser: justinbieber using: Adams4283Ryan
Got user: https://twitter.com/justinbieber username: Justin Bieber #followers: 106419695
Got Followers Ids: 0
Results of search: FollowersOf with: https://twitter.com/justinbieber
Handle results: 0 nextstep: 2/2
End of chain: Store results: 0
Perform custom search: #chain/total: 1/2 using: Adams4283Ryan
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Results of search: A User Id or Url with: https://twitter.com/justinbieber
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/justinbieber
Perform custom search: #chain/total: 2/2 using: Adams4283Ryan
Start search: FollowersOf with: https://twitter.com/justinbieber(5000) using: Adams4283Ryan
Get TwitterUser: justinbieber using: Adams4283Ryan
Got user: https://twitter.com/justinbieber username: Justin Bieber #followers: 106419702
Got Followers Ids: 0
Results of search: FollowersOf with: https://twitter.com/justinbieber
Handle results: 0 nextstep: 2/2
End of chain: Store results: 0
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercomjustinbieber_2018-05-30.txt
Started: 30/05/2018 02:44 AM
Finished: 30/05/2018 02:45 AM
ID: 25fca8ec-4039-491b-9acd-047989772889
Action ran for: 0hr:0min:30s
CAn you please try with 2 -3 other accs?
The api did not return any value.
Can you please try?

Sometimes it happens and due to that API or depending upon accs.



Regards
Danny000
Posts: 130
Joined: Mon Jan 15, 2018 12:19 am

Re: Please tell me how to scrape users

Post by Danny000 »

support@rootjazz wrote: Wed May 30, 2018 5:09 am
To make a full description of this issue, I re-install the system, re-install TD, everything start over again.

OS: windows server 2012 R2 (Fresh and clean)
TD: v1.434 (Fresh installed)

Log submitted: 60744

Only one account input, it's PVA and has all necessary API things, this account should be all good.

I use its own API thing and I did all the procedure and record here, maybe you can find something:

Setup API key, Athorize Accounts, API APP setting unlock, Fix key permission, they all succeeded, I'll skip the realtime logs here.

But the verify accounts is weird. API login is specified successful (inside the log) , but the verify accounts job was tagged Failed.
Realtime log here:
Starting: 31/05/2018 02:07 AM
Validate account via login NancyLexxxxx sofya.grushnikova.1994@mail.ru
* SUCCESS: NancyLe99615972 sofya.grushnikova.1994@mail.ru validated successfully
FAILED VERIFY:

SUCCESS VERIFY:
http://twitter.com/NancyLexxxxx
1 accounts validated out of 1
* ERROR: The given key was not present in the dictionary.
* UNKNOWN ERROR OCCURRED - see logs for more info.
If the problem continues, please submit logs to support.
The given key was not present in the dictionary.

The given key was not present in the dictionary.
And also the custom search failed.

Custom search Setting:
A user url or ID
Followers of

realtime log:
Starting: 31/05/2018 02:03 AM
Perform custom search
Max Results Wanted: 5000
Setup custom search controller
Custom search run: search: https://twitter.com/porn_xxx
Perform custom search: #chain/total: 1/2 using: NancyLe99615972
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Results of search: A User Id or Url with: https://twitter.com/porn_xxx
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/porn_xxx
Perform custom search: #chain/total: 2/2 using: NancyLe99615972
Start search: FollowersOf with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Get TwitterUser: porn_xxx using: NancyLe99615972
Got user: https://twitter.com/porn_xxx username: Gay sex porn videos #followers: 113227
* ERROR: TwitterErrorLog: Exception has been thrown by the target of an invocation.
* ERROR: Exception has been thrown by the target of an invocation.
Results of search: FollowersOf with: https://twitter.com/porn_xxx
Handle results: 0 nextstep: 2/2
End of chain: Store results: 0
Perform custom search: #chain/total: 1/2 using: NancyLe99615972
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Results of search: A User Id or Url with: https://twitter.com/porn_xxx
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/porn_xxx
Perform custom search: #chain/total: 2/2 using: NancyLe99615972
Start search: FollowersOf with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Get TwitterUser: porn_xxx using: NancyLe99615972
* ERROR: TwitterErrorLog: Exception has been thrown by the target of an invocation.
* ERROR: handleAResult: Term: https://twitter.com/porn_xxx Exception has been thrown by the target of an invocation.
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercompornxxx_2018-05-31_2.txt
Started: 31/05/2018 02:03 AM
Finished: 31/05/2018 02:04 AM
ID: de9ff4af-4f66-420f-82d7-139a04111edb
Action ran for: 0hr:0min:18s
I think i missed something similar to the verify accounts and cause all the problem.

Image

Rgds.
support@rootjazz
Posts: 2317
Joined: Tue Feb 02, 2016 1:58 pm

Re: Please tell me how to scrape users

Post by support@rootjazz »

Danny000 wrote: Thu May 31, 2018 9:51 am
support@rootjazz wrote: Wed May 30, 2018 5:09 am
To make a full description of this issue, I re-install the system, re-install TD, everything start over again.

OS: windows server 2012 R2 (Fresh and clean)
TD: v1.434 (Fresh installed)

Log submitted: 60744

Only one account input, it's PVA and has all necessary API things, this account should be all good.

I use its own API thing and I did all the procedure and record here, maybe you can find something:

Setup API key, Athorize Accounts, API APP setting unlock, Fix key permission, they all succeeded, I'll skip the realtime logs here.

But the verify accounts is weird. API login is specified successful (inside the log) , but the verify accounts job was tagged Failed.
Realtime log here:
Starting: 31/05/2018 02:07 AM
Validate account via login NancyLexxxxx sofya.grushnikova.1994@mail.ru
* SUCCESS: NancyLe99615972 sofya.grushnikova.1994@mail.ru validated successfully
FAILED VERIFY:

SUCCESS VERIFY:
http://twitter.com/NancyLexxxxx
1 accounts validated out of 1
* ERROR: The given key was not present in the dictionary.
* UNKNOWN ERROR OCCURRED - see logs for more info.
If the problem continues, please submit logs to support.
The given key was not present in the dictionary.

The given key was not present in the dictionary.
And also the custom search failed.

Custom search Setting:
A user url or ID
Followers of

realtime log:
Starting: 31/05/2018 02:03 AM
Perform custom search
Max Results Wanted: 5000
Setup custom search controller
Custom search run: search: https://twitter.com/porn_xxx
Perform custom search: #chain/total: 1/2 using: NancyLe99615972
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Results of search: A User Id or Url with: https://twitter.com/porn_xxx
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/porn_xxx
Perform custom search: #chain/total: 2/2 using: NancyLe99615972
Start search: FollowersOf with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Get TwitterUser: porn_xxx using: NancyLe99615972
Got user: https://twitter.com/porn_xxx username: Gay sex porn videos #followers: 113227
* ERROR: TwitterErrorLog: Exception has been thrown by the target of an invocation.
* ERROR: Exception has been thrown by the target of an invocation.
Results of search: FollowersOf with: https://twitter.com/porn_xxx
Handle results: 0 nextstep: 2/2
End of chain: Store results: 0
Perform custom search: #chain/total: 1/2 using: NancyLe99615972
Next search step returns single item, can return from this step
Start search: A User Id or Url with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Results of search: A User Id or Url with: https://twitter.com/porn_xxx
Handle results: 1 nextstep: 1/2
Got results from chain: 1 feed into next chained custom search item
Perform chained search: [1] https://twitter.com/porn_xxx
Perform custom search: #chain/total: 2/2 using: NancyLe99615972
Start search: FollowersOf with: https://twitter.com/porn_xxx(5000) using: NancyLe99615972
Get TwitterUser: porn_xxx using: NancyLe99615972
* ERROR: TwitterErrorLog: Exception has been thrown by the target of an invocation.
* ERROR: handleAResult: Term: https://twitter.com/porn_xxx Exception has been thrown by the target of an invocation.
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\search_httpstwittercompornxxx_2018-05-31_2.txt
Started: 31/05/2018 02:03 AM
Finished: 31/05/2018 02:04 AM
ID: de9ff4af-4f66-420f-82d7-139a04111edb
Action ran for: 0hr:0min:18s
I think i missed something similar to the verify accounts and cause all the problem.

Image

Rgds.
Twitters api had some issues last 24 hours of more than that.
Can you please check the action again?
I think the issue should be solved if you update to the latest TD version.

Regards
Danny000
Posts: 130
Joined: Mon Jan 15, 2018 12:19 am

Re: Please tell me how to scrape users

Post by Danny000 »

support@rootjazz wrote: Fri Jun 01, 2018 2:47 am
No, verify accounts still failed.
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Please tell me how to scrape users

Post by martin@rootjazz »

Danny000 wrote: Sat Jun 02, 2018 1:13 pm
support@rootjazz wrote: Fri Jun 01, 2018 2:47 am
No, verify accounts still failed.
The last logs you sent were definitely the Twitter API issue
* ERROR: TwitterErrorLog: Exception has been thrown by the target of an invocation.
Can you regenerate me the logs so I can see the issue without this API issue. Sometimes can be hard to see your actual issue, when there are other issues as well polluting the logs.



Please perform the following:

Open the program. Delete or pause all existing actions. If actions are already running due to autorun on startup, disable autorun at startup and restart the app. All actions must be waiting, then pause all (or delete)


Delete existing logs
HELP > LOGS > VIEW
delete all files in the folder that opens


Perform the action:
***
run a test verify issue with just 1 account to cause your issue
submit logs

***

PLEASE NOTE: Do not perform other actions at the same time. Only use one thread. Do not run any actions after this action.


Then submit logs:
HELP > LOGS > SUBMIT

and let me know the LOGS ID (just the number part is sufficient)




Regards,
Martin
Danny000
Posts: 130
Joined: Mon Jan 15, 2018 12:19 am

Re: Please tell me how to scrape users

Post by Danny000 »

martin@rootjazz wrote: Sat Jun 02, 2018 2:09 pm
Log submitted again: 76352

Realtime log:
Starting: 02/06/2018 15:47 PM
Validate account via login NancyLe99615972 sofya.grushnikova.1994@mail.ru
* SUCCESS: NancyLe99615972 sofya.grushnikova.1994@mail.ru validated successfully
FAILED VERIFY:

SUCCESS VERIFY:
http://twitter.com/NancyLe99615972
1 accounts validated out of 1
* ERROR: The given key was not present in the dictionary.
* UNKNOWN ERROR OCCURRED - see logs for more info.
If the problem continues, please submit logs to support.
The given key was not present in the dictionary.

The given key was not present in the dictionary.
User avatar
martin@rootjazz
Site Admin
Posts: 34375
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Please tell me how to scrape users

Post by martin@rootjazz »

sorry this wasn't noticed before but you on using an old version of the software (by about 3 weeks)

https://rootjazz.com/twitterdub/updatetesting.html

I have a feeling this issue has already been fixed
Post Reply