Scraping profile error
-
- Posts: 57
- Joined: Tue Jun 20, 2023 1:09 pm
Scraping profile error
Hello,
I've been encountering some error when scraping accounts for details. I want to get the name, loc, bio/description, number of followers, number of followings, date of creation. I have previously done this and got the right output but now I am only getting this:
screenname url name location description profile_url followers_count friends_count statuses_count urls location created_date
https://twitter.com/account1
https://twitter.com/account2
https://twitter.com/account3
As shown above, there are columns for the details but those columns are blank.
For input, I have a text file containing the url of the accounts I want to scrape:
https://twitter.com/account1
https://twitter.com/account2
https://twitter.com/account3
For the custom search I chose 'A user ID or URL' then I selected 'details' so it can be saved in a CSV.
Please let me know if I'm doing it right and why I am not getting the right output.
I've been encountering some error when scraping accounts for details. I want to get the name, loc, bio/description, number of followers, number of followings, date of creation. I have previously done this and got the right output but now I am only getting this:
screenname url name location description profile_url followers_count friends_count statuses_count urls location created_date
https://twitter.com/account1
https://twitter.com/account2
https://twitter.com/account3
As shown above, there are columns for the details but those columns are blank.
For input, I have a text file containing the url of the accounts I want to scrape:
https://twitter.com/account1
https://twitter.com/account2
https://twitter.com/account3
For the custom search I chose 'A user ID or URL' then I selected 'details' so it can be saved in a CSV.
Please let me know if I'm doing it right and why I am not getting the right output.
- martin@rootjazz
- Site Admin
- Posts: 35411
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scraping profile error
Let me know exactly how you are setting up the action. Instruction set to replicate exactly or screenshots be helpful
Then submit logs showing the failed action
HELP > LOGS > SUBMIT
then send your logs ID - the numbers are sufficient (displayed after successful uploading of logs)
viewtopic.php?f=32&t=12078
Regards,
Martin
Then submit logs showing the failed action
HELP > LOGS > SUBMIT
then send your logs ID - the numbers are sufficient (displayed after successful uploading of logs)
viewtopic.php?f=32&t=12078
Regards,
Martin
-
- Posts: 57
- Joined: Tue Jun 20, 2023 1:09 pm
Re: Scraping profile error
Sorry, my bad. it's not an error per se. But the result of the scrape is not what is expected. The content of the txt file didn't have data for the other columns, it only has the url of the account and nothing else which is what I sent last time:
screenname url name location description profile_url followers_count friends_count statuses_count urls location created_date
https://twitter.com/account1
https://twitter.com/account2
https://twitter.com/account3
I was expecting that there will be data under each column.
Here's the steps of how I did it:
1. clicked scrape tab
2. Selected the account which will do the scraping process
3. Clicked custom search chose "A user Id or url' for option then "A user Id or url' again under selections
4. Input the filepath of the text file containing the url of the accounts I want to scrape
5. Selected 'Details' to save result to a text file
6. Then lastly clicked "scrape profile"
log: 32009
screenname url name location description profile_url followers_count friends_count statuses_count urls location created_date
https://twitter.com/account1
https://twitter.com/account2
https://twitter.com/account3
I was expecting that there will be data under each column.
Here's the steps of how I did it:
1. clicked scrape tab
2. Selected the account which will do the scraping process
3. Clicked custom search chose "A user Id or url' for option then "A user Id or url' again under selections
4. Input the filepath of the text file containing the url of the accounts I want to scrape
5. Selected 'Details' to save result to a text file
6. Then lastly clicked "scrape profile"
log: 32009
- martin@rootjazz
- Site Admin
- Posts: 35411
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scraping profile error
thanks that's what I needed, testing...
- martin@rootjazz
- Site Admin
- Posts: 35411
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scraping profile error
Tested, it's working here for normal searches
yes confirmed, this is the issue.
The next update will fix this. I shall let you know when it is ready.
Regards,
Martin
But I think this is what the issue is...aritwitter wrote: ↑Tue Sep 12, 2023 10:55 am 3. Clicked custom search chose "A user Id or url' for option then "A user Id or url' again under selections
4. Input the filepath of the text file containing the url of the accounts I want to scrape
yes confirmed, this is the issue.
The next update will fix this. I shall let you know when it is ready.
Regards,
Martin
- martin@rootjazz
- Site Admin
- Posts: 35411
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
-
- Posts: 57
- Joined: Tue Jun 20, 2023 1:09 pm
Re: Scraping profile error
Hi, I'm getting another error. It says proxy authentication required. Then proxy issue detected: 407
Let me know what this is all about and how I can fix it.
Log: 19736
Let me know what this is all about and how I can fix it.
Log: 19736
- martin@rootjazz
- Site Admin
- Posts: 35411
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scraping profile error
you need to authorise the proxy, speak to your proxy provideraritwitter wrote: ↑Mon Sep 18, 2023 10:36 am Hi, I'm getting another error. It says proxy authentication required. Then proxy issue detected: 407
Let me know what this is all about and how I can fix it.
Log: 19736
-
- Posts: 57
- Joined: Tue Jun 20, 2023 1:09 pm
Re: Scraping profile error
The issue with proxy has already been resolved, thanks!
As for the error in the scraping output, it seems like I am still getting the same result, just the url, even after updating the app.
Log: 37781
As for the error in the scraping output, it seems like I am still getting the same result, just the url, even after updating the app.
Log: 37781
- martin@rootjazz
- Site Admin
- Posts: 35411
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scraping profile error
It's working here (for a new action I just tested)aritwitter wrote: ↑Tue Sep 19, 2023 12:56 am
As for the error in the scraping output, it seems like I am still getting the same result, just the url, even after updating the app.
Log: 37781
What version are you on?
HELP >ABOUT
Might have been a build issue, not pulling in the update. I Will do a new build and let you know when ready for testin
Regards,
Martin