Scraping profile error

Support / help / discussion forum for twitter bot
aritwitter
Posts: 57
Joined: Tue Jun 20, 2023 1:09 pm

Scraping profile error

Post by aritwitter »

Hello,

I've been encountering some error when scraping accounts for details. I want to get the name, loc, bio/description, number of followers, number of followings, date of creation. I have previously done this and got the right output but now I am only getting this:

screenname url name location description profile_url followers_count friends_count statuses_count urls location created_date
https://twitter.com/account1
https://twitter.com/account2
https://twitter.com/account3

As shown above, there are columns for the details but those columns are blank.

For input, I have a text file containing the url of the accounts I want to scrape:
https://twitter.com/account1
https://twitter.com/account2
https://twitter.com/account3

For the custom search I chose 'A user ID or URL' then I selected 'details' so it can be saved in a CSV.

Please let me know if I'm doing it right and why I am not getting the right output.
User avatar
martin@rootjazz
Site Admin
Posts: 35135
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping profile error

Post by martin@rootjazz »

Let me know exactly how you are setting up the action. Instruction set to replicate exactly or screenshots be helpful

Then submit logs showing the failed action

HELP > LOGS > SUBMIT

then send your logs ID - the numbers are sufficient (displayed after successful uploading of logs)

viewtopic.php?f=32&t=12078



Regards,
Martin
aritwitter
Posts: 57
Joined: Tue Jun 20, 2023 1:09 pm

Re: Scraping profile error

Post by aritwitter »

Sorry, my bad. it's not an error per se. But the result of the scrape is not what is expected. The content of the txt file didn't have data for the other columns, it only has the url of the account and nothing else which is what I sent last time:

screenname url name location description profile_url followers_count friends_count statuses_count urls location created_date
https://twitter.com/account1
https://twitter.com/account2
https://twitter.com/account3

I was expecting that there will be data under each column.

Here's the steps of how I did it:

1. clicked scrape tab
2. Selected the account which will do the scraping process
3. Clicked custom search chose "A user Id or url' for option then "A user Id or url' again under selections
4. Input the filepath of the text file containing the url of the accounts I want to scrape
5. Selected 'Details' to save result to a text file
6. Then lastly clicked "scrape profile"

log: 32009
User avatar
martin@rootjazz
Site Admin
Posts: 35135
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping profile error

Post by martin@rootjazz »

aritwitter wrote: Tue Sep 12, 2023 10:55 am
log: 32009
thanks that's what I needed, testing...
User avatar
martin@rootjazz
Site Admin
Posts: 35135
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping profile error

Post by martin@rootjazz »

Tested, it's working here for normal searches
aritwitter wrote: Tue Sep 12, 2023 10:55 am 3. Clicked custom search chose "A user Id or url' for option then "A user Id or url' again under selections
4. Input the filepath of the text file containing the url of the accounts I want to scrape
But I think this is what the issue is...

yes confirmed, this is the issue.

The next update will fix this. I shall let you know when it is ready.



Regards,
Martin
User avatar
martin@rootjazz
Site Admin
Posts: 35135
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping profile error

Post by martin@rootjazz »

aritwitter
Posts: 57
Joined: Tue Jun 20, 2023 1:09 pm

Re: Scraping profile error

Post by aritwitter »

Hi, I'm getting another error. It says proxy authentication required. Then proxy issue detected: 407

Let me know what this is all about and how I can fix it.

Log: 19736
User avatar
martin@rootjazz
Site Admin
Posts: 35135
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping profile error

Post by martin@rootjazz »

aritwitter wrote: Mon Sep 18, 2023 10:36 am Hi, I'm getting another error. It says proxy authentication required. Then proxy issue detected: 407

Let me know what this is all about and how I can fix it.

Log: 19736
you need to authorise the proxy, speak to your proxy provider
aritwitter
Posts: 57
Joined: Tue Jun 20, 2023 1:09 pm

Re: Scraping profile error

Post by aritwitter »

The issue with proxy has already been resolved, thanks!

As for the error in the scraping output, it seems like I am still getting the same result, just the url, even after updating the app.

Log: 37781
User avatar
martin@rootjazz
Site Admin
Posts: 35135
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping profile error

Post by martin@rootjazz »

aritwitter wrote: Tue Sep 19, 2023 12:56 am
As for the error in the scraping output, it seems like I am still getting the same result, just the url, even after updating the app.

Log: 37781
It's working here (for a new action I just tested)

What version are you on?

HELP >ABOUT

Might have been a build issue, not pulling in the update. I Will do a new build and let you know when ready for testin



Regards,
Martin
Post Reply