Scrapping Avatar + Banner on TwitterDub

Ask any support / help / issues / problem or question related to any of our other products
Post Reply
its2l82die
Posts: 32
Joined: Fri Jul 29, 2022 2:24 am

Scrapping Avatar + Banner on TwitterDub

Post by its2l82die »

Hey first time posting, happy to be here.

I've recently tried scrapping avatar and banner images using the avatar scraper and I was getting error response from twitter it seems,
Failed JSON response: 403
https://twitter.com/i/api/1.1/users/loo ... ilosopheRR
{"errors":[{"code":18,"message":"Too many terms specified in query."}]}
ERROR RESP FROM TWITTER: 18 Too many terms specified in query.
* TESTING for known failure text: Too many terms specified in query.
* FAILED: 92374087236487692374627 : Too many terms specified in query.
after that it indicated progress failed.
did twitter do some changes to the avatar or banner urls?
also wanted to know is this the only way to scrape avatar and banner images?
its2l82die
Posts: 32
Joined: Fri Jul 29, 2022 2:24 am

Re: Scrapping Avatar + Banner on TwitterDub

Post by its2l82die »

I think i get what is going on here, it tries to send to Twitter api all the scraped IDs at one query is that it? Should it be split somehow to allow the api to fetch the avatars and banners one by one? Not really sure how to do that though...
its2l82die
Posts: 32
Joined: Fri Jul 29, 2022 2:24 am

Re: Scrapping Avatar + Banner on TwitterDub

Post by its2l82die »

SOLVED
I've tried it again with limit on followersOf to 5 and the scrapping succeeded this time around.
Guess as the api error stated too many requests, meant too many requests in one call...

Just documenting my results for others if they encounter the same, or for my owner reference in case I forget.
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrapping Avatar + Banner on TwitterDub

Post by martin@rootjazz »

its2l82die wrote: Fri Jul 29, 2022 3:57 pm I think i get what is going on here, it tries to send to Twitter api all the scraped IDs at one query is that it? Should it be split somehow to allow the api to fetch the avatars and banners one by one? Not really sure how to do that though...
yes seems, seems the request should be page.

Can you just explain the search you were running that triggered this bug. I'll try and cause it now, but if you have instructions on how to make it fail, might save me some time
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrapping Avatar + Banner on TwitterDub

Post by martin@rootjazz »

never mind, figured it out. twitter changed how long the URL can be to request user details in a single request.

The next update will fix this. I shall let you know when it is ready.



Regards,
Martin
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrapping Avatar + Banner on TwitterDub

Post by martin@rootjazz »

its2l82die
Posts: 32
Joined: Fri Jul 29, 2022 2:24 am

Re: Scrapping Avatar + Banner on TwitterDub

Post by its2l82die »

Thank you for the update Martin really appreciate it
Post Reply