Bot not scraping more than 5000 data?

Support / help / discussion forum for twitter bot
Post Reply
chupakabra
Posts: 128
Joined: Sun Feb 09, 2020 3:29 pm

Bot not scraping more than 5000 data?

Post by chupakabra »

Hi Admin,

Here is the submit log code: 38654

Here is the screenshot: https://i.imgur.com/CL6YGYF.png

This is not a new issue I am facing with but I know I have not asked before - but I want to increase the limit far beyond 5000 scraping data.

How do I do that? The reason I want to scrap lakhs of data is to analyse it and set it for another tasks like retweet, follow, like, etc.

I need to increase the limit to unlimited. Where is the option or what is the method?
chupakabra
Posts: 128
Joined: Sun Feb 09, 2020 3:29 pm

Re: Bot not scraping more than 5000 data?

Post by chupakabra »

I just learnt this: viewtopic.php?f=30&t=9641&p=62031&hilit=5000#p62031

Is that true? Let me know.
User avatar
martin@rootjazz
Site Admin
Posts: 34352
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Bot not scraping more than 5000 data?

Post by martin@rootjazz »

chupakabra wrote: Fri Mar 26, 2021 5:39 am I just learnt this: viewtopic.php?f=30&t=9641&p=62031&hilit=5000#p62031

Is that true? Let me know.
No idea. But from your logs screenshot cannot tell.
What limit did you set for the search?
What search is it?
What is your MAX SEARCH on the SETTINGS tab?
chupakabra
Posts: 128
Joined: Sun Feb 09, 2020 3:29 pm

Re: Bot not scraping more than 5000 data?

Post by chupakabra »

Oh I just found that my Max Scrape is at 5000 under settings. SO if I increase it to like 50000 it will scrape 50k results?
User avatar
martin@rootjazz
Site Admin
Posts: 34352
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Bot not scraping more than 5000 data?

Post by martin@rootjazz »

chupakabra wrote: Sun Mar 28, 2021 6:04 am Oh I just found that my Max Scrape is at 5000 under settings. SO if I increase it to like 50000 it will scrape 50k results?
Maybe. Twitter may have a limit it will not serve any more results for. I've haven't tried to scrape 50k results for a long time, so cannot say. But set the value as high as you want, and you will get all the results twitter is prepared to send you


Regards,
Martin
Post Reply