Scraping

Support / help / discussion forum for twitter bot
Post Reply
leon
Posts: 89
Joined: Mon Aug 09, 2021 1:13 pm

Scraping

Post by leon »

Hi,

Scraping takes way to long! Scraped 5k Account's in 10h ..
How to make it faster?
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping

Post by martin@rootjazz »

What are you doing exactly?
How you doing it?
What searches?
Any filters (this will slow things down alot, as you need 1 request per result and if you filter 90% of results, you will need to test a lot more results to get what you want)

and submit app logs showing your action with the issue

HELP > LOGS > SUBMIT

then send your logs ID - the numbers are sufficient (displayed after successful uploading of logs)





Regards,
Martin
leon
Posts: 89
Joined: Mon Aug 09, 2021 1:13 pm

Re: Scraping

Post by leon »

I want to filter people who have DM enabled.
leon
Posts: 89
Joined: Mon Aug 09, 2021 1:13 pm

Re: Scraping

Post by leon »

I have one filter enabled (DM) I get only ~100 Results per Hour no matter how many accounts i'am using.
Is there a way to make it faster. Scraping Followers Only is extremly fast! Please Help asap!

Best Regards
User avatar
martin@rootjazz
Site Admin
Posts: 34345
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scraping

Post by martin@rootjazz »

sircrocodile wrote: Tue Oct 19, 2021 6:24 pm I have one filter enabled (DM) I get only ~100 Results per Hour no matter how many accounts i'am using.
Is there a way to make it faster. Scraping Followers Only is extremly fast! Please Help asap!

Best Regards
If you are using a filter, rather than being able to scrape 100 results with 1 request, the program now needs to perform an additional request PER result, so 101 requests for 100 results, this takes time.

And the results, will depend on how many have DM enabled, so it could be 50% 90% or 1% depending on your search.

The program does need to pause and take it's time, otherwise twitter will block you for scraping too many results.

There is a custom SEARCH PAUSE on the SETTINGS tab, which you can deactivate, which is an additional pause, the other pauses are forced as are required to keep the account running smoothly




Regards,
Martin
leon
Posts: 89
Joined: Mon Aug 09, 2021 1:13 pm

Re: Scraping

Post by leon »

Thank You
Post Reply