Hi,
Scraping takes way to long! Scraped 5k Account's in 10h ..
How to make it faster?
Scraping
- martin@rootjazz
- Site Admin
- Posts: 34375
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scraping
What are you doing exactly?
How you doing it?
What searches?
Any filters (this will slow things down alot, as you need 1 request per result and if you filter 90% of results, you will need to test a lot more results to get what you want)
and submit app logs showing your action with the issue
HELP > LOGS > SUBMIT
then send your logs ID - the numbers are sufficient (displayed after successful uploading of logs)
Regards,
Martin
How you doing it?
What searches?
Any filters (this will slow things down alot, as you need 1 request per result and if you filter 90% of results, you will need to test a lot more results to get what you want)
and submit app logs showing your action with the issue
HELP > LOGS > SUBMIT
then send your logs ID - the numbers are sufficient (displayed after successful uploading of logs)
Regards,
Martin
Re: Scraping
I want to filter people who have DM enabled.
Re: Scraping
I have one filter enabled (DM) I get only ~100 Results per Hour no matter how many accounts i'am using.
Is there a way to make it faster. Scraping Followers Only is extremly fast! Please Help asap!
Best Regards
Is there a way to make it faster. Scraping Followers Only is extremly fast! Please Help asap!
Best Regards
- martin@rootjazz
- Site Admin
- Posts: 34375
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: Scraping
If you are using a filter, rather than being able to scrape 100 results with 1 request, the program now needs to perform an additional request PER result, so 101 requests for 100 results, this takes time.sircrocodile wrote: ↑Tue Oct 19, 2021 6:24 pm I have one filter enabled (DM) I get only ~100 Results per Hour no matter how many accounts i'am using.
Is there a way to make it faster. Scraping Followers Only is extremly fast! Please Help asap!
Best Regards
And the results, will depend on how many have DM enabled, so it could be 50% 90% or 1% depending on your search.
The program does need to pause and take it's time, otherwise twitter will block you for scraping too many results.
There is a custom SEARCH PAUSE on the SETTINGS tab, which you can deactivate, which is an additional pause, the other pauses are forced as are required to keep the account running smoothly
Regards,
Martin