hello,
I'm trying to do faster scraping and would like to know if it's possible to safely run 50 threads on the manager.
I have a list with 4k keywords that I wanna target, and that usually takes 3 days to do if I'm scraping only for the last day. (one thread only)
if I manually split the file in 50 and then use the manager on 50 threads will I run into problems? can I do it with my IP or do I need proxies for this?
I know I could try it instead of asking but, don't wanna risk my IP reputation.
I'm trying to scrape users from the "scrape tracks/term or path" option.
Bmsi Etta
faster scraping?
- martin@rootjazz
- Site Admin
- Posts: 34634
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: faster scraping?
Doing it on one IP is risky, that is quite a lot of searching to do. Maybe try and use threads, but more like 10 rather than 50.bmsietta wrote: ↑Wed Mar 04, 2020 11:40 pm hello,
I'm trying to do faster scraping and would like to know if it's possible to safely run 50 threads on the manager.
I have a list with 4k keywords that I wanna target, and that usually takes 3 days to do if I'm scraping only for the last day. (one thread only)
if I manually split the file in 50 and then use the manager on 50 threads will I run into problems? can I do it with my IP
That would be safest.or do I need proxies for this?
If risking your IP reputation, do not do it. No one can say for sure, unless they have tried it. And even then, there are lots of parameters that may not apply to your situation. So for you working proxies would be the best option.I know I could try it instead of asking but, don't wanna risk my IP reputation.
Re: faster scraping?
I'll be doing this with 5 HTTP proxies and 10 threads, sounds safe and 10 times faster than I've been doing it.
Just to confirm: the syntax is IP,port,user,pass correct?
Thanks a lot Martin, your software is so worth the money.
Just to confirm: the syntax is IP,port,user,pass correct?
Thanks a lot Martin, your software is so worth the money.
- martin@rootjazz
- Site Admin
- Posts: 34634
- Joined: Fri Jan 25, 2013 10:06 pm
- Location: The Funk
- Contact:
Re: faster scraping?
IF the proxy company lets you change IPs, you can just blast the hell out of them*, if they get banned, just change them up. As there is no risk to your local IP / account IP it doesn't really matter.
*not actually recommended, just to point out, you have no actual risk to your local / account IP, so you could if you want
NoJust to confirm: the syntax is IP,port,user,pass correct?
IP:PORT
or
IP:PORT:UN:PW
not the separator is: not ,
Thanks a lot Martin, your software is so worth the money.