faster scraping?

[bmsietta]

hello,
I'm trying to do faster scraping and would like to know if it's possible to safely run 50 threads on the manager.
I have a list with 4k keywords that I wanna target, and that usually takes 3 days to do if I'm scraping only for the last day. (one thread only)

if I manually split the file in 50 and then use the manager on 50 threads will I run into problems? can I do it with my IP or do I need proxies for this?
I know I could try it instead of asking but, don't wanna risk my IP reputation.

I'm trying to scrape users from the "scrape tracks/term or path" option.

Bmsi Etta

[martin@rootjazz]

hello,
I'm trying to do faster scraping and would like to know if it's possible to safely run 50 threads on the manager.
I have a list with 4k keywords that I wanna target, and that usually takes 3 days to do if I'm scraping only for the last day. (one thread only)

if I manually split the file in 50 and then use the manager on 50 threads will I run into problems? can I do it with my IP

Doing it on one IP is risky, that is quite a lot of searching to do. Maybe try and use threads, but more like 10 rather than 50.

or do I need proxies for this?

That would be safest.

I know I could try it instead of asking but, don't wanna risk my IP reputation.

If risking your IP reputation, do not do it. No one can say for sure, unless they have tried it. And even then, there are lots of parameters that may not apply to your situation. So for you working proxies would be the best option.

[bmsietta]

I'll be doing this with 5 HTTP proxies and 10 threads, sounds safe and 10 times faster than I've been doing it.
Just to confirm: the syntax is IP,port,user,pass correct?

Thanks a lot Martin, your software is so worth the money.

[martin@rootjazz]

I'll be doing this with 5 HTTP proxies and 10 threads, sounds safe and 10 times faster than I've been doing it.

IF the proxy company lets you change IPs, you can just blast the hell out of them*, if they get banned, just change them up. As there is no risk to your local IP / account IP it doesn't really matter.

*not actually recommended, just to point out, you have no actual risk to your local / account IP, so you could if you want

Just to confirm: the syntax is IP,port,user,pass correct?

No

IP:PORT
or
IP:PORT:UN:PW

not the separator is: not ,

Thanks a lot Martin, your software is so worth the money.