Scrape issue

Support / help / discussion forum for twitter bot
Danny000
Posts: 130
Joined: Mon Jan 15, 2018 12:19 am

Re: Scrape issue

Post by Danny000 »

Same proxy also used in TumblingJazz, but everytime TJ save results before task failed.

Could you make this little change to TD either?

Or change the way it saves file, like saveing logs, save result to txt file wile scraping.





martin@rootjazz wrote: Wed Sep 05, 2018 2:19 pm in your logs, there are 100s of fail requests logged
http://<proxy ip>/
<html><head><title>502 Bad Gateway</title></head>
<body><h2>502 Bad Gateway</h2><h3>Host Not Found or connection failed</h3></body></html>
If your proxies are not working, the program cannot function and will drop out as shown above
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape issue

Post by martin@rootjazz »

Danny000 wrote: Thu Sep 06, 2018 3:01 am Same proxy also used in TumblingJazz, but everytime TJ save results before task failed.

Could you make this little change to TD either?

Or change the way it saves file, like saveing logs, save result to txt file wile scraping.





martin@rootjazz wrote: Wed Sep 05, 2018 2:19 pm in your logs, there are 100s of fail requests logged
http://<proxy ip>/
<html><head><title>502 Bad Gateway</title></head>
<body><h2>502 Bad Gateway</h2><h3>Host Not Found or connection failed</h3></body></html>
If your proxies are not working, the program cannot function and will drop out as shown above
TD already saves results of searching as it searches. Look in the HELP >SAVED folder for the files named "insearch_..."

But it is still advised to contact your proxy provider, because the proxies are often failing
Danny000
Posts: 130
Joined: Mon Jan 15, 2018 12:19 am

Re: Scrape issue

Post by Danny000 »

martin@rootjazz wrote: Thu Sep 06, 2018 1:23 pm TD already saves results of searching as it searches. Look in the HELP >SAVED folder for the files named "insearch_..."
But it is still advised to contact your proxy provider, because the proxies are often failing
Seems not save data if failed.

This is the realtime log of failed job after hours' work:
Errors in row: 5
* ERROR: Object reference not set to an instance of an object.
Results: 0
Saving to: C:\Users\Administrator\AppData\Roaming\rootjazz\Twitterdub\saved_data\insearch_httpstwittercomhottychix_2018-09-05.txt
Started: 05/09/2018 00:50 AM
Finished: 05/09/2018 04:38 AM
ID: 1d96dee5-cd98-4bf0-b6ba-20ae00dfaece
Action ran for: 3hr:47min:19s
And this is the file saved, only one succeed out of many tasks.

Image
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: Scrape issue

Post by martin@rootjazz »

What version are you on?
HELP > ABOUT

you need to be on at least
2.029 - 04/09/2018
better detection of deserialisation issues
Scraper actions will save out during processing
If you are on that version or higher, what search are you running as for me it is working correctly
Post Reply