How to Scrape "Aged" or "old" or "date range" notes

TumblingJazz Tumblr bot discussion
tahoe012
Posts: 107
Joined: Mon Jan 06, 2020 10:56 am

Re: How to Scrape "Aged" or "old" or "date range" notes

Post by tahoe012 »

Hi Martin,

Downloaded from the Link in your reply
Same error message

On my Laptop working fine.
On the Windows VPS is the Error

Thanks for looking into this.
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: How to Scrape "Aged" or "old" or "date range" notes

Post by martin@rootjazz »

tahoe012 wrote: Sat Oct 10, 2020 11:49 am Hi Martin,

Downloaded from the Link in your reply
Same error message

On my Laptop working fine.
On the Windows VPS is the Error

Thanks for looking into this.
hmm strange. Can you submit logs from the latest version from the machine not working please
tahoe012
Posts: 107
Joined: Mon Jan 06, 2020 10:56 am

Re: How to Scrape "Aged" or "old" or "date range" notes

Post by tahoe012 »

Yes, Thank you

53310
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: How to Scrape "Aged" or "old" or "date range" notes

Post by martin@rootjazz »

tahoe012 wrote: Sun Oct 11, 2020 1:34 am Yes, Thank you

53310
Proxy not working


The HTTP 407 Proxy Authentication Required client error status response code indicates that the request has not been applied because it lacks valid authentication credentials for a proxy server that is between the browser and the server that can access the requested resource.Mar 23, 2019

407 Proxy Authentication Required - MDN - Mozilla


The proxy isn't allow on this machine - speak to your proxy provider



Regards,
Martin
tahoe012
Posts: 107
Joined: Mon Jan 06, 2020 10:56 am

Re: How to Scrape "Aged" or "old" or "date range" notes

Post by tahoe012 »

Martin,

When scraping with; Scrape Posts; Scrape Term of Filepath: Posts Popular. I tried putting in a text file with large list of keywords. 1000 keywords. This did not seem to work. We can only put one keyword at a time?
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: How to Scrape "Aged" or "old" or "date range" notes

Post by martin@rootjazz »

tahoe012 wrote: Wed Oct 14, 2020 8:46 am Martin,

When scraping with; Scrape Posts; Scrape Term of Filepath: Posts Popular. I tried putting in a text file with large list of keywords. 1000 keywords. This did not seem to work. We can only put one keyword at a time?
Should work, can you screenshot what you are doing exactly please
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: How to Scrape "Aged" or "old" or "date range" notes

Post by martin@rootjazz »

tahoe012 wrote: Wed Oct 14, 2020 8:46 am
When scraping with; Scrape Posts; Scrape Term of Filepath: Posts Popular. I tried putting in a text file with large list of keywords. 1000 keywords. This did not seem to work. We can only put one keyword at a time?
Working fine here. It will create one action per line in your file. So for 1000 lines it could take a while, but not *that* long
tahoe012
Posts: 107
Joined: Mon Jan 06, 2020 10:56 am

Re: How to Scrape "Aged" or "old" or "date range" notes

Post by tahoe012 »

Ok I see how that works I used 5 random keywords and it quickly created 5 processes and in the log file are 5 text files with hundreds of blog posts to use for scraping the notes.

I have one Note Scrape going right now and it happened upon a blog with 250,000 notes.. i'll keep this running as it runs thru various blogs till we get to about 1M notes. Is there a way to pause the project and it will restart where it left off? I would not want it to rescrape those 250,000 notes from one blog.
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: How to Scrape "Aged" or "old" or "date range" notes

Post by martin@rootjazz »

tahoe012 wrote: Thu Oct 15, 2020 8:00 am Is there a way to pause the project and it will restart where it left off? I would not want it to rescrape those 250,000 notes from one blog.

No, you need to let it run
tahoe012
Posts: 107
Joined: Mon Jan 06, 2020 10:56 am

Re: How to Scrape "Aged" or "old" or "date range" notes

Post by tahoe012 »

I think I did not do something right. The screen shot shows the set up. The file was a list of Posts scraped from keyword dog.
The objective is to scrape the posts for Notes than run that list of notes thru availability checker.
I was hoping for 1 Million notes and ended up with 1 Million posts.

https://ibb.co/tsq7ctX
24348 are the logs
Post Reply