How to scrape all posts from a blog?

TumblingJazz Tumblr bot discussion
Post Reply
botler
Posts: 7
Joined: Tue Feb 28, 2017 10:39 pm

How to scrape all posts from a blog?

Post by botler »

How does this works i tried 'recent post of' via custom search but i can get only 21 posts per blog?

so how to scrape maybe all post of one blog via url or username ???

thx in advance :)

Code: Select all

Starting: 06.03.2017 21:09 nachm.
Custom posts search: http://www.alexmdc.tumblr.com
Max Results Wanted: 200
Setup custom search controller
One search stage detected, setting default per item per stage value to max: 200
Custom search run: search: http://www.alexmdc.tumblr.com
Perform custom search: #chain/total: 1/1 using: ...
Start search: Recent Posts Of with: http://www.alexmdc.tumblr.com using: ...
* RSS scrape
RSS URL :http://www.alexmdc.tumblr.com/rss
Found: 21 urls
Results of search: Recent Posts Of with: http://www.alexmdc.tumblr.com
Handle results: 21 nextstep: 1/1
End of chain: Store results: 21
 Save results: 21
Saved: 21 to C:\...\AppData\Roaming\rootjazz\Tumbling Jazz\saved\scraped_custom_posts_httpwwwalexmdctumblrcom_2017-03-06_2.txt
Started: 06.03.2017 21:09 nachm.
Finished: 06.03.2017 21:10 nachm.
ID: ...1
Action ran for: 0hr:0min:4s
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: How to scrape all posts from a blog?

Post by martin@rootjazz »

goto the settings tab
there should be an option to specify how to scrape posts from a blog

RSS
ARCHIVE

select archive. RSS may be a bit outdated and won't give all posts
botler
Posts: 7
Joined: Tue Feb 28, 2017 10:39 pm

Re: How to scrape all posts from a blog?

Post by botler »

Image
botler
Posts: 7
Joined: Tue Feb 28, 2017 10:39 pm

Re: How to scrape all posts from a blog?

Post by botler »

this is how my setup looks ?
User avatar
martin@rootjazz
Site Admin
Posts: 34360
Joined: Fri Jan 25, 2013 10:06 pm
Location: The Funk
Contact:

Re: How to scrape all posts from a blog?

Post by martin@rootjazz »

so, as I said select ARCHIVE

you have selected RSS

:)
Post Reply