一、介紹

WebMagic是一個簡單靈活的Java爬蟲框架。基于WebMagic，你可以快速開發出一個高效、易維護的爬蟲。

二、如何學習

1.查看官網

官網地址為:http://webmagic.io/官網詳細文檔:http://webmagic.io/docs/zh/

2.跑通hello world示例(具體可以參考官網，也可以參考博客)

我下面寫的單元測試案例，可作為Hello World示例。

注意需要導入Maven依賴:

<dependency> <groupId>us.codecraft</groupId> <artifactId>webmagic-core</artifactId> <version>0.7.3</version></dependency><dependency> <groupId>us.codecraft</groupId> <artifactId>webmagic-extension</artifactId> <version>0.7.3</version></dependency>

3.帶著一個目的

說說我的目的，最近我開發的博客系統，其中有個導入第三方博客的插件，這個插件比較簡單就是一個搜索框，在對應的搜索框里面填寫URL，點擊搜索即可導入到自己的博客。

以導入博客園單篇文章為例:

下面是我的源代碼(單篇文章導入,我已經將其封裝成一個工具類):

import cn.hutool.core.date.DateUtil;import com.blog.springboot.dto.CnBlogModelDTO;import com.blog.springboot.entity.Posts;import com.blog.springboot.service.PostsService;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.stereotype.Component;import us.codecraft.webmagic.Page;import us.codecraft.webmagic.Site;import us.codecraft.webmagic.Spider;import us.codecraft.webmagic.pipeline.ConsolePipeline;import us.codecraft.webmagic.processor.PageProcessor;import us.codecraft.webmagic.selector.Selectable;import javax.annotation.PostConstruct;/** * 導入博客園文章工具類 */@Componentpublic class WebMagicCnBlogUtils implements PageProcessor { @Autowired private PostsService postService; public static WebMagicCnBlogUtils magicCnBlogUtils; @PostConstruct public void init() { magicCnBlogUtils = this; magicCnBlogUtils.postService = this.postService; } private Site site = Site.me() .setDomain('https://www.cnblogs.com/') .setSleepTime(1000) .setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'); @Override public void process(Page page) { Selectable obj = page.getHtml().xpath('//div[@class=’post’]'); Selectable title = obj.xpath('//h1[@class=’postTitle’]//a'); Selectable content = obj.xpath('//div[@class=’blogpost-body’]'); System.out.println('title:' + title.replace('<[^>]*>', '')); System.out.println('content:' + content); CnBlogModelDTO blog = new CnBlogModelDTO(); blog.setTitle(title.toString()); blog.setContent(content.toString()); Posts post = new Posts(); String date = DateUtil.date().toString(); post.setPostAuthor(1L); post.setPostTitle(title.replace('<[^>]*>', '').toString()); post.setPostContent(content.toString()); post.setPostExcerpt(content.replace('<[^>]*>', '').toString()); post.setPostDate(date); post.setPostDate(date); post.setPostModified(date); boolean importPost = magicCnBlogUtils.postService.insert(post); if (importPost) { System.out.println('success'); } else { System.out.println('fail'); } } @Override public Site getSite() { return site; } /** * 導入單篇博客園文章數據 * * @param url */ public static void importSinglePost(String url) { Spider.create(new WebMagicCnBlogUtils()).addUrl(url).addPipeline(new ConsolePipeline()).run(); }}

單元測試代碼:

import com.blog.springboot.dto.CnBlogModelDTO;import us.codecraft.webmagic.Page;import us.codecraft.webmagic.Site;import us.codecraft.webmagic.Spider;import us.codecraft.webmagic.pipeline.ConsolePipeline;import us.codecraft.webmagic.processor.PageProcessor;import us.codecraft.webmagic.selector.Selectable;public class WebMagicJunitTest implements PageProcessor { private Site site = Site.me() .setDomain('https://www.cnblogs.com/') .setSleepTime(1000) .setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'); @Override public void process(Page page) { Selectable obj = page.getHtml().xpath('//div[@class=’post’]'); Selectable title = obj.xpath('//h1[@class=’postTitle’]//a'); Selectable content = obj.xpath('//div[@class=’blogpost-body’]'); System.out.println('title:' + title.replace('<[^>]*>', '')); System.out.println('content:' + content); } @Override public Site getSite() { return site; } public static void importSinglePost(String url) { Spider.create(new WebMagicJunitTest()).addUrl(url).addPipeline(new ConsolePipeline()).run(); } public static void main(String[] args) { WebMagicJunitTest.importSinglePost('https://www.cnblogs.com/youcong/p/9404007.html'); }

另外我是怎么知道要爬取哪些數據呢?需求第一，然后通過Chrome或Firefox瀏覽器檢查元素，如圖:

到此這篇關于Java爬蟲框架之WebMagic實戰的文章就介紹到這了,更多相關Java WebMagic內容請搜索好吧啦網以前的文章或繼續瀏覽下面的相關文章希望大家以后多多支持好吧啦網！