WebMagic爬虫入门教程（三）爬取汽车之家的实例-品牌车系车型结构等

本文使用WebMagic爬取汽车之家的品牌车系车型结构价格能源产地国别等；java代码

备注，只是根据url变化爬取的，没有使用爬取script页面具体的数据，也有反爬机制，知识简单爬取html标签

项目github地址：https://github.com/yongzhuo/JavaLearning/tree/master/src/java/Webmagic

爬取的网页：

需要配置pom.xml

        <!--测试--><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.12</version></dependency><dependency><groupId>us.codecraft</groupId><artifactId>webmagic-core</artifactId><version>0.7.3</version></dependency><dependency><groupId>us.codecraft</groupId><artifactId>webmagic-extension</artifactId><version>0.7.3</version></dependency><dependency><groupId>us.codecraft</groupId><artifactId>webmagic-samples</artifactId><version>0.7.3</version></dependency><groupId>us.codecraft</groupId><artifactId>webmagic-selenium</artifactId><version>0.7.3</version></dependency>

代码如下:跑@test测试代码，数据存在d:\\data\\AutoHome中

import org.apache.commons.io.FileUtils;
import org.junit.Test;
import us.codecraft.webmagic.Page;
import us.codecraft.webmagic.Site;
import us.codecraft.webmagic.Spider;
import us.codecraft.webmagic.processor.PageProcessor;
import us.codecraft.webmagic.selector.Html;
import us.codecraft.webmagic.selector.Selectable;import java.io.File;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.List;/***Created by mo*On 2018/4/23  ***10:04.******/
public class CarAutoHome implements PageProcessor {private static int myid = 0;int size = 0;// 抓取网站的相关配置，包括编码、抓取间隔、重试次数等private Site site = Site.me().setRetryTimes(5).setSleepTime(1000).setTimeOut(100000).setCharset("gbk").addHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36");Calendar now = Calendar.getInstance();String time = new SimpleDateFormat("HHmm").format(now.getTime());@Overridepublic Site getSite() {return site;}@Overridepublic void process(Page page) {Html html = page.getHtml();myid++;size++;try {String url = page.getUrl().get();String label = url.replaceAll("https://www.autohome.com.cn/","").replace("/","");int op = 0;List<String> one_car = new ArrayList<>();List<Selectable> nodes = html.xpath("//div[@class='uibox-con rank-list rank-list-pic']/dl").nodes();for (Selectable item : nodes) {Selectable item_2 = item;List<Selectable> cars = item_2.xpath("//dd/ul/li").nodes();String Car_brand = item_2.xpath("//dt/div/a/text()").get();//List<String> one_car = new ArrayList<>();for (Selectable model : cars) {String Car_https = model.xpath("//h4/a/@href").get();String Car_model = model.xpath("//h4/a/text()").get();String cart = label + "mo_mo" + Car_brand + "mo_mo" + Car_model + "mo_mo" + Car_https;one_car.add(cart);op++;System.out.println(op + "mo_mo" + cart);}}//            File file = new File("D:\\workspace\\java\\WebAnimationTest\\src\\main\\java\\Webmagic\\AutoHome\\car_brand_0423.txt");//汽车品牌File file = new File("D:\\data\\AutoHome"+ time+".txt");//进口，出口
//            File file = new File("D:\\workspace\\java\\WebAnimationTest\\src\\main\\java\\Webmagic\\AutoHome\\origit_data\\car_oil.txt");//进口，出口FileUtils.writeLines(file, one_car, true);} catch (Exception e) {}}//    public static void main(String[] args) throws IOException, InterruptedException {@Test//获取品牌，车系等public void brand_model(){try {long startTime, endTime;System.out.println("开始爬取...");startTime = System.currentTimeMillis();//"https://www.autohome.com.cn/2951/?pvareaid=105126"//a00/a0/a/b/c/d/suv/mpv/s/p"https://www.autohome.com.cn/3948/#levelsource=000000000_0&pvareaid=101594"List<String> list = new ArrayList<>();list.add("a00");list.add("a0");list.add("a");list.add("b");list.add("c");list.add("d");list.add("suv");list.add("mpv");list.add("s");list.add("p");list.add("mb");list.add("qk");for(String list_car :list){String https = "https://www.autohome.com.cn/"+ list_car +"/";Spider.create(new CarAutoHome()).addUrl(https).thread(5).run();endTime = System.currentTimeMillis();System.out.println(list_car + "\t"+ (endTime-startTime) +"\t");Thread.sleep(10000);}} catch (Exception e) { }}@Test//获取进口或者是出口数据public void car_board(){try {long startTime, endTime;System.out.println("开始爬取...");startTime = System.currentTimeMillis();List<String> list = new ArrayList<>();list.add("0_0-0.0_0.0-0-0-0-1-0-0-0-0");list.add("0_0-0.0_0.0-0-0-0-3-0-0-0-0");for(String list_car :list){String https = "https://www.autohome.com.cn/car/"+ list_car +"/";Spider.create(new CarAutoHome()).addUrl(https).thread(5).run();endTime = System.currentTimeMillis();System.out.println(list_car + "\t"+ (endTime-startTime) +"\t");Thread.sleep(10000);}} catch (Exception e) { }}@Test//获取座位号等public void brand_sit(){try {long startTime, endTime;System.out.println("开始爬取...");startTime = System.currentTimeMillis();List<String> list = new ArrayList<>();list.add("0_0-0.0_0.0-0-0-0-0-0-0-5-0");list.add("0_0-0.0_0.0-0-0-0-0-0-0-7-0");list.add("0_0-0.0_0.0-0-0-0-0-0-0-4-0");list.add("0_0-0.0_0.0-0-0-0-0-0-0-2-0");list.add("0_0-0.0_0.0-0-0-0-0-0-0-6-0");list.add("0_0-0.0_0.0-0-0-0-0-0-0-8-0");for(String list_car :list){String https = "https://www.autohome.com.cn/car/"+ list_car +"/";Spider.create(new CarAutoHome()).addUrl(https).thread(5).run();endTime = System.currentTimeMillis();System.out.println(list_car + "\t"+ (endTime-startTime) +"\t");Thread.sleep(10000);}} catch (Exception e) { }}@Test//结构：两箱，3相public void brand_structure(){try {long startTime, endTime;System.out.println("开始爬取...");startTime = System.currentTimeMillis();List<String> list = new ArrayList<>();list.add("0_0-0.0_0.0-0-0-1-0-0-0-0-0");list.add("0_0-0.0_0.0-0-0-2-0-0-0-0-0");list.add("0_0-0.0_0.0-0-0-3-0-0-0-0-0");list.add("0_0-0.0_0.0-0-0-4-0-0-0-0-0");list.add("0_0-0.0_0.0-0-0-5-0-0-0-0-0");list.add("0_0-0.0_0.0-0-0-6-0-0-0-0-0");list.add("0_0-0.0_0.0-0-0-7-0-0-0-0-0");list.add("0_0-0.0_0.0-0-0-8-0-0-0-0-0");list.add("0_0-0.0_0.0-0-0-9-0-0-0-0-0");for(String list_car :list){String https = "https://www.autohome.com.cn/car/"+ list_car +"/";Spider.create(new CarAutoHome()).addUrl(https).thread(5).run();endTime = System.currentTimeMillis();System.out.println(list_car + "\t"+ (endTime-startTime) +"\t");Thread.sleep(10000);}} catch (Exception e) { }}@Test//能源：汽油public void brand_engine(){try {long startTime, endTime;System.out.println("开始爬取...");startTime = System.currentTimeMillis();List<String> list = new ArrayList<>();list.add("0_0-0.0_0.0-0-0-0-0-1-0-0-0");list.add("0_0-0.0_0.0-0-0-0-0-2-0-0-0");list.add("0_0-0.0_0.0-0-0-0-0-3-0-0-0");list.add("0_0-0.0_0.0-0-0-0-0-4-0-0-0");list.add("0_0-0.0_0.0-0-0-0-0-5-0-0-0");for(String list_car :list){String https = "https://www.autohome.com.cn/car/"+ list_car +"/";Spider.create(new CarAutoHome()).addUrl(https).thread(5).run();endTime = System.currentTimeMillis();System.out.println(list_car + "\t"+ (endTime-startTime) +"\t");Thread.sleep(10000);}} catch (Exception e) { }}@Test//能源：汽油public void brand_gas(){try {long startTime, endTime;System.out.println("开始爬取...");startTime = System.currentTimeMillis();List<String> list = new ArrayList<>();list.add("0_0-0.0_0.0-0-0-0-0-1-0-0-0");list.add("0_0-0.0_0.0-0-0-0-0-2-0-0-0");list.add("0_0-0.0_0.0-0-0-0-0-3-0-0-0");list.add("0_0-0.0_0.0-0-0-0-0-4-0-0-0");list.add("0_0-0.0_0.0-0-0-0-0-5-0-0-0");for(String list_car :list){String https = "https://www.autohome.com.cn/car/"+ list_car +"/";Spider.create(new CarAutoHome()).addUrl(https).thread(5).run();endTime = System.currentTimeMillis();System.out.println(list_car + "\t"+ (endTime-startTime) +"\t");Thread.sleep(10000);}} catch (Exception e) { }}@Test//汽油量,V,Lpublic void brand_oil(){try {long startTime, endTime;System.out.println("开始爬取...");startTime = System.currentTimeMillis();List<String> list = new ArrayList<>();list.add("0_0-0.0_1.0-0-0-0-0-0-0-0-0");list.add("0_0-1.1_1.6-0-0-0-0-0-0-0-0");list.add("0_0-1.7_2.0-0-0-0-0-0-0-0-0");list.add("0_0-2.1_2.5-0-0-0-0-4-0-0-0");list.add("0_0-2.6_3.0-0-0-0-0-5-0-0-0");list.add("0_0-3.1_4.0-0-0-0-0-4-0-0-0");list.add("0_0-4.0_0.0-0-0-0-0-0-0-0-0");for(String list_car :list){String https = "https://www.autohome.com.cn/car/"+ list_car +"/";Spider.create(new CarAutoHome()).addUrl(https).thread(5).run();endTime = System.currentTimeMillis();System.out.println(list_car + "\t"+ (endTime-startTime) +"\t");Thread.sleep(10000);}} catch (Exception e) { }}@Test//价格等public void brand_price(){try {long startTime, endTime;System.out.println("开始爬取...");startTime = System.currentTimeMillis();List<String> list = new ArrayList<>();list.add("1_5-0.0_0.0-0-0-0-0-0-0-0-0");list.add("5_10-0.0_0.0-0-0-0-0-0-0-0-0");list.add("10_15-0.0_0.0-0-0-0-0-0-0-0-0");list.add("15_20-0.0_0.0-0-0-0-0-0-0-0-0");list.add("20_25-0.0_0.0-0-0-0-0-0-0-0-0");list.add("25_30-0.0_0.0-0-0-0-0-0-0-0-0");list.add("30_35-0.0_0.0-0-0-0-0-0-0-0-0");list.add("35_50-0.0_0.0-0-0-0-0-0-0-0-0");list.add("50_70-0.0_0.0-0-0-0-0-0-0-0-0");list.add("70_100-0.0_0.0-0-0-0-0-0-0-0-0");for(String list_car :list){String https = "https://www.autohome.com.cn/car/"+ list_car +"/";Spider.create(new CarAutoHome()).addUrl(https).thread(5).run();endTime = System.currentTimeMillis();System.out.println(list_car + "\t"+ (endTime-startTime) +"\t");Thread.sleep(10000);}} catch (Exception e) { }}}

希望对你有所帮助！

WebMagic爬虫入门教程（三）爬取汽车之家的实例-品牌车系车型结构等相关推荐

java爬虫入门--用jsoup爬取汽车之家的新闻
概述使用jsoup来进行网页数据爬取.jsoup 是一款Java 的HTML解析器,可直接解析某个URL地址.HTML文本内容.它提供了一套非常省力的API,可通过DOM,CSS以及类似于jQuer ...
python爬虫利用Scrapy框架爬取汽车之家奔驰图片--实战
先看一下利用scrapy框架爬取汽车之家奔驰A级的效果图 1)进入cmd命令模式下,进入想要存取爬虫代码的文件,我这里是进入e盘下的python_spider文件夹内 C:\Users\15538&g ...
Python爬虫入门教程：爬取妹子图网站 - 独行大佬
妹子图网站---- 安装requests打开终端:使用命令pip3 install requests等待安装完毕即可使用接下来在终端中键入如下命令?123# mkdir demo # cd demo# ...
python爬虫入门教程：爬取网页图片
在现在这个信息爆炸的时代,要想高效的获取数据,爬虫是非常好用的.而用python做爬虫也十分简单方便,下面通过一个简单的小爬虫程序来看一看写爬虫的基本过程: 准备工作语言:python IDE:py ...
python3 [入门基础实战] 爬虫入门之xpath爬取脚本之家python栏目
这次爬取的确实有些坎坷,经过了两个晚上吧,最后一个晚上还是爬取数据到最后一公里了,突然报错了.又得继续重新进行爬取先来个爬取结果图,这次爬取的是标题,url,还有日期,估计也就只有这么多内容,用的单 ...
python爬虫入门教程｜爬取游民福利，大量妹子图片，源码＋多张福利
点赞再看,养成习惯文章目录分析一下步骤源码一键运行下载本地待解决难点有这样一个网站,对它做了一下爬取,先看看效果,这样. 上图是一小部分,目测大概有 260*20*66+ ,下面就授人 ...
Python 爬虫实战入门——爬取汽车之家网站促销优惠与经销商信息
在4S店实习,市场部经理让我写一个小程序自动爬取汽车之家网站上自家品牌的促销文章,因为区域经理需要各店上报在网站上每一家经销商文章的露出频率,于是就自己尝试写一个爬虫,正好当入门了. 一.自动爬取并输 ...
python爬虫入门教程(三)：淘女郎爬虫 ( 接口解析 | 图片下载 )
2019/10/28更新网站已改版,代码已失效(其实早就失效了,但我懒得改...)此博文仅供做思路上的参考代码使用python2编写,因已失效,就未改写成python3 爬虫入门系列教程: pyt ...
python爬取汽车之家_python爬虫实战之爬取汽车之家网站上的图片
随着生活水平的提高和快节奏生活的发展.汽车开始慢慢成为人们的必需品,浏览各种汽车网站便成为购买合适.喜欢车辆的前提.例如汽车之家网站中就有最新的报价和图片以及汽车的相关内容,是提供信息最快最全的中国汽 ...

WebMagic爬虫入门教程（三）爬取汽车之家的实例-品牌车系车型结构等

WebMagic爬虫入门教程（三）爬取汽车之家的实例-品牌车系车型结构等相关推荐

最新文章

热门文章