python爬虫之获取携程网所有航班机票信息，与携程网共同变化的爬虫博客

我们看一下效果

我实现的功能是

1、爬虫热门城市之间航班信息的查询

2、存储到mysql数据库中；

当然你也可以爬取所有城市之间的航班信息，我会提供完整的city代码给你；只是我需要实现的是自动功能；

我们要学会怎么爬虫，而不是复制代码，然后改不出来就一直停滞那里，然后骂别人写的垃圾；

我们先看一下携程网的信息

从以上我们可以获取两个信息

1、我是从上海到成都的

2、7条航班信息

教你如何爬虫
我不教静态网页的爬虫，因为太简单了；我们直接上手携程网

我们到这个上述这个页面以后，我推荐用谷歌浏览器，方便；

1、谷歌浏览器

2、携程网显示机票信息页面

3、f12

4、然后我们刷新页面，network下就有很多的文件了

选择XHR，然后出现以下几条信息

点击其中一条，然后点击右侧的preview

我们可以看到里面有数据，不一定是第一条，我们查看里面的数据有没有我们要的信息

举个例子，其中的pageData，我们点击，然后发现previewer里面的数据很少，根本没有一条是我们需要的；所以肯定不是pageData这个文件

其实是第一条product

现在已经有眉目了，我们需要这个product文件的信息，点击Headers

从上面我们可以知道我们爬虫的地址是url = 'http://flights.ctrip.com/itinerary/api/12808/products'

这个请求是post请求；那么我们需要发送什么样的数据才会有这样的结果呢？

废话不多说，直接贴上代码

记住，不要频繁的爬取，会被网站设置填写校验码才能搜素的；然后你就会爬不出来任何东西；

如果你已经这样了，那么要么换一个无线网，要么换一个电脑，他们的Ip不一样就可以了；

我的表结构是这样的

因为我后续会对这个表进行处理，所以用了Django框架的表生成

from django.db import models
# Create your models here.
class airport(models.Model):id = models.AutoField(primary_key=True)   #是否设置为主键fcity = models.CharField(max_length=32)#开始城市tcity = models.CharField(max_length=32)#目的城市date = models.CharField(max_length=32)#日期airlineName = models.CharField(max_length=32)#航空公司flightNumber = models.CharField(max_length=32)#航空公司编码airportName = models.CharField(max_length=32)#机场名称departureDate = models.CharField(max_length=32)#出发时间arrivalDate = models.CharField(max_length=32)#结束时间punctualityRate = models.CharField(max_length=32)#优惠率jprice = models.CharField(max_length=32)  #经济舱价格fprice = models.CharField(max_length=32)  # 公务舱价格def __str__(self):return "<airport:{fcity=%s,tcity=%s,date=%s,airlineName=%,flightNumber=%,airportName=%,departureDate=%,arrivalDate=%,punctualityRate=%,jprice=%,fprice=%}>"\%(self.id,self.fcity,self.tcity,self.date,self.airlineName,self.flightNumber,self.airportName,self.departureDate,self.arrivalDate,self.punctualityRate,self.jprice,self.fprice)

from prettytable import PrettyTable
import requests
import json
import pymysql
import time
from operator import itemgetter
class FLIGHT(object):def __init__(self):self.url = 'http://flights.ctrip.com/itinerary/api/12808/products'self.headers={"User-Agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0","Content-Type": "application/json",  # 声明文本类型为 json 格式"referer": r"https://flights.ctrip.com/itinerary/oneway/SHA-TAO?date=2020-04-11"}self. city = {"BJS": "北京","SHA": "上海", "CAN": "广州","SZX": "深圳","CTU": "成都","HGH": "杭州", "WUH": "武汉","SIA": "西安","CKG": "重庆","TAO": "青岛","CSX": "长沙","NKG": "南京","XMN": "厦门","KMG": "昆明","DLC": "大连","TSN": "天津","CGO": "郑州","SYX": "三亚","TNA": "济南","FOC": "福州"}def insert(self,value):db = pymysql.connect("localhost", "root", "123456", "python")cursor = db.cursor()sql = "INSERT INTO airport_airport(fcity,tcity,date,airlineName,flightNumber,airportName,departureDate,arrivalDate,punctualityRate,jprice,fprice) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)"try:cursor.execute(sql, value)db.commit()print('插入数据成功')except:db.rollback()print("插入数据失败")db.close()def xiecheng(self,date):##循环遍历for fromcode, fromcity in sorted(self.city.items(), key=itemgetter(0)):for tocode, tocity in sorted(self.city.items(), key=itemgetter(0)):if fromcode != tocode:request_payload = {"flightWay": "Oneway","classType": "ALL","hasChild": 'false',"hasBaby": 'false',"searchIndex": 1,"airportParams": [{"dcity":fromcode, "acity":tocode,"dcityname":fromcity ,"acityname":tocity ,"date": date}]};time.sleep(20)print(fromcode,fromcity,tocode,tocity,date)self.findall(request_payload,fromcity,tocity,date)def findall(self,request_payload,fromcity,tocity,date):# 这里传进去的参数必须为 json 格式response = requests.post(self.url, data=json.dumps(request_payload), headers=self.headers).textrouteList = json.loads(response).get('data').get('routeList')#循环这个数据集合print("接受的数据集合:",routeList)if(routeList!='' and routeList!=None):for route in routeList:if len(route.get('legs')) == 1:print("开始存储数据库....")legs = route.get('legs')[0]flight = legs.get('flight')airlineName= flight.get('airlineName')# companyflightNumber= flight.get('flightNumber')# companyidairportName=flight.get('departureAirportInfo').get('airportName')#portnamedepartureDate=flight.get('departureDate')#starttimearrivalDate=flight.get('arrivalDate')#endtimepunctualityRate= flight.get('punctualityRate')#ratelowestPrice = legs.get('characteristic').get('lowestPrice')  # 经济舱价格fprice = legs.get('characteristic').get('lowestCfPrice')#公务舱价格#存储到数据库中value=[fromcity,tocity,date,airlineName,flightNumber,airportName,departureDate,arrivalDate,punctualityRate,fprice,lowestPrice]print("路线：",fromcity,tocity,date,airlineName,flightNumber,airportName,departureDate,arrivalDate,punctualityRate,fprice,lowestPrice)self.insert(value)else:return;if __name__ == "__main__":fly = FLIGHT()date ="2020-05-10"#创建数据库#循环遍历fly.xiecheng(date)

完整的city部分是这样的

city={"AAT":"阿勒泰","ACX":"兴义","AEB":"百色","AKU":"阿克苏","AOG":"鞍山","AQG":"安庆","AVA":"安顺","AXF":"阿拉善左旗","BAV":"包头","BFJ":"毕节","BHY":"北海","BJS":"北京","BPE":"秦皇岛","BPL":"博乐","BPX":"昌都","BSD":"保山","CAN":"广州","CDE":"承德","CGD":"常德","CGO":"郑州","CGQ":"长春","CHG":"朝阳","CIF":"赤峰","CIH":"长治","CKG":"重庆","CSX":"长沙","CTU":"成都","CWJ":"沧源","CYI":"嘉义","CZX":"常州","DAT":"大同","DAX":"达县","DBC":"白城","DCY":"稻城","DDG":"丹东","DIG":"香格里拉(迪庆)","DLC":"大连","DLU":"大理","DNH":"敦煌","DOY":"东营","DQA":"大庆","DSN":"鄂尔多斯","DYG":"张家界","EJN":"额济纳旗","ENH":"恩施","ENY":"延安","ERL":"二连浩特","FOC":"福州","FUG":"阜阳","FUO":"佛山","FYJ":"抚远","GOQ":"格尔木","GYS":"广元","GYU":"固原","HAK":"海口","HDG":"邯郸","HEK":"黑河","HET":"呼和浩特","HFE":"合肥","HGH":"杭州","HIA":"淮安","HJJ":"怀化","HKG":"香港","HLD":"海拉尔","HLH":"乌兰浩特","HMI":"哈密","HPG":"神农架","HRB":"哈尔滨","HSN":"舟山","HTN":"和田","HUZ":"惠州","HYN":"台州","HZG":"汉中","HZH":"黎平","INC":"银川","IQM":"且末","IQN":"庆阳","JDZ":"景德镇","JGD":"加格达奇","JGN":"嘉峪关","JGS":"井冈山","JHG":"西双版纳","JIC":"金昌","JIQ":"黔江","JIU":"九江","JJN":"晋江","JMJ":"澜沧","JMU":"佳木斯","JNG":"济宁","JNZ":"锦州","JSJ":"建三江","JUH":"池州","JUZ":"衢州","JXA":"鸡西","JZH":"九寨沟","KCA":"库车","KGT":"康定","KHG":"喀什","KHN":"南昌","KJH":"凯里","KMG":"昆明","KNH":"金门","KOW":"赣州","KRL":"库尔勒","KRY":"克拉玛依","KWE":"贵阳","KWL":"桂林","LCX":"龙岩","LDS":"伊春","LFQ":"临汾","LHW":"兰州","LJG":"丽江","LLB":"荔波","LLF":"永州","LLV":"吕梁","LNJ":"临沧","LPF":"六盘水","LUM":"芒市","LXA":"拉萨","LYA":"洛阳","LYG":"连云港","LYI":"临沂","LZH":"柳州","LZO":"泸州","LZY":"林芝","MDG":"牡丹江","MFK":"马祖","MFM":"澳门","MIG":"绵阳","MXZ":"梅州","NAO":"南充","NBS":"白山","NDG":"齐齐哈尔","NGB":"宁波","NGQ":"阿里","NKG":"南京","NLH":"宁蒗","NNG":"南宁","NNY":"南阳","NTG":"南通","NZH":"满洲里","OHE":"漠河","PZI":"攀枝花","RHT":"阿拉善右旗","RIZ":"日照","RKZ":"日喀则","RLK":"巴彦淖尔","SHA":"上海","SHE":"沈阳","SIA":"西安","SJW":"石家庄","SWA":"揭阳","SYM":"普洱","SYX":"三亚","SZX":"深圳","TAO":"青岛","TCG":"塔城","TCZ":"腾冲","TEN":"铜仁","TGO":"通辽","THQ":"天水","TLQ":"吐鲁番","TNA":"济南","TSN":"天津","TVS":"唐山","TXN":"黄山","TYN":"太原","URC":"乌鲁木齐","UYN":"榆林","WEF":"潍坊","WEH":"威海","WMT":"遵义(茅台)","WNH":"文山","WNZ":"温州","WUA":"乌海","WUH":"武汉","WUS":"武夷山","WUX":"无锡","WUZ":"梧州","WXN":"万州","XFN":"襄阳","XIC":"西昌","XIL":"锡林浩特","XMN":"厦门","XNN":"西宁","XUZ":"徐州","YBP":"宜宾","YCU":"运城","YIC":"宜春","YIE":"阿尔山","YIH":"宜昌","YIN":"伊宁","YIW":"义乌","YNJ":"延吉","YNT":"烟台","YNZ":"盐城","YTY":"扬州","YUS":"玉树","YZY":"张掖","ZAT":"昭通","ZHA":"湛江","ZHY":"中卫","ZQZ":"张家口","ZUH":"珠海","ZYI":"遵义(新舟)"}"""{"KJI":"布尔津"}"""