Python：好租网商务楼数据爬取

前言：

爬的方法和以前房天下类似，只是在使用next_sibling的时候一直不成功，所以先找了parent然后再找儿子的方式曲线救国解决了。
因为我的需求只是名字、区域、地址和楼层四个，所以参数不多，需要的可以自己在我的代码基础上增加。

代码：

import requests
from bs4 import BeautifulSoupdef get_building_url(url):'''获取每一页的大楼的内容（名称和链接）'''r = requests.get(url=url, headers=headers)soup = BeautifulSoup(r.text, 'lxml')for temp in soup.find_all(name='h1', attrs={'class': 'h1-title'}):building_dict = {'name': '', 'district': '', 'address': '', 'floor': ''}if temp.find(name='a'):building_href = temp.find(name='a').attrs['href'] #网址if building_href.startswith('/sh_'):building_href = 'https://www.haozu.com' + building_hrefbuilding_dict['name'] = temp.find(name='a').attrs['alt']  # 名称building_dict['district'], building_dict['address'], building_dict['floor'] = get_normal_building_information(building_href)else:building_href = 'https:' + building_hrefbuilding_dict['name'] = temp.find(name='a').stringbuilding_dict['district'], building_dict['address'], building_dict['floor'] = get_share_building_information(building_href)print(building_dict)export_buildingInfo(building_dict)def get_normal_building_information(url):'''获取每一个普通大楼的具体信息（区域、地址和楼层）'''r = requests.get(url=url, headers=headers)soup = BeautifulSoup(r.text, 'lxml')# print(soup)information = soup.find(name='div', attrs={'house-address'}).span.text.split('\xa0')district = information[0] #区域address = information[-1] #地址if soup.find(text='总楼层', name='span'):floor = soup.find(text='总楼层', name='span').parent.find(name='span', attrs={'class': 's2'}).attrs['title'] #楼层else:floor = ''return district, address, floordef get_share_building_information(url):'''获取每一个共享大楼的具体信息（区域、地址和楼层）'''r = requests.get(url=url, headers=headers)soup = BeautifulSoup(r.text, 'lxml')# print(soup)information = soup.find(name='p', attrs={'class':'p2'}).text# print(information)district = information[1:4].strip() #区域address = information.split(']')[1].split(' ')[0] #地址if soup.find(text='总楼层：', name='div'):floor = soup.find(text='总楼层：', name='div').parent.find(name='span', attrs={'class': 'li-con'}).attrs['title'] #楼层else:floor = ''return district, address, floordef export_buildingInfo(building_dict):'''导出大楼的信息'''with open('上海楼宇信息.txt', 'a', encoding='utf-8') as file:file.write('||'.join(building_dict.values()))file.write('\n')headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36','cookie': 'PHPSESSID=v9qdqo7slhi6t9mh3bhc6b75u3; citydomain=sh; haozu_user=%7B%22index%22%3A5%2C%22name%22%3A%22sem%22%2C%22value%22%3A%22sem_baidu%22%2C%22opt_scope%22%3A2%2C%22ca_n%22%3A%2295622646681%22%7D; Hm_lvt_826deb6478895f40cc4a3e9b54b0ba24=1590042924; _pk_ref.1.0dea=%5B%22%22%2C%22%22%2C1590042925%2C%22https%3A%2F%2Fwww.baidu.com%2Fbaidu.php%3Fsc.0f00000uEDLSpLgiCo05K50ear24i6vpsLPthZeMvmWjEXpNfxibearrTnyevkqZQU1y47v5azx134kS7veG7JKYwFdLX8tWsGevlVId2yobizOK0rIJubsZk1HDL6UcA0piKTrV9oz0otLQLXr5WFOc0CupdgJ4lHnGiV1mZSR-b3YClAgwVJPItNhqOlLWIjmlkWNo_JeYMUKAxyUXP39TPTWw.7b_NR2Ar5Od663rj6tJhgTrKDDIgQYTPZ-oLU8hjEa1zk_wGYtVerQKd9u4mhPSeGzurM_lXMugd9u4g_NT521TGJIGHz3qis1f_uQrPhkf.U1Yk0ZDqzTSlCzRznQH11tn-nWjfkTSB0ZKGm1Ys0ZK1pyI85H63uycYnyNWnvndP1c4n179nWuhnARYuWw-n179Pjnd0ZfqzTSlCqj71l2tEnodV555ssKGUHYznWR0u1dsT1c0Iybqmh7GuZR0TA-b5Hc0mv-b5Hfsr0KVIjYknjD4g1DsnHIxnW0vn-t1PW0k0AVG5H00TMfqn1fL0ANGujYkPjmzg1nkPWTsg1cknHDkg1nknHDkg1nkPH64g1nknWTsg1nkPjm1g1nsrjcY0AFG5HDdPNtkPH9xnW0Yg1ckPsKVm1Yknj0kg1D3PH03Pjmzn17xnHDknWnYPj03nHKxn0KkTA-b5Hc0TyPGujYs0ZFMIA7M5H00mycqn7ts0ANzu1Yz0ZKs5HfYP1f4n1cd0A4vTjYsQW0snj0snj0s0AdYTjYs0AwbUL0qn0KzpWYs0Aw-IWdsmsKhIjYs0ZKC5H00ULnqn0KBI1Ykn0K8IjYs0ZPl5fK9TdqGuAnqTZnVuyPJ0A-bm1dribGH0ZwdT1Ykn1R3n10Lrjnvn1cdnjb4njnLnsKzug7Y5HDdrH0sPjc4nH6Yn1R0Tv-b5H-9nAcLnvcsnj0krAuhrj00mLPV5HRvrH6srHTzPbc1rjKanHn0mynqnfKs%22%5D; _pk_ses.1.0dea=*; lookBuilding=12184; 1buildView=%5B%7B%22viewId%22%3A%2212184%22%2C%22userId%22%3A0%2C%22circleId%22%3A0%2C%22streetId%22%3A%227267%22%2C%22districtId%22%3A%22203%22%2C%22cityId%22%3A%2213%22%2C%22user_uuid%22%3A%22%22%2C%22type%22%3A1%2C%22category%22%3A1%2C%22viewTime%22%3A1590042966%7D%5D; showheader=1; Hm_lpvt_826deb6478895f40cc4a3e9b54b0ba24=1590043591; _pk_id.1.0dea=7e465a97f3decd72.1590042925.1.1590043602.1590042925.'
}for i in range(1, 150):url = r"https://www.haozu.com/sh/zuxiezilou/o" + str(i) + "/"try:get_building_url(url=url)except:pass

Python：好租网商务楼数据爬取相关推荐

python爬人人贷代码视频_【IT专家】人人贷网的数据爬取(利用python包selenium)
本文由我司收集整编,推荐下载,如有疑问,请与我司联系人人贷网的数据爬取(利用 python 包 selenium ) 2016/12/31 0 记得之前应同学之情,帮忙爬取人人贷网的借贷人信息,综合 ...
人人贷网的数据爬取（利用python包selenium）
记得之前应同学之情,帮忙爬取人人贷网的借贷人信息,综合网上各种相关资料,改善一下别人代码,并能实现数据代码爬取,具体请看我之前的博客:http://www.cnblogs.com/Yiutto/p/5 ...
python如何爬虫股票数据_简单爬虫：东方财富网股票数据爬取(python_017)
需求:将东方财富网行情中心的股票数据爬取下来,包括上证指数.深圳指数.上证A股.深圳A股.新股.中小板.创业板等一.目标站点分析东方财富网的行情中心页面包含了所有股票信息.在左侧的菜单栏中包含了 ...
金投网煤炭数据爬取-精进版
金投网数据爬取-加强版注:博主经过一个暑假的游手好闲,学习到了xpath及一些其他的有意思的小东西.对于之前爬取金投网的课题有了一些新的思路,特此卷土重来,让我们开始吧! 目录标题金投网数据爬取- ...
网贷数据爬取及据分析
关于数据来源本项目写于2017年七月初,主要使用Python爬取网贷之家以及人人贷的数据进行分析. 网贷之家是国内最大的P2P数据平台,人人贷国内排名前二十的P2P平台. 源码地址数据爬取抓包分 ...
Python爬虫|高德地图地铁数据爬取与制图
目录一.高德地图数据爬取 1.爬取思路 2.python核心代码二.Arcmap制图一.高德地图数据爬取 1.爬取思路首先,谷歌浏览器打开高德地图官网,点击上方菜单栏地铁进入地铁线路网站如下, ...
python爬虫实例——某二手车数据爬取
某二手车网站数据爬取要求: 找到所要爬取的网站网址(url): 今天案例的网址(url):https://www.guazi.com/gy/dazhong/o1/#bread. 观察网站,点开检查, ...
[Python]百度慧眼人口热力图数据爬取--以深圳市为例
百度慧眼人口热力图数据爬取--以深圳市为例数据爬取坐标转换 1.读取坐标映射表 2.利用sklearn进行回归分析 3.坐标转换输出完整代码利用python爬取深圳市百度慧眼人口热力图数据,线 ...
python爬虫案例-陶瓷公司数据爬取
用requests爬取要注意HTTPConnectionPool(host=xxx, port=xxx): Max retries exceeded with url...异常,出现这个异常的解决方法 ...

Python：好租网商务楼数据爬取

前言：

代码：

Python：好租网商务楼数据爬取相关推荐

最新文章

热门文章