Python3 bs4 + requests 简单的爬虫爬取LOL胜率加点

Python Bs4 + request 简单的爬虫

很早之前的python写着玩的东西，如果爬一些简单的东西，还用不到scrapy之类的爬虫框架

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import requests
import threadingcommon_data = threading.Lock()
winning_data = threading.Lock()def splite_table2(element, encoding, champions):winning_data.acquire()div_len = len(element)dic = dict()for i in range(div_len):span_element = element[i].find_all('div')span_len = len(span_element)for x in range(span_len):text = span_element[x].text.replace('\t', '').replace('\n', '')if text != '':dic[x] = textdic = sorted(dic.items(), key=lambda d: d[0], reverse=False)for key, values in dic:with open("胜利最高的技能加点.txt", 'a+', encoding=encoding) as f:if key == 0:f.write(champions + '={' + values + ',')elif key == 17:f.write(values + '}\n')else:f.write(values + ',')print(champions + '》》' + '胜利最高的技能加点.txt')winning_data.release()def splite_table(element, encoding, champions):common_data.acquire()div_len = len(element)dic = dict()for i in range(div_len):span_element = element[i].find_all('div')span_len = len(span_element)for x in range(span_len):text = span_element[x].text.replace('\t', '').replace('\n', '')if text != '':dic[x] = textdic = sorted(dic.items(), key=lambda d: d[0], reverse=False)for key, values in dic:with open("最常见的技能加点.txt", 'a+', encoding=encoding) as f:if key == 0:f.write(champions + '={' + values + ',')elif key == 17:f.write(values + '}\n')else:f.write(values + ',')print(champions + '》》' + '最常见的技能加点.txt')common_data.release()def jishukong(text, encoding, champions):document = BeautifulSoup(text, 'lxml')div_element = document.find_all('div', class_='skill-order clearfix')div_selected = div_element[0].find_all('div', class_='skill-selections')[1:5]div_selected2 = div_element[1].find_all('div', class_='skill-selections')[1:5]splite_table(div_selected, encoding, champions)splite_table2(div_selected2, encoding, champions)class Mythread(threading.Thread):def __init__(self, url, champ):self.url = urlself.champ = champsuper(Mythread, self).__init__()def run(self):self.url = self.url + self.champres = requests.get(self.url, verify=False)res.close()if res.ok and res.status_code == 200:res.encoding = res.apparent_encoding# spells_table = split_spells(res.text, res.apparent_encoding ,champions)# return spells_tableprint(self.champ + '》》等待处理')jishukong(res.text, res.apparent_encoding, self.champ)else:passdef main():Champs = "Aatrox", "Ahri", "Akali", "Alistar", "Amumu", "Anivia", "Annie", "Ashe", "AurelionSol", "Azir", "Bard", \"Blitzcrank", "Brand", "Braum", "Caitlyn", "Camille", "Cassiopeia", "Chogath", "Corki", "Darius", "Diana", \"Draven", "DrMundo", "Ekko", "Elise", "Evelynn", "Ezreal", "Fiddlesticks", "Fiora", "Fizz", "Galio", \"Gangplank", "Garen", "Gnar", "Gragas", "Graves", "Hecarim", "Heimerdinger", "Illaoi", "Irelia", "Ivern", \"Janna", "JarvanIV", "Jax", "Jayce", "Jhin", "Jinx", "Kaisa", "Kalista", "Karma", "Karthus", "Kassadin", \"Katarina", "Kayle", "Kayn", "Kennen", "Khazix", "Kindred", "Kled", "KogMaw", "Leblanc", "LeeSin", \"Leona", "Lissandra", "Lucian", "Lulu", "Lux", "Malphite", "Malzahar", "Maokai", "MasterYi", "MissFortune", \"MonkeyKing", "Mordekaiser", "Morgana", "Nami", "Nasus", "Nautilus", "Nidalee", "Nocturne", "Nunu", \"Olaf", "Orianna", "Ornn", "Pantheon", "Poppy", "Pyke", "Quinn", "Rakan", "Rammus", "RekSai", "Renekton", \"Rengar", "Riven", "Rumble", "Ryze", "Sejuani", "Senna", "Sett", "Shaco", "Shen", "Shyvana", "Singed", "Sion", "Sivir", \"Skarner", "Sona", "Soraka", "Swain", "Syndra", "TahmKench", "Taliyah", "Talon", "Taric", "Teemo", \"Thresh", "Tristana", "Trundle", "Tryndamere", "TwistedFate", "Twitch", "Udyr", "Urgot", "Varus", \"Vayne", "Veigar", "Velkoz", "Vi", "Viktor", "Vladimir", "Volibear", "Warwick", "Xayah", "Xerath", \"XinZhao", "Yasuo", "Yorick", "Zac", "Zed", "Ziggs", "Zilean", "Zoe", "Zyra"t_list = list()for i in range(len(Champs)):champs_name = Champs[i]i = Mythread('http://jishukong.com/champion?champKey=', champs_name)i.start()print(champs_name + '》》 开始搜集数据')t_list.append(i)for i in t_list:i.join()if __name__ == '__main__':main()

Python3 bs4 + requests 简单的爬虫爬取LOL胜率加点相关推荐

python简单网站爬虫-爬取北京7天最高、最低气温
python简单网站爬虫-爬取北京7天最高.最低气温前置操作: 1.待爬取网站: 北京天气的网址: http://www.weather.com.cn/weather1d/101010100.sht ...
简单python爬虫爬取游戏wiki立绘
简单python爬虫爬取游戏wiki立绘玩二次元手游是感叹美少女立绘真好看啊,可惜就是抽不到,于是看到b站wiki上有角色立绘,就写了个爬虫准备将立绘趴下来欣赏(舔). 本人爬虫的技术只算是初学,代 ...
scrapy初步-简单静态爬虫(爬取电影天堂所有电影)
之前用java写过一个简单的爬取电影天堂信息的爬虫,后来发现用python写这种简单的爬虫程序更简单,异步网络框架在不使用多线程和多进程的情况下也能增加爬取的速度,目前刚开始学scrapy,用这个写了 ...
使用python爬虫——爬取淘宝图片和知乎内容
本文主要内容: 目标:使用python爬取淘宝图片:使用python的一个开源框架pyspider(非常好用,一个国人写的)爬取知乎上的每个问题,及这个问题下的所有评论最简单的爬虫--如下pytho ...
python爬虫爬取豆瓣电影信息城市_python爬虫，爬取豆瓣电影信息
hhhhh开心,搞了一整天,查了不少python基础资料,终于完成了第一个最简单的爬虫:爬取了豆瓣top250电影的名字.评分.评分人数以及短评. 代码实现如下:#第一个最简单的爬虫 #爬取了豆瓣to ...
python爬取豆瓣短评_爬虫-爬取豆瓣短评
爬虫-爬取豆瓣短评啥是爬虫? 按照一定的规则,自动地抓取互联网信息的程序. 为啥要用爬虫? 可以利用爬虫自动地采集互联网中的信息,采集回来后进行相应的存储或处理,在需要检索某些信息的时候,只需在 ...
python3爬虫爬取英语单词到MySQL数据库
前言:我的第一篇博客,本人大一菜鸟一枚,第一次写python的项目,需要大量的数据,第一次接触爬虫这个东西,感觉十分有趣,想以后往这方面发展,特写下自己的爬虫学习经历,希望把自己的学习经历分享给初学者 ...
python爬网易新闻_爬虫基本介绍 python3 爬虫爬取网易新闻排行榜
爬虫基本介绍 1. 什么是爬虫? 爬虫是请求⽹网站并提取数据的⾃自动化程序 2. 爬虫的基本流程发起请求通过HTTP库向目标站点发起请求,即发送一个Request,请求可以包含额外的headers ...
python3 爬虫爬取华为应用市场 APP应用评论（一）爬取全部评论——学霸君
python3 爬虫爬取华为应用市场 APP应用评论(一)爬取全部评论--学霸君 import requests import re import requests from bs4 import ...

Python3 bs4 + requests 简单的爬虫爬取LOL胜率加点

Python Bs4 + request 简单的爬虫

Python3 bs4 + requests 简单的爬虫爬取LOL胜率加点相关推荐

最新文章

热门文章

Python3 bs4 + requests 简单的爬虫 爬取LOL胜率加点

Python Bs4 + request 简单的爬虫

Python3 bs4 + requests 简单的爬虫 爬取LOL胜率加点相关推荐

最新文章

热门文章

Python3 bs4 + requests 简单的爬虫爬取LOL胜率加点

Python3 bs4 + requests 简单的爬虫爬取LOL胜率加点相关推荐