在使用python做爬虫的过程中，有些页面的的部分数据是通过js异步加载的，js调用接口的请求中有时还带有些加密的参数很难破解无法使用requests这样的包直接爬取数据，因此需要借助seleniu来完成js的自动加载。

通过selenium 模拟浏览器的真是操作来获取页面中的所有请求，并且可以查找到一下页面上一些隐藏的元素，这些元素在html源码中无法看到，并且和能通过xpath和正则来捕获，因此需要使用selenium来查找隐藏元素，例如视频网站的播放按钮

代码如下

Python资源共享群：484031800

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import time

导入自动化测试模块

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.common.action_chains import ActionChains
import re
import json
d = DesiredCapabilities.CHROME
d['loggingPrefs'] = {'performance': 'ALL'}

设置谷歌浏览器参数，设置语言和浏览器版本（使用真实浏览器头信息代替字符浏览器头）

options = webdriver.ChromeOptions()
options.add_argument('lang=zh_CN.UTF-8')
options.add_argument('user-agent="Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Mobile Safari/537.36"')

建立一个浏览器对象，调用PhantoJS来生成一个对象

browser = webdriver.Chrome(executable_path='D:\Python36\chromedriver.exe', chrome_options=options, desired_capabilities=d)
video_url = 'https://v.youku.com/v_show/id_XNDIyMjU1NjgzMg==.html?spm=a2ha1.12675304.m_2556_c_8261.d_2&s=a4de6bdc5675415ea280&scm=20140719.manual.2556.show_a4de6bdc5675415ea280'
browser.get(video_url)
time.sleep(10)
menu = browser.find_element_by_css_selector(".ykplayer") # 查找页面隐藏元素

menu = browser.find_element_by_css_selector("div.x-video-play-ico.i-pause > dt").text

print(menu)
print(type(menu))
menu = browser.find_element_by_css_selector(".ykplayer").text # 获得隐藏元素的内容
print(menu)
print(type(menu))
###################

ActionChains(browser).click()

browser.find_element_by_css_selector('.ykplayer').click() # 查找到隐藏元素位置后执行点击鼠标左键操作
time.sleep(10)
lo = browser.get_log('performance') # 捕获浏览器network中的数据
browser.get_network_conditions()
datalist = []
for entry in lo:
try:
print(entry)
datalist.append(entry)
except Exception as e:
continue
browser.close()
</pre>

视频中被播放按钮是一个隐藏元素，用xpaht定位到指定的标签后也无法获取这个标签中的元素

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1564037447512" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

<input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

通过 browser.find_element_by_css_selector(“.ykplayer”) 方法查找 .ykplayer 可以查找到，查找后输出的内容这个视频的标题，这样就实现了使用chromedriver 自动播放视频的功能

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1564037447517" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

<input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>