美文网首页工作生活
fetch google arts and culture bi

fetch google arts and culture bi

作者: 狼无雨雪 | 来源:发表于2019-07-05 12:54 被阅读0次
"""
really used in fetching url from https://artsandculture.google.com/entity/m0bwbv?categoryid=art-movement
"""
from selenium import webdriver
import time
import os
from bs4 import BeautifulSoup
# os.environ["PATH"] += os.pathsep + 'D:\google-art-downloader-master'

browser = webdriver.Chrome()
browser.get('https://artsandculture.google.com/entity/m0bwbv?categoryid=art-movement')
asserts_all=set()
while browser.find_element_by_xpath('//*[@id="exp_tab_popular"]/div/div/div[2]').get_attribute("aria-hidden") != "true":
    pageSource = browser.page_source

    soup = BeautifulSoup(pageSource,'lxml')
    asserts = soup.find_all('a',{'class':"e0WtYb kdYEFe ZEnmnd PJLMUc"})
    for assert_value in asserts:
        asserts_all.add(assert_value.get("href"))
    print(len(asserts_all))
    browser.find_element_by_xpath('//*[@id="exp_tab_popular"]/div/div/div[2]').click()
    time.sleep(2)
with open("asserts.txt",'w',encoding="utf8") as write_file:
    for line in asserts_all:
        write_file.write(line+"\n")

browser.close()

相关文章

网友评论

    本文标题:fetch google arts and culture bi

    本文链接:https://www.haomeiwen.com/subject/dszrhctx.html