I'm trying to scrape a part of page that is generated by JS so the value isn't present in the source code.
I am using Selenium (first time) with BeautifulSoap (familiar with this).
CAUTION: The URL is an adult site!
Here is my code:
import sys import urllib2 from selenium import webdriver from selenium.webdriver.firefox.options import Options from bs4 import BeautifulSoup import re import pandas as pd import os url = "https://www.kink.com/shoot/5540" options = Options() options.headless = True driver = webdriver.Firefox(options=options) driver.implicitly_wait(30) driver.get(url) soup = BeautifulSoup(driver.page_source, 'lxml') print url print soup
However the soup I am getting doesn't contain the JS generated bit.
I specifically need the rating part only that says: "Average Rating: 58 ratings".
What am I missing?