Selenium and BeautifulSoap, scraping JS generated part of page

Refresh

April 2019

Views

40 time

1

I'm trying to scrape a part of page that is generated by JS so the value isn't present in the source code.

I am using Selenium (first time) with BeautifulSoap (familiar with this).

CAUTION: The URL is an adult site!

Here is my code:

import sys
import urllib2
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from bs4 import BeautifulSoup
import re
import pandas as pd
import os

url = "https://www.kink.com/shoot/5540"

options = Options()
options.headless = True
driver = webdriver.Firefox(options=options)
driver.implicitly_wait(30)
driver.get(url)

soup = BeautifulSoup(driver.page_source, 'lxml')

print url

print soup

However the soup I am getting doesn't contain the JS generated bit.

I specifically need the rating part only that says: "Average Rating: 58 ratings".

What am I missing?

0 answers