Selenium and BeautifulSoap, scraping JS generated part of page


2 weeks ago


40 time


I'm trying to scrape a part of page that is generated by JS so the value isn't present in the source code.

I am using Selenium (first time) with BeautifulSoap (familiar with this).

CAUTION: The URL is an adult site!

Here is my code:

import sys
import urllib2
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from bs4 import BeautifulSoup
import re
import pandas as pd
import os

url = ""

options = Options()
options.headless = True
driver = webdriver.Firefox(options=options)

soup = BeautifulSoup(driver.page_source, 'lxml')

print url

print soup

However the soup I am getting doesn't contain the JS generated bit.

I specifically need the rating part only that says: "Average Rating: 58 ratings".

What am I missing?

0 answers