Capturing JSON data from intermediate events using Selenium

Refresh

April 2019

Views

163 time

1

Below I have setup a script which simply executes a search on a website. The goal is to capture JSON data utilizing Selenium from an event that is fired from an intermediate script, namely the POST request to "https://www.botoxcosmetic.com/sc/api/findclinic/FindSpecialists" as seen in the included image, but without directly sending a request to that URL using Selenium or the requests library. What is the best way to do this, preferably in Python but open to any language?

from selenium import webdriver
base_url = 'https://www.botoxcosmetic.com/women/find-a-botox-cosmetic-specialist'
driver = webdriver.Chrome()
driver.find_element_by_class_name('normalZip').send_keys('10022')
driver.find_element_by_class_name('normalSearch').click()

enter image description here

1 answers

-1

You will need to use a proxy, my suggestion would be to use the BrowserMob Proxy, the following is directly taken from https://browsermob-proxy-py.readthedocs.io/en/stable/.

First of all install the BrowserMob Proxy:

pip install browsermob-proxy

Then use it in your script:

from browsermobproxy import Server
server = Server("path/to/browsermob-proxy")
server.start()
proxy = server.create_proxy()

from selenium import webdriver
profile  = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)


proxy.new_har("google")
driver.get("http://www.google.co.uk")
proxy.har # returns a HAR JSON blob

server.stop()
driver.quit()

You can use the BrowserMob Proxy to intercept all traffic sent from the browser.