Questions tagged [python-requests]

1

votes
2

answer
41

Views

Getting full html back from a website request using Python

I'm trying to send an http request to a website (for ex, Digikey) and read back the full html. For example, I'm using this link: https://www.digikey.com/products/en?keywords=part_number to get a part number such as: https://www.digikey.com/products/en?keywords=511-8002-KIT. However what I get back i...
ItM
1

votes
2

answer
42

Views

Identify the word 'method' from a tag and extract the text

I need to identify all tags which has the word 'method' in it. I developed a python code using requests and regex. The code will first read a text file to extract the ID and then use request to open the URL to identify the tags that have 'method' keyword in it however the output is returning empty l...
RRg
1

votes
1

answer
21

Views

Debugging a python requests module 400 error

I'm doing a post request using python and confluence REST API in order to update confluence pages via a script. I ran into a problem which caused me to receive a 400 error in response to a requests.put(url, data = jsonData, auth = (username, passwd), headers = {'Content-Type' : 'application/json'})...
0

votes
0

answer
3

Views

MissingSchema - Invalid URL Error with Requests module

Trying to get data from the eBay API using GetItem. But requests isn't reading or getting the URL properly, any ideas? Getting this error: requests.exceptions.MissingSchema: Invalid URL '': No schema supplied. Perhaps you meant http://? I swear I had this code working before but now it's not, so I'm...
Scott
1

votes
1

answer
3k

Views

How to fill a form with post request and get response

I have a form like following: url = 'http:/foo.com' State:  CA OR AL Populartion:  100 200 300 Show Tree So, basically the form has two options, State and Population and each has some options.. The idea is to select the options from the form and then submit. On submit, the results are displayed...
Fraz
1

votes
1

answer
48

Views

Why is BeautifulSoup4 missing the first file URL?

I'm trying to catalog the files on this website as a personal exercise. When I run the following code I don't know why I'm not getting the first file url on this website. Any help is appreciated. import requests from bs4 import BeautifulSoup import regex url = 'https://www.liberliber.it/online/autor...
Frontsky
1

votes
2

answer
64

Views

How to specify requests to return data as JSON?

I'm new to the Python requests module and I'm trying to export game data from a user from the Lichess.org API (a chess website). Here is my code: import requests, json url = 'https://www.lichess.org/api/games/user/mbellm' r = requests.get(url, params={'max':2, 'analysed':'true', 'clocks':'true', 'ev...
Matt
1

votes
2

answer
22

Views

Python How to GET Image and then POST (via requests library)

I am using python to try and get an image from one API, and then post it to a separate API. So far I have this code: def create_item(token, user_id): url = '' headers = {'Authorization': 'Token {0}'.format(token)} image_url = '' image_obj = requests.get(image_url) data = { 'image': image_obj.content...
Hybrid
1

votes
0

answer
11

Views

Is there any solution for fetching metadata of Crossref using DOIs (Rest API calls)

I am retrieving metadata of Crossref using crossref rest API I have CSV file of DOIs from which I fetch DOI using python and make API call for each DOI to retrieve metadata from Crossref. I have to fetch metadata for many DOIs but after retrieving some metadata it gives connection error import reque...
sufyan faizi
0

votes
1

answer
19

Views

How scrape a website in which i post information

I want to scrape announcements information from the https://nseindia.com/corporates/corporateHome.html?id=allAnnouncements. Specifically i want to goto Corporate information tab on left hand side of website and then open the link of corporate announcements under equities. After that i want to post i...
user159944
1

votes
0

answer
68

Views

Python requests.put, how to set key to null

Does anyone know how to send None with requests? As I understand that by having None will make Python ignore 'port_id' and that it will not be included in the body. Is there a way to change that and make sure None is sent? data = {'port_id': None, 'token': TOKEN} r = requests.put(URL + 'api/port/'+...
Alberto Santos
1

votes
0

answer
85

Views

sorry page not found message while using valid WOEID for Twitter API

I am trying to generate the trending topics for WOEID 23424911 (Nepal). It generates the following message TwitterError: [{u'message': u'Sorry, that page does not exist.', u'code': 34}] code used: import twitter consumer_key=..... consumer_secret=...... access_token=....... access_token_secret=........
lpt
1

votes
1

answer
77

Views

Python web scraping - access HTML after a few seconds?

I'm using Python to access this site and scrape the HTML: http://forum.toribash.com/tori_spy.php As you can see, if you access that webpage, the content changes over the duration of a few seconds. This is a page that shows the most recent posts on a forum and I'm making a Discord bot that would be a...
johnny mcdangerface
1

votes
0

answer
41

Views

Python Scraper - Request Post Function Not Returning Correct Page

I am working on my first website scraper and have ran into another issue. Below is my code. The website that is returned is the main page not the specific site for the parcel number I searched. Am I using the wrong html class to identify the search function? Or is there something missing in the Py...
Taylor29
1

votes
2

answer
47

Views

How can I take specific data from an API and export it without writing the data to a file?

I'm new to Python and using APIs and would love some help! Using the URL: http://api.worldbank.org/v2/countries?format=xml I want to gather each country's 'iso2Code' and 'name'. I want to be able to output the data in a cleaner XML file format or an array (but would love the knowledge on how to do b...
Sam Green
1

votes
1

answer
575

Views

python 3 requests 'ascii' codec can't encode character

this is the same question asked a million times on SO but I can't find a situation exactly like mine: import requests url = 'https://markets.ft.com/data/indices/tearsheet/summary?s=TR0000:TOR' response = requests.get(url) print (response.text) I get the following error: UnicodeEncodeError: 'ascii' c...
Ry John
1

votes
2

answer
225

Views

Requests.get failing when using inside function

I am using python requests library to get the data from adzuna api . if i try r = requests.get('http://api.adzuna.com/v1/api/jobs/gb/search/1?app_id=appID\ &app_key=appKEY&results_per_page=50&\ what=entry%20level&content-type=application/json') print r.text it is fetching me data But if i wrap...
anilkunchalaece
1

votes
1

answer
187

Views

Scrap a website with hidden parameters using the requests library

So I am trying to scrap the following webpage which protected by a login page. None the less when I run the code it keeps redirecting me to the login page. I think this might be due to the fact that the login page has a hidden parameter, though I don't really know how to integrate it into my payload...
Fozoro
1

votes
1

answer
72

Views

Trouble targeting a certain HTML tag with Python, requests, and BeautifulSoup

I'm writing an app in Python using requests and BeautifulSoup and have encountered a problem finding the text of a specific element. Essentially you enter a zip code and it requests a Bing search (Bing has easier to use search query URLs than Google) for '[zip code] weather'. I'm able to pull the...
joon_bug
1

votes
1

answer
256

Views

Getting rows of coordinates from text file and checking whether points are inside polygon

I have hundreds of coordinates (control points) in a text file in the following order (x y): 5480.000 -4880.000 5480.000 -4860.000 5480.000 -4840.000 5480.000 -4820.000 5480.000 -4800.000 5480.000 -4780.000 5480.000 -4760.000 5480.000 -4740.000 5480.000 -4720.000 5480.000 -4700.000 and polygon's coo...
kwak
1

votes
0

answer
48

Views

Can't catch unsuccessful parsing events when using requests to parse a list of image urls python3

Hi I am using the following function to parse a list of img reference urls. The reference urls are in format 'https://......strings/'; the parsed urls are in format 'http://..../img.jpg'. Input: a text file, each line is a reference url Expected output: a list of parsed urls. Parsing Rules: For eac...
CrazyFrog
1

votes
0

answer
51

Views

Who changed the response._content_consumed to True when debuging testing case of python requests

When I read the source code of the requests I try to debug the testing case for tracing code with vscode and python extension. When I run the test_digestauth_401_count_reset_on_redirect function in test_lowlevel.py, I encounter a weird question.After getting the first response of four from the inter...
Kevin00000000
1

votes
1

answer
256

Views

oAuth1.0a doesn't work for wordpress API from Python using requests, although call works from Postman

I am fairly new to programming and Python and I tried to POST to the wordpress REST API with OAuth1.0a to create a post to my blog. In Postman it all works well, so the credentials are ok. But my Python code below returns 401 - rest_cannot_create. I have been trying to make this work for 5 hours now...
tobsen
1

votes
1

answer
72

Views

Grab data from password protected ASP website using Python's requests

I'm trying to get the whole content of a password-protected ASP site using Python's requests. The programmer of the ASP Site told me that using PowerShell he is able to get the data using the following script: $c = $host.UI.PromptForCredential('Your Credentials', 'Enter Credentials','','') $r = Invo...
Ñhosko
1

votes
1

answer
678

Views

How to handle HTTP Status Codes across a Python application?

I'm trying to standardize my handling of HTTP status codes returned from various APIs in an effort to reduce the amount of code I copy across files. For my current application, I have a set of files that each contain one class that all derive inheritance from one master class. I'm using the Python...
Michael Bianchi
1

votes
0

answer
75

Views

Speeding up Pandas Apply for API Calls with Requests

I have a list of IP addresses in a df. These IP addresses are sent in GET requests to the ARIN database using requests, and I am interested in getting the organization or customer of that IP address. I am using a requests Session() inside of a requests-futures FuturesSession() to hopefully speed up...
OverflowingTheGlass
1

votes
0

answer
70

Views

what are siteminder hidden variables and how is data passed to them?

I am trying to access a Siteminder protected page using Python. Here's what I am doing: import requests r = requests.session() postUrl = 'http://example.gfm.com:1212/abc.htm' params = { 'user': 'faisal', 'password': 'faisal', 'target': '-SM-http%3a%2f%asdasdasdsad%2efmr%2ecom%3a8081%asdasdasdsad%as...
Faisal
1

votes
0

answer
156

Views

How to mimic an XHR request in Ahrefs.com with Python?

I am trying to scrape data from a page that comes from an XHR request. The request is made when the user clicks a link. I've been trying to mimic the request with my scraper, by using the hash in the link's 'onClick' attribute. I can get it to work for the first link, but I need to iterate each of t...
Jake 1986
1

votes
0

answer
36

Views

How to specify the link I want to download the file in python3

I found several questions related to my question but none helped me. I have to download .hdf files for a few years. This year I have at least 3 files per month. I tried to loop in the shell, but it will take 1000 years until I can download all the files, outside that will block my band. When I acces...
Lucas Fagundes
1

votes
1

answer
421

Views

Beautifulsoup scraping table from website with requests for pandas

I am trying to download the data on this website https://coinmunity.co/ ...in order to manipulate later it in Python or Pandas I have tried to do it directly to Pandas via Requests, but did not work, using this code: res = requests.get('https://coinmunity.co/') soup = BeautifulSoup(res.content, 'lxm...
skeitel
1

votes
1

answer
38

Views

How to handle more complicated post requests in python/php?

I have a python program which uses requests. I want to upload data via post request like this: payload = (('key1', 'value1'), ('key1', 'value2')) r = requests.post('https://www.example.com/test.php', payload) print(r.text) In php I receive this data for testing like this: In the example of the doc...
peter12395
1

votes
0

answer
79

Views

How to get the download link hidden behind a radio button?

I'm trying to download a csv file from this link. I know from this thread that we need to use the requests library to get the link by first submitting the form, in this case, to let the server know we want the csv file. However, since I'm not familiar with html and the previous example has now an up...
George Liu
1

votes
0

answer
298

Views

How to crawl ASP.NET web applications

I'm writing an application which needs to communicate with a ASP.NET website that doesn't have an API. I'm using Python 3.5.2 with Requests 2.18.4 to achieve my purpose. The problem is the site is using _dopostback() so I achieved my goal using Selenium but it was slow and it opened a browser for e...
Iman Kermani
1

votes
0

answer
93

Views

Using requests to access third-party API needing Oauth2 google authentication

For test automation purpose, I would like to use requests and requests-oauthlib library to access an API. The API use an oauth2 authentication with google account, however it's not a google API. As my test tool should be able to run unattended, I would like to be able to obtain an access token that...
ch7kor
1

votes
0

answer
482

Views

python requests requests.exceptions.ConnectionError: Failed to establish a new connection: [Errno 10060]

I'm trying to crawl a website. url = 'http://www.hellotrade.com/business/' headers = { 'User-Agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36', 'Connection':'close' } res = requests.get(url, headers = headers, timeout = 30) It runs perfec...
shihs
1

votes
0

answer
113

Views

Self-made Yahoo query for Finance data - error with pandas read_csv

Since the yahoo api does not work I tried a tiny python script to mimic a csv download. Unfortunately I get back: HTTP Error 401: Unauthorized But when I just copy paste the URL into the browser it downloads just fine. This is the code I use: import pandas as pd import datetime as dt import request...
Andreas K.
1

votes
0

answer
156

Views

ruby equivalent of python request session

I want to send requests over session object in Ruby to get data from authentication required API endpoint. I did this with Python as follows: import requests s = requests.session() s.post('https://test.url.com/mobile/User/[email protected]&password=123456abc') s.get('https://test.ur...
Mesut GÜNEŞ
1

votes
0

answer
69

Views

Scraping a website for specific data where URLs are inconsistent

I want to scrape http://www.narrpr.com/ for data, but I'm running into an issue. Most of the time, formatting URLs to access the specific pages you want to scrape is easy. However, in this instance, the URLs are formatted in the following fashion (for example): http://www.narrpr.com/homes/mo/indepen...
dougdimmadome
1

votes
0

answer
47

Views

How to download and install third party programs using Python

I got a task from my teacher to make a programm that should do following things: It should find a download url of offline installer of any product (for example Google Chrome); Find what is the latest version of the product; Then silently (without user interaction) install Chrome; Show keys of silent...
Sergei Petrov
1

votes
0

answer
56

Views

DKPro doesn't work with Jython due to missing requests package

I've followed the documentation of the installation of DKPro Core for natural language processing but have found that no matter what way I install it under Debian Jessie I cannot get it to work. It works fine with Groovy but not with Jython as documented at: https://dkpro.github.io/dkpro-core/pages/...
Eamonn Kenny

View additional questions