22,006 questions
-3
votes
0
answers
49
views
Get less verbose tracebacks from requests [closed]
I have this code:
import requests
url = 'some url see below'
try:
response = requests.request('GET', url, verify=False, timeout=5)
print('Response ok.')
except Exception as err:
errtxt = '...
0
votes
0
answers
38
views
UltraMsg API returns "Wrong token. Please provide token as a GET parameter" despite passing it in URL params in Python Requests
I am building a WhatsApp bot using FastAPI and UltraMsg. When my webhook receives a message, it generates an AI response and tries to send it back using the requests library.
Even though I am ...
0
votes
1
answer
73
views
python requests XHR - get response 405
Code written in Python on Mac.
Url retrived from inspect in google chrome.
Trying to fetch information from an XHR but does not succeed. The print at the end gives the response <Response [405]>. ...
4
votes
2
answers
120
views
How to fix "requests.exceptions.ConnectionError" when using requests library python?
I have been trying to make a scraper that downloads comics from Poorly Drawn Lines but I keep on getting a requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end ...
Best practices
0
votes
1
replies
79
views
Best Practices for Handling API Rate Limits in Python Applications
I am developing a Python application that interacts with a third-party API. The API enforces strict rate limits, and I want to make sure my application handles these efficiently without causing errors ...
Best practices
0
votes
6
replies
90
views
How to handle "JSON payload" templates for web requests to an API?
I'm working with a 3rd party API. The API uses an elasticsearch-like query language. So queries can get quiet complex. Often I have such a complex query (eg. long json string) which has 1 or maybe 2 ...
2
votes
2
answers
177
views
I have a problem with the request module in Automate Boring Stuff With Python - Chapter 13
I am new to python and programming in general and I got to Chapter 13 of Automate The Boring Stuff With Python which teaches Web Scraping, I succeeded in some projects but I'm having trouble getting ...
0
votes
1
answer
80
views
Is there a way to pull web data with python requests on a link guarded by edgepilot?
I am working on automatically storing reports I receive as a download link via email everyday. The cybersecurity company we work with uses Edgepilot (a security proxy) to protect our links so every ...
0
votes
1
answer
87
views
How to correctly handle no network access when using Python requests.get()
I'm retrieving data from a URL using reqests.get(). My device is connected by wifi. Occasionally, there is no internet access and my appliaction fails with the following error:
OSError: [Errno 101] ...
Best practices
1
vote
8
replies
114
views
Check if request.json() is NoneType
I am making API Calls and expect to get some null responses or NoneTypes. However I am trying to handle them gracefully.
This is the default behavior:
res = req.json()
res.get("key", {})....
0
votes
0
answers
73
views
REST API request succeeds via cURL but fails with 500 Internal Server Error when sent from Python (requests) inside ERPNext (Frappe)
I’m integrating ERPNext (Frappe Framework) with the Pakistan FBR DI API.
I’m facing a strange issue:
When I send the request using cURL, the invoice is accepted successfully.
When I send the exact ...
0
votes
1
answer
104
views
Loading images in HTML – Differentiation between no response and error response
I'm building a kind of gallery app. Users are adding images by inserting URLs into the database and the HTML page is then hotlinking those. Obviously, because of link rotting, the gallery is full of ...
Best practices
1
vote
3
replies
92
views
Best Practice for Certificate Authentication
If I need to authenticate with a certificate, I have a few options available:
Add the certificate to the bundle packaged with requests/certifi
Add the certificate to the system store and override ...
0
votes
0
answers
97
views
GitHub API endpoint no longer returning results for referral sources
When querying the referral sources endpoint on GitHub I'm only getting an empty list returned. I can get data for other endpoints in the same repo, even for other traffic endpoints. I'm expecting ...
3
votes
3
answers
120
views
Python requests.request returning garbage
Calling an api. For almost all calls the response comes clean but for a few there is garbage in the returned json. Executing the garbage one in the Firefox browser with the same parameters returns ...
0
votes
1
answer
93
views
Using HTTPkerberosauth with a javascript enabled web scraper
I'm working on integration tests for a web application that's running in a Docker container within our GitLab CI/CD pipeline. The application is a frontend that requires Kerberos/SPNEGO authentication ...
1
vote
0
answers
55
views
Connection management while maintaining requests.Session
What I found
While using the requests library for HTTP requests, I noticed that there were many TCP connections in the FIN_WAIT2 state on the server. Through code analysis, I found that these ...
0
votes
1
answer
168
views
POST requests from Cloud Run arriving as GET requests in my Django REST API
I have a serverless pipeline on Google Cloud. It consists of three total steps:
A video uploader that sends videos to Google Cloud Storage. Working fine.
An eventarc + pub/sub trigger that fires ...
3
votes
1
answer
101
views
Unable to post large file with python requests library
I try to post large file and read response as streaming with requests python library.
I have to send a large text file (2 million lines of log ) and receive response back as streaming. If I did by ...
0
votes
0
answers
86
views
python request getting different result than postman
I am attempting to improve my python by working on a side-project, just interacting with websites to look at avaialble golf tee times. Zero intention/plans to do anything commercial, this is just a ...
0
votes
0
answers
56
views
One request fails, but others don't; it also succeeds in terminal
I checked a lot of trouble-shoot topics, but didn't find what might be wrong here... (This is a second attempt at fixing this issue, but this time I dug deeper.)
I'm going off this API documentation: ...
1
vote
1
answer
253
views
Trouble scraping dynamic lottery results table – inconsistent parsing
I’ve been trying to scrape lottery results from a website that shows draws. The data is presented in a results table, but I keep running into strange issues where sometimes the numbers are captured ...
0
votes
1
answer
200
views
Congress API Auth Token Failure
Congress API Down
Over the past few days I've been building along term web app that interfaces with the official Congress.gov API (found here),but as of 20 minutes ago, the entire API seems to have ...
2
votes
0
answers
266
views
Post a local video file as a REEL using Instagram API in Python
I am able to post a video from a public URL as a REEL:
import requests
token = "..."
api_version="v23.0"
headers = {
"Content-Type": "application/json",
...
1
vote
2
answers
323
views
How to download protected PDF (ViewDocument) using Selenium or requests?
I'm trying to download a protected PDF from the New York State Courts NYSCEF website using Python. The URL looks like this:
https://iapps.courts.state.ny.us/nyscef/ViewDocument?docIndex=...
0
votes
2
answers
83
views
Is this Sumtracker API request correct?
I wish to use the Sumtracker API, and I want to connect and do a basic get for this endpoint.
Let's say that my API key is 1234567890abc. The code is here:
import requests
url = "https://...
1
vote
0
answers
67
views
MS Graph request returns 403
To automate processing of my e-mail messages on MS account, I have registered some application called "NameOfMyRegisteredApplication"; gave it all MS Graph Read rights; and then I ran the ...
0
votes
1
answer
120
views
How to emulate browser using requests in Python?
I am trying to download a files from API via Python requests. I cannot emulate a browser despite using the headers. I tried using requests.get(link, headers=headers) and it did not work. I also don't ...
3
votes
0
answers
147
views
Python requests library refusing to accept correct verify certificate
The following application performs a basic HTTP GET request against https://google.com, retrieves the peer certificate and saves it in PEM format into a file called cert.pem.
After that it attempts ...
0
votes
1
answer
117
views
python asyncio: allow user to choose fire-and-forget vs blocking-group-of-functions
I'd like to send an http request, and then operate on its response:
import requests
s=requests.Session()
....
(session setup...)
....
(url and params setup...)
....
r=s.get(url,params=params)
print('...
1
vote
1
answer
280
views
Why httpx timeout not working with stream response
I use httpx to make request to my LLM. Here is the code.
import httpx
import logging
import json
from httpx import TimeoutException
try:
content = ''
logging.info(f"request payload:{...
3
votes
2
answers
387
views
Can't scrape all the titles from the map on the webpage using the requests module
I'm trying to create a script in Python to scrape all available titles that show up when clicking on the black-colored area on the map in this website.
For example, when I click on a certain area on ...
1
vote
2
answers
71
views
How do I make a variable in the requests payload for python?
I have the following payload for the requests module in python and I want to make MAC and NAME a variable as I iterate through a csv of information? Is there a way to do this?
payload = '''{
...
1
vote
0
answers
89
views
Python requests failed to verify certificate, while urllib3 and curl do it successfully
Trying to done simple GET request. Site have self-signed certificate. I was export it using Firefox (downloaded chained certificates) as a "pem" file.
Here versions of libraries:
[user@host]$...
0
votes
0
answers
95
views
how do i complete the Checkout bot at the payment process that uses tokens? (using python requests)
I'm working on a Python bot that monitors a Shopify webshop, adds a product to the cart, and tries to continue to checkout. I'm using requests.Session() and BeautifulSoup to handle the stock check, ...
0
votes
1
answer
163
views
How can I resume downloading file from where it left off without using the HTTP Range header and avoid re-downloading already received bytes? [closed]
I’m working on a downloader in Python using requests. My goal is to resume downloading large files after an interruption (e.g., crash or network failure).
✅ My constraints:
I already have part of the ...
0
votes
1
answer
368
views
Python bypass cloudflare, connecting to websocket
I need to connect to a websocket in Python, but the problem is that the server where this websocket is located has Cloudflare protection, which blocks the connection to it and returns a 403 error. Try ...
0
votes
1
answer
64
views
How to parse link extensions out of a script tag in BeautifulSoup with RegEx in Python
I am trying to parse link extensions out of a script tag in a webpage loaded with requests. I'm able to request the page and load the script tag as a tag element in BeautifulSoup, it looks like this:
{...
1
vote
0
answers
24
views
Pytest can't simulate stale API chunk in deduplication logic — fetch_recent_klines() never returns [] as expected
I'm testing a function `fetch_recent_klines()` that fetches 1m kline data from the Bybit V5 API.
It's designed to **halt and return `[]`** if it receives the same chunk of data twice — to detect stale ...
0
votes
1
answer
36
views
Using Requests to scrape: name, SKU, Price, Discount price, Can´t obtain data
import requests
from lxml import html
from selenium.webdriver.common.by import By
headers = {
"user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) ...
2
votes
1
answer
140
views
Is this safe when it come to prevent requests from outside the own website?
I am intending to specify the single origins where I want my different endpoints to receive requests from. After looking at different options I thought that this might be the simplest way, and it ...
0
votes
0
answers
94
views
Python json.loads() truncating inner data
I'm having issues when parsing JSON data from an API request.
I get all pages from the API request using this code:
try:
limit = 100 # Max limit per page, can be reduced as needed
...
0
votes
0
answers
55
views
Make data available across views and request flask
I have a platform that I'm building, this platformed is designed to be very modular and each component manages its state, now one key component of my platform is a JWT Token that I have linked to a ...
0
votes
1
answer
134
views
Python API call freezes instead of raising TimeoutError despite short timeout
I have a very sensitive production script that I’ve been running consistently for the past 4 years. This specific function (runAPI) is called about 100 times every day and has never caused any issues —...
1
vote
1
answer
102
views
Python Request Session throws connection errors after exactly six hours
TLDR: Python request session transmits data to API successfully for six hours and then suddenly only receives ConnectionErrors. Rate limits and other basic errors shouldn't be the issue as the ...
1
vote
1
answer
69
views
Can I be sure of the order requests are sent in in a Python requests session?
I am using Python requests to communicate with an external API. When I do something like:
with requests.Session() as sess:
resp1 = sess.put(
api_url + f"stuff/{id}/endpoint1",
...
0
votes
0
answers
78
views
Why do i keep getting bad request when trying to send GraphApi batch request
My goal is to send a batch request of with the conversation id so I could get all the conversations that the user have in their inbox , I tried to read the docs many times but I find their docs ...
0
votes
0
answers
48
views
AIOHTTP and ASYNCIO GRAPHQL request is replacing session_id in Cookie with another value and causing the request to return nothing but 200 status code
So I'm messing around with the leetcode GRAPHQL api and trying to hit up company endpoints, in the follow synchronous code that I wrote, it works perfectly and the cookie is also saved correctly. It ...
1
vote
1
answer
73
views
Request through proxy - connect step - header seems not sent
I have to make http requests through a proxy which requires ntlm authentication and a specific user agent.
Here is the relevant part of the code :
headers = {"User-Agent": "...
0
votes
0
answers
512
views
"Connection broken: IncompleteRead(50454677 bytes read, 35845891 more expected)" upon fetching large Collection from Postman API
Upon fecthing a large collection item from Postman's API I get the following error:
RuntimeError: Chunked transfer error: ('Connection broken:
IncompleteRead(50454677 bytes read, 35845891 more ...