Skip to content

Repository files navigation

Python Proxy Headers

Documentation Status PyPI version

Extensions for Python HTTP libraries that send custom proxy headers on HTTPS CONNECT and read proxy response headers back. Works with any proxy provider; works best with ProxyMesh.

The Problem

When making HTTPS requests through a proxy, the connection is established via a CONNECT tunnel. During this process:

  1. Sending proxy headers - Most Python HTTP libraries don't provide an easy way to send custom proxy headers (like X-ProxyMesh-Country) to the proxy server during the CONNECT handshake.

  2. Reading proxy response headers - The proxy's CONNECT response headers are typically discarded, making it impossible to read proxy response headers (like X-ProxyMesh-IP) that the proxy sends back.

This library solves both problems for popular Python HTTP libraries.

Supported Libraries

Library Module Use Case
urllib3 urllib3_proxy_manager Low-level HTTP client
requests requests_adapter Simple HTTP requests
aiohttp aiohttp_proxy Async HTTP client
httpx httpx_proxy Modern HTTP client
pycurl pycurl_proxy libcurl bindings
cloudscraper cloudscraper_proxy Cloudflare bypass
autoscraper autoscraper_proxy Automatic web scraping

Installation

pip install python-proxy-headers

Or install the client extra you need. Each extra pins a range that CI tests:

pip install 'python-proxy-headers[requests]'
pip install 'python-proxy-headers[urllib3]'
pip install 'python-proxy-headers[httpx]'
pip install 'python-proxy-headers[aiohttp]'
pip install 'python-proxy-headers[pycurl]'
pip install 'python-proxy-headers[cloudscraper]'
pip install 'python-proxy-headers[autoscraper]'

The httpx extra is httpx>=0.28,<1. Importing python_proxy_headers.httpx_proxy on httpx 1.x raises ImportError. See the httpx docs. Proxy header names are specific to your provider; ProxyMesh documents theirs at docs.proxymesh.com.

Note: This package has no required dependencies. Install only the extra you need.

Quick Start

requests

from python_proxy_headers.requests_adapter import ProxySession

with ProxySession(proxy_headers={'X-ProxyMesh-Country': 'US'}) as session:
    session.proxies = {'https': 'http://user:pass@proxy.example.com:8080'}
    response = session.get('https://httpbin.org/ip')
    
    # Proxy response headers are merged into response.headers
    print(response.headers.get('X-ProxyMesh-IP'))

httpx

from python_proxy_headers.httpx_proxy import get

response = get(
    'https://httpbin.org/ip',
    proxy='http://user:pass@proxy.example.com:8080'
)

# Proxy CONNECT response headers are merged into response.headers
print(response.headers.get('X-ProxyMesh-IP'))

aiohttp

import asyncio
from python_proxy_headers.aiohttp_proxy import ProxyClientSession

async def main():
    async with ProxyClientSession() as session:
        async with session.get(
            'https://httpbin.org/ip',
            proxy='http://user:pass@proxy.example.com:8080'
        ) as response:
            # Proxy response headers merged into response.headers
            print(response.headers.get('X-ProxyMesh-IP'))

asyncio.run(main())

pycurl (low-level)

import pycurl
from python_proxy_headers.pycurl_proxy import set_proxy_headers, HeaderCapture

c = pycurl.Curl()
c.setopt(pycurl.URL, 'https://httpbin.org/ip')
c.setopt(pycurl.PROXY, 'http://proxy.example.com:8080')

# Add these two lines to any existing pycurl code
set_proxy_headers(c, {'X-ProxyMesh-Country': 'US'})
capture = HeaderCapture(c)

c.perform()

print(capture.proxy_headers)  # Proxy response headers from the CONNECT response
c.close()

cloudscraper

from python_proxy_headers.cloudscraper_proxy import create_scraper

# Drop-in replacement for cloudscraper.create_scraper()
scraper = create_scraper(proxy_headers={'X-ProxyMesh-Country': 'US'})
scraper.proxies = {'https': 'http://proxy.example.com:8080'}

response = scraper.get('https://example.com')
# All CloudScraper features (Cloudflare bypass) preserved

Testing

A test harness is included to verify proxy header functionality:

# Header checks and private-hook checks (no proxy credentials)
python test_header_security.py
python test_httpx_guard.py
python test_client_internals.py

# Set your proxy
export PROXY_URL='http://user:pass@proxy.example.com:8080'

# Test all modules
python test_proxy_headers.py

# Test specific modules
python test_proxy_headers.py requests httpx

# Verbose output (show header values)
python test_proxy_headers.py -v

Documentation

For detailed documentation, API reference, and more examples:

Other languages

Why ProxyMesh

ProxyMesh created this library to help customers use custom headers to control proxy behavior, as documented at docs.proxymesh.com. It works with any proxy that supports custom headers on CONNECT.

License

MIT License