Skip to content

Browser Requests

The requests module provides HTTP request capabilities within the browser context, enabling seamless API calls that inherit the browser's session state, cookies, and authentication.

Overview

The browser requests module offers a requests-like interface for making HTTP calls directly within the browser's JavaScript context. This approach provides several advantages over traditional HTTP libraries:

  • Session inheritance: Automatic cookie, authentication, and CORS handling
  • Browser context: Requests execute in the same security context as the page
  • No session juggling: Eliminate the need to transfer cookies and tokens between automation and API calls
  • SPA compatibility: Perfect for Single Page Applications with complex authentication flows

Request Class

The main interface for making HTTP requests within the browser context.

pydoll.browser.requests.request.Request 

Request(tab)

High-level interface for making HTTP requests using the browser's fetch API.

This class provides a requests-like interface that executes HTTP requests in the browser's JavaScript context. All requests inherit the browser's current session state including cookies, authentication headers, and other automatic browser behaviors. This allows for seamless interaction with websites that require authentication or have complex cookie management.

Key Features: - Executes requests in the browser's JavaScript context using fetch API - Automatically includes browser cookies and session state - Preserves browser's security context and CORS policies - Captures both request and response headers for analysis - Supports all standard HTTP methods (GET, POST, PUT, DELETE, etc.)

Note: - Headers passed to methods are additional headers, not replacements - Browser's automatic headers (User-Agent, Accept, etc.) are preserved - Cookies are managed automatically by the browser

Initialize a new Request instance bound to a browser tab.

PARAMETER DESCRIPTION
tab

The browser tab instance where requests will be executed. This tab provides the JavaScript execution context and maintains the browser's session state (cookies, authentication, etc.).

TYPE: Tab

tab instance-attribute 

tab = tab

_network_events_enabled instance-attribute 

_network_events_enabled = False

_requests_sent instance-attribute 

_requests_sent = []

_requests_received instance-attribute 

_requests_received = []

request async 

request(method, url, params=None, data=None, json=None, headers=None, **kwargs)

Execute an HTTP request in the browser's JavaScript context.

This method uses the browser's fetch API to make requests, inheriting all browser session state including cookies, authentication, and security context. The request is executed as if made by the browser itself.

PARAMETER DESCRIPTION
method

HTTP method (GET, POST, PUT, DELETE, etc.). Case insensitive.

TYPE: str

url

Target URL for the request. Can be relative or absolute.

TYPE: str

params

Query parameters to append to the URL. These are URL-encoded and merged with any existing query string in the URL.

TYPE: Optional[dict[str, str]] DEFAULT: None

data

Request body data. Behavior depends on type: - dict/list/tuple: URL-encoded as form data (application/x-www-form-urlencoded) - str/bytes: Sent as-is with no Content-Type modification Mutually exclusive with 'json' parameter.

TYPE: Optional[Union[dict, list, tuple, str, bytes]] DEFAULT: None

json

Data to be JSON-serialized as request body. Automatically sets Content-Type to application/json. Mutually exclusive with 'data'.

TYPE: Optional[dict[str, Any]] DEFAULT: None

headers

Additional headers to include. These are ADDED to browser's automatic headers, not replacements. Format: [{'name': 'X-Custom', 'value': 'value'}]

TYPE: Optional[list[HeaderEntry]] DEFAULT: None

**kwargs

Additional fetch API options (e.g., credentials, mode, cache).

DEFAULT: {}

RETURNS DESCRIPTION
Response

Response object containing status, headers, content, and cookies from
Response

both the request and response phases.
RAISES DESCRIPTION
HTTPError

If the request execution fails or network error occurs.
Note
  • Browser cookies are automatically included
  • CORS policies are enforced by the browser
  • Authentication headers are preserved from browser session

get async 

get(url, params=None, **kwargs)

Execute a GET request for retrieving data.

PARAMETER DESCRIPTION
url

Target URL to retrieve data from.

TYPE: str

params

Query parameters to append to URL.

TYPE: Optional[dict[str, str]] DEFAULT: None

**kwargs

Additional fetch options.

DEFAULT: {}

RETURNS DESCRIPTION
Response

Response object with retrieved data.

post async 

post(url, data=None, json=None, **kwargs)

Execute a POST request for creating or submitting data.

PARAMETER DESCRIPTION
url

Target URL for data submission.

TYPE: str

data

Form data to submit (URL-encoded).

TYPE: Optional[Union[dict, list, tuple, str, bytes]] DEFAULT: None

json

JSON data to submit.

TYPE: Optional[dict[str, Any]] DEFAULT: None

**kwargs

Additional fetch options.

DEFAULT: {}

RETURNS DESCRIPTION
Response

Response object with server's response to the submission.

put async 

put(url, data=None, json=None, **kwargs)

Execute a PUT request for updating/replacing resources.

PARAMETER DESCRIPTION
url

Target URL of resource to update.

TYPE: str

data

Form data for the update.

TYPE: Optional[Union[dict, list, tuple, str, bytes]] DEFAULT: None

json

JSON data for the update.

TYPE: Optional[dict[str, Any]] DEFAULT: None

**kwargs

Additional fetch options.

DEFAULT: {}

RETURNS DESCRIPTION
Response

Response object confirming the update operation.

patch async 

patch(url, data=None, json=None, **kwargs)

Execute a PATCH request for partial resource updates.

PARAMETER DESCRIPTION
url

Target URL of resource to partially update.

TYPE: str

data

Form data with changes to apply.

TYPE: Optional[Union[dict, list, tuple, str, bytes]] DEFAULT: None

json

JSON data with changes to apply.

TYPE: Optional[dict[str, Any]] DEFAULT: None

**kwargs

Additional fetch options.

DEFAULT: {}

RETURNS DESCRIPTION
Response

Response object confirming the partial update.

delete async 

delete(url, **kwargs)

Execute a DELETE request for removing resources.

PARAMETER DESCRIPTION
url

Target URL of resource to delete.

TYPE: str

**kwargs

Additional fetch options.

DEFAULT: {}

RETURNS DESCRIPTION
Response

Response object confirming the deletion.

head async 

head(url, **kwargs)

Execute a HEAD request to retrieve only response headers.

Useful for checking resource existence, size, or modification date without downloading the full content.

PARAMETER DESCRIPTION
url

Target URL to check headers for.

TYPE: str

**kwargs

Additional fetch options.

DEFAULT: {}

RETURNS DESCRIPTION
Response

Response object with headers but no body content.

options async 

options(url, **kwargs)

Execute an OPTIONS request to check allowed methods and capabilities.

Used for CORS preflight checks and discovering server capabilities.

PARAMETER DESCRIPTION
url

Target URL to check options for.

TYPE: str

**kwargs

Additional fetch options.

DEFAULT: {}

RETURNS DESCRIPTION
Response

Response object with allowed methods and CORS headers.

_build_url_with_params staticmethod 

_build_url_with_params(url, params)

Build final URL with query parameters.

_build_request_options 

_build_request_options(method, headers, json, data, **kwargs)

Build request options dictionary.

_add_request_body 

_add_request_body(options, json, data)

Add request body and appropriate Content-Type header.

_handle_json_options staticmethod 

_handle_json_options(options, json)

Handle JSON options.

_handle_data_options staticmethod 

_handle_data_options(options, data)

Handle data options.

_execute_fetch_request async 

_execute_fetch_request(url, options)

Execute the fetch request using browser's runtime.

_build_response staticmethod 

_build_response(result, response_headers, request_headers, cookies)

Build Response object from fetch result.

_register_callbacks async 

_register_callbacks()

Register network event listeners to capture request/response metadata.

Sets up CDP event listeners to capture all network activity during the request execution. This includes both outgoing request data and incoming response data, which are used for header and cookie extraction.

Note

Network events are only enabled if not already active on the tab.

_clear_callbacks async 

_clear_callbacks()

Clean up network event listeners and disable network monitoring.

Removes all registered event callbacks and disables network events if they were enabled by this request instance.

_extract_received_headers 

_extract_received_headers()

Extract headers from response network events.

RETURNS DESCRIPTION
list[HeaderEntry]

List of headers received from the server during response.

_extract_sent_headers 

_extract_sent_headers()

Extract headers from request network events.

RETURNS DESCRIPTION
list[HeaderEntry]

List of headers that were actually sent in the request.

_extract_headers_from_events staticmethod 

_extract_headers_from_events(events, event_extractors)

Extract headers from network events using appropriate extractors.

PARAMETER DESCRIPTION
events

List of network events to process.

TYPE: Union[list[RequestSentEvent], list[RequestReceivedEvent]]

event_extractors

Mapping of event keys to header extraction functions.

TYPE: dict[str, Callable[[Any], list[HeaderEntry]]]

RETURNS DESCRIPTION
list[HeaderEntry]

Deduplicated list of headers from all matching events.
Note

Headers are deduplicated based on name-value pairs to avoid duplicate entries from multiple event types.

_extract_request_sent_headers 

_extract_request_sent_headers(params)

Extract headers from main request event.

PARAMETER DESCRIPTION
params

Event parameters containing request details.

TYPE: RequestWillBeSentEventParams

RETURNS DESCRIPTION
list[HeaderEntry]

List of headers that were sent with the request.

_extract_request_sent_extra_info_headers 

_extract_request_sent_extra_info_headers(params)

Extract headers from extra request info event.

This event contains additional header information that may not be present in the main request event, such as security-related headers.

PARAMETER DESCRIPTION
params

Extra info event parameters containing additional headers.

TYPE: RequestWillBeSentExtraInfoEventParams

RETURNS DESCRIPTION
list[HeaderEntry]

List of additional headers sent with the request.

_extract_response_received_headers 

_extract_response_received_headers(params)

Extract headers from main response event.

PARAMETER DESCRIPTION
params

Event parameters containing response details.

TYPE: ResponseReceivedEventParams

RETURNS DESCRIPTION
list[HeaderEntry]

List of headers received from the server.

_extract_response_received_extra_info_headers 

_extract_response_received_extra_info_headers(params)

Extract headers from extra response info event.

This event contains additional response header information, including Set-Cookie headers and security-related headers that may be filtered from the main response event.

PARAMETER DESCRIPTION
params

Extra info event parameters containing additional headers.

TYPE: ResponseReceivedExtraInfoEventParams

RETURNS DESCRIPTION
list[HeaderEntry]

List of additional headers received from the server.

_convert_dict_to_header_entries staticmethod 

_convert_dict_to_header_entries(headers_dict)

Convert header dictionary to standardized HeaderEntry format.

PARAMETER DESCRIPTION
headers_dict

Dictionary mapping header names to values.

TYPE: dict

RETURNS DESCRIPTION
list[HeaderEntry]

List of HeaderEntry objects with 'name' and 'value' keys.

_extract_set_cookies 

_extract_set_cookies()

Extract and parse all Set-Cookie headers from response events.

Processes response events to find Set-Cookie headers and converts them into structured cookie objects. Handles multiple Set-Cookie headers and multi-line cookie declarations.

RETURNS DESCRIPTION
list[CookieParam]

List of unique cookies extracted from Set-Cookie headers.

_filter_response_extra_info_events 

_filter_response_extra_info_events()

Filter network events to find those containing Set-Cookie information.

RETURNS DESCRIPTION
list[RequestReceivedEvent]

List of events that contain extra response information including cookies.

 
_parse_set_cookie_header(set_cookie_header)

Parse a Set-Cookie header value into individual cookie objects.

Handles both single and multi-line Set-Cookie headers, extracting cookie name-value pairs while ignoring attributes like Path, Domain, etc.

PARAMETER DESCRIPTION
set_cookie_header

Raw Set-Cookie header value from HTTP response.

TYPE: str

RETURNS DESCRIPTION
list[CookieParam]

List of parsed cookie objects with name and value.

 
_parse_cookie_line(line)

Parse a single cookie line to extract name and value.

Extracts only the cookie name and value, ignoring all cookie attributes like Path, Domain, Secure, HttpOnly, etc. Rejects cookies with empty names.

PARAMETER DESCRIPTION
line

Single line from Set-Cookie header.

TYPE: str

RETURNS DESCRIPTION
Optional[CookieParam]

CookieParam object with name and value, or None if parsing fails or name is empty.

_add_unique_cookies staticmethod 

_add_unique_cookies(cookies, new_cookies)

Add cookies to list while avoiding duplicates.

PARAMETER DESCRIPTION
cookies

Existing list of cookies to add to.

TYPE: list[CookieParam]

new_cookies

New cookies to add if not already present.

TYPE: list[CookieParam]

_convert_header_entries_to_dict staticmethod 

_convert_header_entries_to_dict(headers)

Convert HeaderEntry objects to a plain dictionary format.

Used for preparing headers for the JavaScript fetch API which expects a simple object mapping header names to values.

PARAMETER DESCRIPTION
headers

List of HeaderEntry objects with 'name' and 'value' keys.

TYPE: list[HeaderEntry]

RETURNS DESCRIPTION
dict[str, str]

Dictionary mapping header names to values.

Response Class

Represents the response from HTTP requests, providing a familiar interface similar to the requests library.

pydoll.browser.requests.response.Response 

Response(status_code, content=b'', text='', json=None, response_headers=None, request_headers=None, cookies=None, url='')

HTTP response object for browser-based fetch requests.

This class provides a standardized interface for handling HTTP responses obtained through the browser's fetch API. It mimics the requests.Response interface while preserving all browser-specific metadata including cookies, headers, and network timing information.

Key Features: - Compatible with requests.Response API for easy migration - Preserves both request and response headers for analysis - Automatic cookie extraction from Set-Cookie headers - Lazy JSON parsing with caching - Browser-context aware (respects CORS, security policies) - Content available in multiple formats (text, bytes, JSON)

The response contains all data captured during the browser's fetch execution, including redirects, authentication flows, and any browser-applied transformations.

Initialize a new Response instance with browser fetch results.

PARAMETER DESCRIPTION
status_code

HTTP status code returned by the server (e.g., 200, 404, 500).

TYPE: int

content

Raw response body as bytes. Used for binary data or when text encoding is uncertain.

TYPE: bytes DEFAULT: b''

text

Response body as decoded string. Pre-decoded by browser's fetch API.

TYPE: str DEFAULT: ''

json

Pre-parsed JSON data if response Content-Type was application/json. If None, json() method will attempt to parse from text on demand.

TYPE: Optional[dict[str, Any]] DEFAULT: None

response_headers

Headers received from the server, including Set-Cookie, Content-Type, and any custom headers sent by the server.

TYPE: Optional[list[HeaderEntry]] DEFAULT: None

request_headers

Headers that were actually sent in the request, including browser-generated headers (User-Agent, Accept, etc.) and custom headers.

TYPE: Optional[list[HeaderEntry]] DEFAULT: None

cookies

Cookies extracted from Set-Cookie headers during the response. These represent new/updated cookies from this specific request.

TYPE: Optional[list[CookieParam]] DEFAULT: None

url

Final URL after any redirects. May differ from original request URL if the server performed redirects during the request.

TYPE: str DEFAULT: ''

_status_code instance-attribute 

_status_code = status_code

_content instance-attribute 

_content = content

_text instance-attribute 

_text = text

_json instance-attribute 

_json = json

_response_headers instance-attribute 

_response_headers = response_headers or []

_request_headers instance-attribute 

_request_headers = request_headers or []

_cookies instance-attribute 

_cookies = cookies or []

_url instance-attribute 

_url = url

_ok instance-attribute 

_ok = status_code in STATUS_CODE_RANGE_OK

ok property 

ok

Check if the request was successful (2xx status codes).

RETURNS DESCRIPTION
bool

True if status code is in the 200-399 range, False otherwise.
Note

This follows HTTP conventions where 2xx codes indicate success and 3xx codes indicate redirection (still considered "ok").

cookies property 

cookies

Get cookies that were set by the server during this response.

RETURNS DESCRIPTION
list[CookieParam]

List of cookies extracted from Set-Cookie headers. Each cookie
list[CookieParam]

contains name and value, with cookie attributes (Path, Domain, etc.)
list[CookieParam]

automatically handled by the browser.
Note

These are only NEW/UPDATED cookies from this response. Existing browser cookies are managed automatically by the browser context.

request_headers property 

request_headers

Get headers that were actually sent in the HTTP request.

RETURNS DESCRIPTION
list[HeaderEntry]

List of headers sent to the server, including both custom headers
list[HeaderEntry]

provided by the user and automatic headers added by the browser
list[HeaderEntry]

(User-Agent, Accept, Authorization, etc.).
Note

This shows the ACTUAL headers sent, which may differ from what was originally specified due to browser modifications.

headers property 

headers

Get headers received from the server in the HTTP response.

RETURNS DESCRIPTION
list[HeaderEntry]

List of response headers sent by the server, including standard
list[HeaderEntry]

headers (Content-Type, Content-Length, etc.) and any custom headers.
Note

Some security-sensitive headers may be filtered by the browser and not appear in this list due to CORS policies.

status_code property 

status_code

Get the HTTP status code returned by the server.

RETURNS DESCRIPTION
int

Integer status code (e.g., 200 for OK, 404 for Not Found, 500 for Server Error).

text property 

text

Get the response content as a decoded string.

RETURNS DESCRIPTION
str

Response body decoded as UTF-8 string. If no text was provided
str

during initialization, it will be decoded from the raw content.
Note

Decoding uses 'replace' error handling to avoid crashes on invalid UTF-8 sequences.

content property 

content

Get the raw response content as bytes.

RETURNS DESCRIPTION
bytes

Unmodified response body as bytes. Useful for binary data
bytes

(images, files, etc.) or when you need to handle encoding manually.

url property 

url

Get the final URL of the response after any redirects.

RETURNS DESCRIPTION
str

The final URL that was accessed, which may differ from the
str

original request URL if redirects occurred.

json 

json()

Parse and return the response content as JSON data.

Attempts to parse the response text as JSON. Uses caching to avoid re-parsing the same content multiple times.

RETURNS DESCRIPTION
Union[dict[str, Any], list]

Parsed JSON data as dictionary, list, or other JSON-compatible type.
RAISES DESCRIPTION
ValueError

If the response content is not valid JSON or if parsing fails.
Note
  • Uses lazy parsing: JSON is only parsed when first accessed
  • Subsequent calls return cached result for better performance
  • If JSON was pre-parsed during initialization, that result is returned

raise_for_status 

raise_for_status()

Raise an HTTPError if the response indicates an HTTP error status.

Checks the status code and raises an exception for client errors (4xx) and server errors (5xx). Successful responses (2xx) and redirects (3xx) do not raise an exception.

RAISES DESCRIPTION
HTTPError

If status code is 400 or higher, indicating an error.
Note

This method is compatible with requests.Response.raise_for_status() for easy migration from the requests library.

Usage Examples

Basic HTTP Methods

from pydoll.browser.chromium import Chrome

async with Chrome() as browser:
    tab = await browser.start()
    await tab.go_to("https://api.example.com")

    # GET request
    response = await tab.request.get("/users/123")
    user_data = await response.json()

    # POST request
    response = await tab.request.post("/users", json={
        "name": "John Doe",
        "email": "john@example.com"
    })

    # PUT request with headers
    response = await tab.request.put("/users/123", 
        json={"name": "Jane Doe"},
        headers={"Authorization": "Bearer token123"}
    )

Response Handling

# Check response status
if response.ok:
    print(f"Success: {response.status_code}")
else:
    print(f"Error: {response.status_code}")
    response.raise_for_status()  # Raises HTTPError for 4xx/5xx

# Access response data
text_data = response.text
json_data = await response.json()
raw_bytes = response.content

# Inspect headers and cookies
print("Response headers:", response.headers)
print("Request headers:", response.request_headers)
for cookie in response.cookies:
    print(f"Cookie: {cookie.name}={cookie.value}")

Advanced Features

# Request with custom headers and parameters
response = await tab.request.get("/search", 
    params={"q": "python", "limit": 10},
    headers={
        "User-Agent": "Custom Bot 1.0",
        "Accept": "application/json"
    }
)

# File upload simulation
response = await tab.request.post("/upload",
    data={"description": "Test file"},
    files={"file": ("test.txt", "file content", "text/plain")}
)

# Form data submission
response = await tab.request.post("/login",
    data={"username": "user", "password": "pass"}
)

Integration with Tab

The request functionality is accessed through the tab.request property, which provides a singleton Request instance for each tab:

# Each tab has its own request instance
tab1 = await browser.get_tab(0)
tab2 = await browser.new_tab()

# These are separate Request instances
request1 = tab1.request  # Request bound to tab1
request2 = tab2.request  # Request bound to tab2

# Requests inherit the tab's context
await tab1.go_to("https://site1.com")
await tab2.go_to("https://site2.com")

# These requests will have different cookie/session contexts
response1 = await tab1.request.get("/api/data")  # Uses site1.com cookies
response2 = await tab2.request.get("/api/data")  # Uses site2.com cookies

Hybrid Automation

This module is particularly powerful for hybrid automation scenarios where you need to combine UI interactions with API calls. For example, log in through the UI, then use the authenticated session for API calls without manually handling cookies or tokens.