Skip to content

Extraction

The structured-extraction API: define a model, get typed and validated data back. Learn more in Structured extraction.

pydoll.extractor.model.ExtractionModel

Bases: BaseModel

Base class for declarative extraction models.

Inherits from pydantic.BaseModel, gaining automatic validation, type coercion, serialization (model_dump, model_dump_json), and JSON Schema generation (model_json_schema).

Subclasses define fields using Field() descriptors with selectors and/or semantic descriptions. The extraction engine uses this metadata to extract structured data from web pages.

Example::

class Article(ExtractionModel):
    title: str = Field(selector='h1', description='Article title')
    author: str = Field(selector='.author', description='Author name')

model_config class-attribute instance-attribute 

model_config = ConfigDict(arbitrary_types_allowed=True)

get_extraction_fields classmethod 

get_extraction_fields()

Get extraction metadata for all fields, collecting lazily on first access.

Each subclass gets its own cache, even if a parent class has already been collected. This ensures inherited fields are included correctly.

RETURNS DESCRIPTION
dict[str, ExtractionMetadata]

Dictionary mapping field name to ExtractionMetadata.
RAISES DESCRIPTION
InvalidExtractionModel

If a field has metadata but lacks both selector and description.

pydoll.extractor.field.Field 

Field(*, selector=None, attribute=None, description=None, default=PydanticUndefined, transform=None)

Define extraction metadata for a model field.

Wraps pydantic.Field() and registers ExtractionMetadata for the engine. Auto-detects CSS vs XPath from selector syntax (same logic as Tab.query()).

At least one of selector or description must be provided: - selector only: extracted via CSS/XPath. - description only: metadata for future LLM extraction. - both: CSS extraction with LLM fallback in future auto strategy.

PARAMETER DESCRIPTION
selector

CSS or XPath selector (auto-detected, like Tab.query()).

TYPE: Optional[str] DEFAULT: None

attribute

HTML attribute to extract (default: innerText).

TYPE: Optional[str] DEFAULT: None

description

Semantic description of the field.

TYPE: Optional[str] DEFAULT: None

default

Default value if extraction fails. PydanticUndefined means required.

TYPE: object DEFAULT: PydanticUndefined

transform

Post-processing callable applied to raw extracted string.

TYPE: Optional[Callable[[str], Union[str, int, float, bool, object]]] DEFAULT: None

RETURNS DESCRIPTION
FieldInfo

Pydantic FieldInfo with extraction registry key in json_schema_extra.
RAISES DESCRIPTION
InvalidExtractionModel

If neither selector nor description is provided.