Skip to content

DOM traversal

Once you have an element, you often need the ones around it: its children, its siblings, elements inside a shadow root, or content inside an iframe. This guide covers moving through the DOM tree from a known starting point. To locate that starting element in the first place, see Element finding.

Get child elements

get_children_elements() returns the descendants of an element. max_depth controls how deep it goes (1 is direct children only), and tag_filter keeps only the tags you name.

from pydoll.sync import Chrome

def main():
    with Chrome() as browser:
        tab = browser.start()
        tab.go_to('https://books.toscrape.com')

        container = tab.find(class_name='row', tag_name='ol')

        direct = container.get_children_elements(max_depth=1)
        print(f'{len(direct)} direct children')

        # descendants up to 2 levels deep, links only
        links = container.get_children_elements(max_depth=2, tag_filter=['a'])
        print(f'{len(links)} links within two levels')

main()

import asyncio

from pydoll import Chrome


async def main():
    async with Chrome() as browser:
        tab = await browser.start()
        await tab.go_to('https://books.toscrape.com')

        container = await tab.find(class_name='row', tag_name='ol')

        direct = await container.get_children_elements(max_depth=1)
        print(f'{len(direct)} direct children')

        # descendants up to 2 levels deep, links only
        links = await container.get_children_elements(max_depth=2, tag_filter=['a'])
        print(f'{len(links)} links within two levels')

asyncio.run(main())

Get sibling elements

get_siblings_elements() returns the elements at the same level as your element, excluding it. tag_filter narrows the result to specific tags.

active = tab.find(class_name='active')

siblings = active.get_siblings_elements()
print(f'{len(siblings)} siblings')

link_siblings = active.get_siblings_elements(tag_filter=['a'])

active = await tab.find(class_name='active')

siblings = await active.get_siblings_elements()
print(f'{len(siblings)} siblings')

link_siblings = await active.get_siblings_elements(tag_filter=['a'])

Scoped search vs direct children

A scoped find() or query() searches all descendants of an element. When you want only the direct children, use the CSS child combinator > or an XPath step, which query() accepts:

container = tab.find(id='cards')

# every .card anywhere in the subtree
all_cards = container.find(class_name='card', find_all=True)

# only the .card elements that are direct children
direct_cards = container.query('> .card', find_all=True)

container = await tab.find(id='cards')

# every .card anywhere in the subtree
all_cards = await container.find(class_name='card', find_all=True)

# only the .card elements that are direct children
direct_cards = await container.query('> .card', find_all=True)

Use get_children_elements() when you want to explore structure or filter by tag; use a scoped find()/query() when you want elements matching specific attributes anywhere in the subtree.

Read text and attributes

From any element you can read its visible text and its HTML attributes:

book = tab.find(class_name='product_pod')

title = book.find(tag_name='h3')
print(title.text())                       # visible text

link = title.find(tag_name='a')
print(link.get_attribute('href'))             # an attribute value
print(link.get_attribute('title'))

book = await tab.find(class_name='product_pod')

title = await book.find(tag_name='h3')
print(await title.text())                       # visible text

link = await title.find(tag_name='a')
print(link.get_attribute('href'))             # an attribute value
print(link.get_attribute('title'))

text() returns the element's visible text; get_attribute(name) returns the attribute string, or None when the attribute is absent.

Shadow DOM

Many components hide their internals inside a shadow root, which regular DOM queries can't see. Access the shadow host, get its shadow root, then search inside it.

host = tab.find(id='my-component')
shadow = host.get_shadow_root()

button = shadow.query('.internal-btn')
button.click()

host = await tab.find(id='my-component')
shadow = await host.get_shadow_root()

button = await shadow.query('.internal-btn')
await button.click()

Inside a shadow root, use query() with CSS

find() and XPath are not supported on a ShadowRoot and raise NotImplementedError. Search shadow roots with query() and CSS selectors only.

query() inside a shadow root takes the usual find_all, timeout, and raise_exc parameters:

items = shadow.query('.item', find_all=True)
dynamic = shadow.query('#late', timeout=5, raise_exc=False)

items = await shadow.query('.item', find_all=True)
dynamic = await shadow.query('#late', timeout=5, raise_exc=False)

Web components nest, so a shadow root can contain another shadow host:

outer = tab.find(tag_name='outer-component')
outer_shadow = outer.get_shadow_root()

inner = outer_shadow.query('inner-component')
inner_shadow = inner.get_shadow_root()

deep = inner_shadow.query('.deep-btn')

outer = await tab.find(tag_name='outer-component')
outer_shadow = await outer.get_shadow_root()

inner = await outer_shadow.query('inner-component')
inner_shadow = await inner.get_shadow_root()

deep = await inner_shadow.query('.deep-btn')

Discover shadow roots on a page

When you don't know which shadow roots exist (debugging, or dynamic widgets like Cloudflare Turnstile), find_shadow_roots() returns all of them. Shadow hosts often load late, so pass timeout to poll until they appear:

shadow_roots = tab.find_shadow_roots(timeout=10)

for sr in shadow_roots:
    print(f'mode={sr.mode}, host={sr.host_element}')
    checkbox = sr.query('input[type="checkbox"]', raise_exc=False)
    if checkbox:
        checkbox.click()

shadow_roots = await tab.find_shadow_roots(timeout=10)

for sr in shadow_roots:
    print(f'mode={sr.mode}, host={sr.host_element}')
    checkbox = await sr.query('input[type="checkbox"]', raise_exc=False)
    if checkbox:
        await checkbox.click()

By default the search covers the main document (including same-origin iframes). Pass deep=True to also reach shadow roots inside cross-origin iframes (OOPIFs), which is what widgets like Turnstile use:

shadow_roots = tab.find_shadow_roots(deep=True, timeout=10)

shadow_roots = await tab.find_shadow_roots(deep=True, timeout=10)

Work inside an iframe

An iframe has its own DOM context. Find the iframe element, then call find() or query() on it; Pydoll routes the search into the frame automatically. Keep chaining for nested iframes.

iframe = tab.query('iframe.embedded-content', timeout=10)

button = iframe.find(tag_name='button', class_name='submit')
button.click()

# nested iframe
inner = iframe.find(tag_name='iframe')
link = inner.find(text='Download PDF')
link.click()

iframe = await tab.query('iframe.embedded-content', timeout=10)

button = await iframe.find(tag_name='button', class_name='submit')
await button.click()

# nested iframe
inner = await iframe.find(tag_name='iframe')
link = await inner.find(text='Download PDF')
await link.click()

For a full iframe guide including CAPTCHA frames and troubleshooting, see Iframes.

Screenshots inside iframes

tab.take_screenshot() captures the top-level page only. To capture iframe content, find an element inside the frame and call element.take_screenshot().

What's next