Use Playwright
In this guide, you'll learn how to use Playwright for web scraping in your Apify Actors.
Introduction
Playwright is a tool for web automation and testing that can also be used for web scraping. It allows you to control a web browser programmatically and interact with web pages just as a human would.
Some of the key features of Playwright for web scraping include:
- Cross-browser support - Playwright supports Chromium, Firefox, and WebKit with a single API, ensuring consistent behavior across all browsers.
- Auto-waiting - Playwright automatically waits for elements to be ready before performing actions, reducing flaky scripts and eliminating the need for manual sleep calls.
- Headless and headful modes - Playwright can run with or without a visible browser window, making it suitable for both local development and containerized environments.
- Powerful selectors - Playwright provides CSS selectors, XPath, text matching, and its own resilient locator API for targeting elements on a page.
- Network interception - Playwright can intercept and modify network requests, allowing you to block unnecessary resources or mock API responses during scraping.
To create Actors which use Playwright, start from the Playwright & Python Actor template.
On the Apify platform, the Actor will already have Playwright and the necessary browsers preinstalled in its Docker image, including the tools and setup necessary to run browsers in headful mode.
When running the Actor locally, you'll need to finish the Playwright setup yourself before you can run the Actor.
- Linux / macOS
- Windows
source .venv/bin/activate
playwright install --with-deps
.venv\Scripts\activate
playwright install --with-deps
Example Actor
This is a simple Actor that recursively scrapes titles from all linked websites, up to a maximum depth, starting from URLs in the Actor input.
It uses Playwright to open the pages in an automated Chrome browser, and to extract the title and anchor elements after the pages load.
import asyncio
from urllib.parse import urljoin
from playwright.async_api import async_playwright
from apify import Actor, Request
# Note: To run this Actor locally, ensure that Playwright browsers are installed.
# Run `playwright install --with-deps` in the Actor's virtual environment to install them.
# When running on the Apify platform, these dependencies are already included
# in the Actor's Docker image.
async def main() -> None:
# Enter the context of the Actor.
async with Actor:
# Retrieve the Actor input, and use default values if not provided.
actor_input = await Actor.get_input() or {}
start_urls = actor_input.get('start_urls', [{'url': 'https://apify.com'}])
max_depth = actor_input.get('max_depth', 1)
# Exit if no start URLs are provided.
if not start_urls:
Actor.log.info('No start URLs specified in actor input, exiting...')
await Actor.exit()
# Open the default request queue for handling URLs to be processed.
request_queue = await Actor.open_request_queue()
# Enqueue the start URLs with an initial crawl depth of 0.
for start_url in start_urls:
url = start_url.get('url')
Actor.log.info(f'Enqueuing {url} ...')
new_request = Request.from_url(url, user_data={'depth': 0})
await request_queue.add_request(new_request)
Actor.log.info('Launching Playwright...')
# Launch Playwright and open a new browser context.
async with async_playwright() as playwright:
# Configure the browser to launch in headless mode as per Actor configuration.
browser = await playwright.chromium.launch(
headless=Actor.configuration.headless,
args=['--disable-gpu'],
)
context = await browser.new_context()
# Process the URLs from the request queue.
while request := await request_queue.fetch_next_request():
url = request.url
if not isinstance(request.user_data['depth'], (str, int)):
raise TypeError('Request.depth is an unexpected type.')
depth = int(request.user_data['depth'])
Actor.log.info(f'Scraping {url} (depth={depth}) ...')
try:
# Open a new page in the browser context and navigate to the URL.
page = await context.new_page()
await page.goto(url)
# If the current depth is less than max_depth, find nested links
# and enqueue them.
if depth < max_depth:
for link in await page.locator('a').all():
link_href = await link.get_attribute('href')
link_url = urljoin(url, link_href)
if link_url.startswith(('http://', 'https://')):
Actor.log.info(f'Enqueuing {link_url} ...')
new_request = Request.from_url(
link_url,
user_data={'depth': depth + 1},
)
await request_queue.add_request(new_request)
# Extract the desired data.
data = {
'url': url,
'title': await page.title(),
}
# Store the extracted data to the default dataset.
await Actor.push_data(data)
except Exception:
Actor.log.exception(f'Cannot extract data from {url}.')
finally:
await page.close()
# Mark the request as handled to ensure it is not processed again.
await request_queue.mark_request_as_handled(request)
if __name__ == '__main__':
asyncio.run(main())
Conclusion
In this guide you learned how to create Actors that use Playwright to scrape websites. Playwright is a powerful tool that can be used to manage browser instances and scrape websites that require JavaScript execution. See the Actor templates to get started with your own scraping tasks. If you have questions or need assistance, feel free to reach out on our GitHub or join our Discord community. Happy scraping!