Running Playwright in production: concurrency, sessions and not falling over

Browsers vs contexts

Launching a browser is expensive; opening a browser context inside it is cheap. A context is an isolated session — its own cookies and storage — so you can run many independent jobs in one browser process. Default to a small pool of browsers, each hosting several contexts.

Cap concurrency deliberately

Every open page costs memory and CPU. Past a certain point, adding more parallel jobs makes every job slower until pages start timing out. Measure how long a typical job takes and how much memory it uses, then set a hard limit — for example with a semaphore around "get a page".

import asyncio
slots = asyncio.Semaphore(8)          # tune to your machine

async def run_job(browser, url):
    async with slots:
        ctx = await browser.new_context()
        try:
            page = await ctx.new_page()
            await page.goto(url, timeout=45_000)
            return await page.title()
        finally:
            await ctx.close()          # always release

Fail fast when you're full

If requests arrive faster than you can serve them, an unbounded queue turns into minutes-long waits and gateway timeouts. Wait for a slot for a short, fixed time; if none frees up, return a clear "busy, retry shortly" error. Callers can back off and retry, and nothing hangs.

Keep sessions logged in

• Persistent context: launch_persistent_context(user_data_dir) keeps a full browser profile on disk.
• Storage state: save cookies and local storage to a file after logging in, and load it into new contexts.

Either way, the agent stops hitting login walls on every run.

Timeouts, retries and waiting

• Set explicit navigation timeouts, and treat a timeout as a clean, retryable error rather than a crash.
• Wait for the thing you need (an element, a network response) instead of sleeping for a fixed time.
• Retry once or twice with a fresh page, not endlessly.

Health checks

Browsers can hang without crashing. Periodically check each one still responds, alert when one doesn't, and restart it. A pool that silently runs one browser short is slower for everyone.

Sizing the machine

RAM is usually the first limit, then CPU. A 64 GB server gives you room for many contexts plus the agent and your own tools; smaller machines work if you keep concurrency modest. Watch memory under real load before you raise limits.

Common questions