Browser Pool Orchestration With Airflow: Checkout, Run, Release

Spinning up a fresh profile for every task is wasteful. Each one needs a proxy assigned, settings checked and logins repeated, and dozens of brand-new browsers hitting the same endpoints in parallel put an unnecessary burst on the target site.

The fix is architecture. You don't create profiles on demand; that's the usual mistake. You maintain a pool of long-lived, already configured profiles and check them out like library books. One profile per task. Task finishes, profile goes back on the shelf. Next task grabs a different one. This is a common way for data engineering and QA teams to run browser automation at scale.

By the end of this tutorial, you'll have an Airflow DAG that checks out profiles from a REST-managed pool, runs Playwright tasks against them, releases them back, and handles failures gracefully. The same pattern works for Prefect, Temporal, or any workflow orchestrator that can hit HTTP endpoints.

Prerequisites

Before we start:

• Apache Airflow 2.6+ — installed and running, with the HTTP provider (pip install apache-airflow-providers-http)
• JustBrowser ($9.99/month or $99.99/year) — the REST API is included in the one plan and in the 7-day trial, so you can build the whole pool before you're charged
• Redis — for distributed profile locking (prevents race conditions)
• A Windows x64 or macOS (Apple Silicon) machine running JustBrowser — its API listens only on 127.0.0.1, so the Airflow worker that runs these tasks must run on that same machine. There is no Linux build, and nothing to put in a container
• Python 3.10+ — for the task code
• Basic Airflow knowledge — you should know what a DAG, operator, and XCom are

You don't need Kubernetes, and it wouldn't help here: the browsers run on a desktop machine, not in a cluster. A single Airflow scheduler with a worker on the JustBrowser machine handles most workloads. For the wider question of collecting data at volume without straining the sites you visit, see the guide to collecting public web data at scale.

Step 1: Create Your Profile Pool

First, set up the profiles you'll be checking out. These aren't disposable — each one is a long-lived browser with its own cookies, storage and proxy.

In JustBrowser, create 10-20 profiles with varied configurations:

• Identities: Vary the OS identities — Windows 11, Windows 10, macOS — with different screen, GPU and font settings, so the pool reflects the mix of devices you need to test. The browser version tracks the current Chromium release on every profile. JustBrowser's native engine handles the 40+ identity parameters at the C++ level.
• Proxies: Give each profile its own proxy where the work needs a fixed location, so a task always runs from the place it is meant to. Pinning a proxy to a profile is covered in its own tutorial.
• Setup: Open each profile once, sign in where your workflow needs it and confirm the proxy responds before adding it to the pool. Profiles keep their cookies and storage between runs.

Export the profile IDs. You'll need them for the pool registry:

# profiles.py
PROFILE_POOL = [
    "profile_a1b2c3d4",
    "profile_e5f6g7h8",
    "profile_i9j0k1l2",
    # ... your 10-20 profiles
]

Why 10-20? Depends on your task volume and run duration. If each task takes 2 minutes and you're running 100 tasks/hour, you need at least 4 profiles (100 tasks × 2 min ÷ 60 min). Double or triple that for headroom and cooldown time.

Step 2: Build the Browser Pool Orchestration Manager

This is the part everyone skips and then regrets. You need a central registry that tracks which profiles are available, which are checked out, and which are cooling down after failures.

# pool_manager.py
import redis
import time
import json
from typing import Optional

class ProfilePoolManager:
    def __init__(self, redis_url: str, profile_ids: list, lock_ttl: int = 1800):
        self.redis = redis.from_url(redis_url)
        self.profile_ids = profile_ids
        self.lock_ttl = lock_ttl  # 30 min default — longer than any task

    def checkout(self, task_id: str) -> Optional[str]:
        """Acquire exclusive lock on an available profile."""
        for profile_id in self.profile_ids:
            lock_key = f"profile_lock:{profile_id}"
            cooldown_key = f"profile_cooldown:{profile_id}"

            # Skip if cooling down
            if self.redis.exists(cooldown_key):
                continue

            # Try to acquire lock (atomic)
            acquired = self.redis.set(
                lock_key,
                json.dumps({"task_id": task_id, "acquired_at": time.time()}),
                nx=True,  # Only set if doesn't exist
                ex=self.lock_ttl
            )

            if acquired:
                return profile_id

        return None  # No profiles available

    def release(self, profile_id: str, failed: bool = False, cooldown_seconds: int = 900):
        """Release profile back to pool. Optional cooldown after failure."""
        lock_key = f"profile_lock:{profile_id}"
        self.redis.delete(lock_key)

        if failed:
            cooldown_key = f"profile_cooldown:{profile_id}"
            self.redis.setex(cooldown_key, cooldown_seconds, "cooling")

    def get_available_count(self) -> int:
        """How many profiles are free right now?"""
        available = 0
        for profile_id in self.profile_ids:
            lock_key = f"profile_lock:{profile_id}"
            cooldown_key = f"profile_cooldown:{profile_id}"
            if not self.redis.exists(lock_key) and not self.redis.exists(cooldown_key):
                available += 1
        return available

The nx=True flag on Redis SET is doing the heavy lifting here. It's atomic — if two workers try to grab the same profile in the same millisecond, only one succeeds. The other loops to the next available profile.

That 30-minute TTL (lock_ttl) is your safety net. Worker dies mid-task without releasing? Lock auto-expires. Tune this to be longer than your longest task but short enough that crashed workers don't block the pool for hours.

Step 3: Create the JustBrowser API Client

JustBrowser's local REST API handles profile launch/stop. Wrap it in a client class so your DAG code stays clean:

# justbrowser_client.py
import requests
from typing import Optional

class JustBrowserClient:
    def __init__(self, api_base: str = "http://127.0.0.1:36542/api/v1", api_key: str = None):
        # token from the app's API/AI tab; Authorization: Bearer is required
        self.api_base = api_base
        self.headers = {"Authorization": f"Bearer {api_key}"}

    def launch_profile(self, profile_id: str, headless: bool = False) -> str:
        """Start the profile and return its CDP URL."""
        resp = requests.post(
            f"{self.api_base}/profiles/{profile_id}/start",   # with api_base = http://127.0.0.1:36542/api/v1
            json={"headless": headless},
            headers=self.headers
        )
        resp.raise_for_status()
        return resp.json()["data"]["cdp_url"]

    def stop_profile(self, profile_id: str):
        """Stop a running profile."""
        requests.post(
            f"{self.api_base}/profiles/{profile_id}/stop",
            headers=self.headers
        )

The cdp_url this returns is a standard CDP (Chrome DevTools Protocol) URL. Playwright's connectOverCDP() and Puppeteer's connect({ browserWSEndpoint }) both speak it. We covered the integration details in the Playwright/Puppeteer guide.

("What if the API is down?" Add retry logic with exponential backoff — three retries, 2/4/8 second delays. It's left out here to keep the examples readable, but in production you want it.)

Step 4: Write the Airflow DAG

Here's a complete DAG that collects public product-page data from a storefront you are allowed to access, using pooled profiles:

# dags/browser_pool_scrape.py
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.exceptions import AirflowSkipException
import asyncio
import os

from profiles import PROFILE_POOL
from pool_manager import ProfilePoolManager
from justbrowser_client import JustBrowserClient

# Config
REDIS_URL = "redis://localhost:6379/0"
JUSTBROWSER_API = "http://127.0.0.1:36542/api/v1"
JUSTBROWSER_TOKEN = os.environ["JB_API_TOKEN"]  # from the app's AI page
TARGET_URLS = [
    "https://example-store.com/products/1",
    "https://example-store.com/products/2",
    # ... your target URLs
]

default_args = {
    "owner": "scraping-team",
    "depends_on_past": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=5),
}

def scrape_product(url: str, **context):
    """Checkout profile, scrape URL, release profile."""
    pool = ProfilePoolManager(REDIS_URL, PROFILE_POOL)
    client = JustBrowserClient(JUSTBROWSER_API, api_key=JUSTBROWSER_TOKEN)

    task_id = context["task_instance"].task_id
    profile_id = None

    try:
        # Checkout
        profile_id = pool.checkout(task_id)
        if not profile_id:
            raise AirflowSkipException("No profiles available — will retry")

        # Launch
        cdp_url = client.launch_profile(profile_id, headless=False)

        # Scrape (using Playwright async)
        result = asyncio.run(_run_scrape(cdp_url, url))

        # Stop browser
        client.stop_profile(profile_id)

        # Release — success
        pool.release(profile_id, failed=False)

        return result

    except Exception as e:
        if profile_id:
            client.stop_profile(profile_id)  # Clean up
            pool.release(profile_id, failed=True, cooldown_seconds=900)  # 15 min cooldown
        raise

async def _run_scrape(cdp_url: str, url: str) -> dict:
    """Actual Playwright scraping logic."""
    from playwright.async_api import async_playwright

    async with async_playwright() as p:
        browser = await p.chromium.connect_over_cdp(cdp_url)
        context = browser.contexts[0]
        page = await context.new_page()

        await page.goto(url, wait_until="networkidle")

        # Your scraping logic here
        title = await page.title()
        # price = await page.locator(".price").text_content()
        # ...

        await page.close()

    return {"url": url, "title": title}

with DAG(
    "browser_pool_scrape",
    default_args=default_args,
    description="Collect product data using pooled multi-account browser profiles",
    schedule_interval=timedelta(hours=6),
    start_date=datetime(2026, 7, 1),
    catchup=False,
    max_active_tasks=10,  # Limit concurrency to pool size
) as dag:

    tasks = []
    for i, url in enumerate(TARGET_URLS):
        task = PythonOperator(
            task_id=f"scrape_product_{i}",
            python_callable=scrape_product,
            op_kwargs={"url": url},
        )
        tasks.append(task)

Key details worth getting right:

max_active_tasks=10 — This limits concurrent task execution to (roughly) your pool size. If you have 15 profiles but allow 50 concurrent tasks, 35 will skip immediately. Not the end of the world with retries, but wasteful. And annoying to debug when you're staring at logs wondering why half your tasks are skipping.

AirflowSkipException — When no profiles are available, skip rather than fail. Task retries based on your retries config. Gentler than crashing.

Profile stop before release — Always stop the browser before releasing the profile. Otherwise the next task that checks it out might get a stale CDP endpoint.

15-minute cooldown on failure — If a task fails (timeout, rate limit, crash), cooling the profile down prevents immediate re-use and respects the site's limits. Tune this based on your targets — strict sites need longer cooldowns, and if a site tells you to stop, stop.

Step 5: Add Failure Handling Hooks

The DAG above handles failures in the task function, but what if the task crashes hard (OOM, worker death) before reaching the except block? Add a callback:

def on_task_failure(context):
    """Emergency profile release on hard failures."""
    pool = ProfilePoolManager(REDIS_URL, PROFILE_POOL)
    task_id = context["task_instance"].task_id

    # Find which profile this task had checked out
    for profile_id in PROFILE_POOL:
        lock_key = f"profile_lock:{profile_id}"
        lock_data = pool.redis.get(lock_key)
        if lock_data:
            import json
            data = json.loads(lock_data)
            if data.get("task_id") == task_id:
                pool.release(profile_id, failed=True, cooldown_seconds=1800)
                break

default_args = {
    # ... existing args
    "on_failure_callback": on_task_failure,
}

This scans the Redis locks to find which profile the dead task held, then releases it with a longer cooldown (30 min instead of 15). Hard crashes are usually worse than graceful failures — something went genuinely wrong, not just a slow page.

Common Errors and How to Fix Them

Redis connection refused

redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379

Cause: Redis isn't running, or it's on a different host/port.

Fix: Start Redis (redis-server) or update REDIS_URL to point to your actual Redis instance.

Profile launch returns 404

requests.exceptions.HTTPError: 404 Client Error: Not Found

Cause: Profile ID doesn't exist in JustBrowser, or the API path is wrong.

Fix: Verify profile IDs in JustBrowser's GUI. Check the API base URL — default is http://127.0.0.1:36542/api/v1; the port is configurable in the app's API tab, so confirm it matches.

All profiles stuck in checkout

Cause: Tasks crashed without releasing locks, and TTL hasn't expired yet.

Fix: Wait for TTL expiry, or manually clear locks: redis-cli KEYS "profile_lock:*" | xargs redis-cli DEL. Also add the on_failure_callback hook if you haven't.

Playwright times out connecting to CDP

playwright._impl._errors.Error: Timeout 30000ms exceeded

Cause: Profile launched but browser startup is slow, or the cdp_url is stale.

Fix: Increase Playwright's timeout (timeout=60000). Make sure you're launching the profile fresh — don't reuse a cdp_url from a previous session.

Next Steps

Once this browser pool orchestration DAG is running, you'll probably want to add:

• Monitoring: Track checkout wait times, failure rates by profile, cooldown queue depth. If one profile consistently fails, remove it from the pool. JustAnalytics can help you build dashboards for this without the GDPR compliance headaches.
• Dynamic scaling: Use Airflow's pool feature or a custom sensor to scale max_active_tasks based on available profiles. Overkill for most setups, but nice to have.
• Profile rotation: Periodically retire old profiles and add fresh ones. Long-lived profiles accumulate stale cookies and storage after hundreds of sessions.

For teams running ad verification pipelines, pair this setup with ClickzProtect on the traffic side, which scores incoming clicks for bot signals.

If you automate across several tools, Velocity Digital Labs, the studio behind JustBrowser, also publishes an AI coding agent productivity benchmark.

The profile-pool pattern scales a long way. At very large pool sizes you start worrying about Redis latency, but for most automation workloads this architecture is enough.

Frequently Asked Questions

Why use a profile pool instead of creating profiles on demand?

Pooled profiles keep their cookies, logins and settings between runs, so each task starts from a known state instead of repeating setup and sign-in. A pool also prevents race conditions — you never want two workers grabbing the same profile simultaneously. The checkout/release pattern guarantees isolation.

How do I handle profile failures in an Airflow DAG?

Use on_failure_callback to release the profile back to the pool even if the task crashes. Mark profiles as "cooling down" after failures so they don't immediately get reassigned. A 15-30 minute cooldown respects the target site's rate limits and gives you a chance to look at why the task failed before the profile runs again.

Can I use Prefect instead of Airflow for browser pool orchestration?

Yes. The pattern is the same — acquire profile at task start, release at end, use a distributed lock (Redis) to prevent double checkout. Prefect's task retries and state handlers map directly to the checkout/release lifecycle. The examples here work with minimal changes in Prefect.

How many browser profiles can one machine handle?

It depends on the pages you load. Every running profile is a full browser process, so measure memory per profile on your own workload and leave headroom for the OS and Airflow. Headless is lighter, but it follows headless defaults for screen and viewport. For larger pools, add machines, each running JustBrowser with its own Airflow worker, and share the Redis lock between them.

---

Try JustBrowser

Native Chromium multi-account browser for teams — not extension-based. Real C++ engine patches at the canvas / WebGL / audio / font / screen layer, with 40+ identity parameters rendered by the engine itself. Local REST API + CDP for Playwright, Puppeteer, Selenium. $9.99/month or $99.99/year. 7-day free trial, card required — cancel any time in the seven days and you are not charged. Unlimited profiles.

Get started → · How it differs from Multilogin / GoLogin / AdsPower