Giving an AI Agent a Browser Without Giving It Your Logins

A lot of useful information still lives behind a login, on a site with no API: a research portal someone pays for, a broker’s reports page, an announcement that only renders in a real browser. Letting an agent read those pages is an obvious feature.

Driving the browser is the easy part. Playwright works. The hard part is that a logged-in browser is the user’s identity. Whatever the agent can do in that tab, it does as them. So I think of a browser for an agent as identity infrastructure first, with some automation attached.

One container per session, one way out

The isolation unit I’d choose is a container per browser session: Chromium, Playwright, a VNC server so a person can watch, and a small action server. Unprivileged, read-only root filesystem, tmpfs scratch space, a process limit, explicit memory and CPU limits. Pin the image by digest so “the browser” means one specific build, not whatever was pulled last.

Mount exactly one profile volume per identity — or a throwaway tmpfs home for anonymous browsing — plus a small quarantine directory for downloads. Nothing else: no application data, no Redis, no Docker socket.

The network is where most of the security thinking goes. Put session containers on a Docker network marked internal: true, so they physically can’t reach the internet except through an egress proxy. Then split the policy across three places:

1. The service that launches sessions checks each identity’s domain allowlist and resolves the hostname first, with a short cache and a timeout. Never cache a failed lookup.
2. The runtime inside the session checks every redirect hop, because an allowed domain can redirect anywhere.
3. The proxy refuses anything that resolves to loopback, link-local (which covers cloud metadata endpoints), private ranges, CGNAT, or IPv4-mapped IPv6, and any port other than 80 and 443.

With Squid that last layer can look like this:

acl blocked_dst dst 127.0.0.0/8 169.254.0.0/16 10.0.0.0/8 \
                    172.16.0.0/12 192.168.0.0/16 100.64.0.0/10
acl web_ports port 80 443
http_access deny blocked_dst
http_access deny !web_ports
# Per-identity allowlists are enforced upstream.
# The proxy has no idea whose request this is, so after the denies it allows.
http_access allow all

That last line looks alarming out of context. It’s deliberate. The proxy doesn’t know which user or identity a request belongs to, so it can only enforce what’s true for every session. Anything identity-specific belongs in the component that does know. I’d also never intercept TLS here — no man-in-the-middle certificate anywhere — and I’d strip query strings from proxy logs, because URLs carry tokens more often than you’d like.

Control is a number

At any moment an identity is in one state: needs login, ready, under agent control, under human control, action required, locked, revoked. The state names aren’t what make it safe. What makes it safe is that every transition is a compare-and-swap on a version number:

result = db.execute(
    update(Identity)
    .where(Identity.id == identity_id, Identity.version == expected)
    .values(version=expected + 1, state=new_state)
)
if result.rowcount != 1:
    raise Conflict("browser control has changed")

When someone clicks “Take over”, one transaction cancels queued agent actions, bumps the version, revokes the agent’s grants, and enables keyboard and mouse in the viewer. Handing control back bumps it again. Any agent action still in flight carries the old version and fails at the database, whatever any screen happens to show. Two tabs, a slow network, a stale UI — none of it matters, because the number decides.

Passwords and one-time codes should always be typed by a person. A login session can start in human control. The agent never sees credentials, and I wouldn’t build a “remember my password for the agent” feature at all. If a site logs you out, a human logs you back in.

Viewer access should be short-lived and single-use, bound to the user, identity, session, permission, and the current control version. Proxy the VNC WebSocket through the API rather than exposing it, and give each session its own VNC password.

Saved logins are a durability problem

Once users can save a logged-in profile, losing it becomes the worst bug you can ship — worse than a crash, because nothing looks broken.

Some rules I’d hold to: a persistent profile has exactly one writer. Login checkpoints are atomic and include IndexedDB, because plenty of sites keep their session there rather than in cookies. Disposable copies mount the source read-only and can’t write a checkpoint back.

The launcher should keep an encrypted journal of what it started — session, container, image, version, per-session credentials — written with fsync and atomic rename, in owner-only files, refusing symlinks, with no plaintext fallback. Write a tombstone before destroying a container, or a crash at the wrong moment leaves a container nobody knows about.

After the launcher restarts, the journal alone shouldn’t bring a session back. Check that the container ID, labels, image, network, and mounts match exactly, then challenge the runtime inside it: are you the same process, at the same version, is Chromium ready, are you busy? Quarantine anything that fails, and keep counting it against capacity until it’s gone.

Failure modes worth testing

A recovery path with the wrong default. Imagine a fallback that re-acquires a session with persistent=False as its default argument. One day a session has to go through that path, and the user’s saved account comes back as a fresh, logged-out profile. Nothing crashes. People just find themselves logged out. Derive the storage policy from current, authorised state every time, never from a default.

The browser started but the client didn’t hear about it. The user clicks “Open browser” and gets a gateway timeout or a 499. The container actually started. Retrying would start a second one on the same profile. Instead, observe read-only for a few seconds and accept only the same identity at exactly the next version, in an active state. Anything else is shown as uncertain.

Access denied that looks like empty data. A real browser visit to some official site returns 403. A plain HTTP fetch of the same URL returns 200 with an empty app shell. If both are treated as “successful, nothing found”, the agent will confidently report that there’s nothing to report. A denial should be a typed “unavailable” result the agent has to acknowledge.

Sessions that outlive their browsers. Restarts, idle cleanup, and expired leases leave database rows pointing at containers that no longer exist. If the agent’s response to “stale session” is to retry, you get loops. The rule I’d use:

if launcher.confirms_lost(session):   # unreachable is not the same as lost
    restart_same_identity(session)    # reopen the last URL only
    if action in READ_ONLY:           # navigate, read, query, scroll...
        return rerun(action)
    return Error("browser_restarted")  # never replay a click

Confirm the loss first, because a launcher that’s briefly unreachable hasn’t lost anything. Restart the same identity and reopen the URL. Re-run reads. Anything that might have changed the page returns an error the agent has to handle. And never replace a session a person is controlling, or one with an action in flight.

Start read-only

The last decision is the most important: what the agent may do. I’d classify actions in tiers — observe, read, draft input, consequential, prohibited — and start with reads and narrowly approved drafts. Submitting a form or placing an order through a website should wait for an approval mechanism you’re willing to stand behind. There’s more on that in the human-in-the-loop post.

Cheaper designs exist — several browser contexts inside one Chromium, for instance — and they’re tempting at scale. I’d only move to them once the isolation story is as clear as a container per session. Read-only covers more of what people actually want than you’d expect.