GPT‑6 Astra Computer Use in Practice: Designing a Safer Playwright Browser Agent

GPT‑6 Astra computer use is more than selecting the next coordinate from a screenshot. OpenAI recommends code execution with Playwright or PyAutoGUI for Astra because code can combine repeated actions, conditions and state checks.

Writing capable code is not the same as being safe to operate a real account. This guide turns browser automation into a small observe → plan → constrained execution → verify → approve loop. It focuses on the Astra, Responses API and Playwright architecture rather than repeating a general prompt-injection guide.

1. Choose among four integration patterns

Prefer Playwright for web work and consider PyAutoGUI for desktop software available only through the screen. When a stable API or MCP tool exists, constrain consequential changes to that explicit interface. A hybrid is often strongest: Playwright for public research and a dedicated approved function for CRM updates.

2. Separate the architecture into five layers

User objective
   ↓
Astra / Responses API ── planning and code generation
   ↓
Policy gate ── destination, action, secret and approval checks
   ↓
Isolated Playwright runtime ── constrained execution
   ↓
Observer and verifier ── screenshot + DOM + actual final state
   ↓
Audit trail, human approval and recovery

The policy gate between model and browser is essential. Do not execute code based only on the model's description. Inspect code and destinations, and expose only the necessary capabilities. Run a separate browser profile or container rather than a personal browser, with only the required account signed in.

3. Keep the execution tool narrow

Like the official pattern, the model can receive a function tool that accepts Playwright code. In production, the function must enforce the session, allowed hosts, time limit and result size.

type BrowserRun = { code: string; taskId: string };

async function executeBrowserCode(input: BrowserRun) {
  assertAllowedSyntax(input.code);
  const session = await getIsolatedSession(input.taskId);
  const result = await session.run(input.code, {
    timeoutMs: 15_000,
    allowedHosts: ['docs.example.com', 'app.example.com'],
    maxPageCount: 3,
    networkWrite: false,
  });
  return {
    stdout: result.stdout.slice(0, 20_000),
    screenshot: result.screenshot,
    url: result.url,
  };
}

This is an architecture sketch, not a complete OpenAI SDK sandbox. The important property is that runtime policy overrides generated code. String inspection alone is not a sandbox; restrict process, filesystem and network access at the environment level.

4. Do not allow one program to perform an entire consequential workflow

Combining actions is valuable, but a program that goes from login to purchase removes review boundaries. Limit each execution to a small unit that can be observed or reversed.

1. Observe the URL, important DOM elements and screenshot.
2. Let Astra produce one small objective and code block.
3. Check host, action and sensitive-data use at the policy gate.
4. Run for a bounded period in the sandbox.
5. Verify with DOM, data and screenshot—not the model's claim.
6. Pause immediately before transmission, payment or deletion.
7. Record the result and continue only when appropriate.

5. An approval screen must expose the exact change

A button that only asks “Continue?” is insufficient. Show the action, recipient or account, file and final URL, data to be transmitted, before-and-after state, expected cost, and controls to cancel, edit or approve.

Words on a page or in a document cannot grant permission. External content that requests disabling security, sending to another address or entering a token must never become a user instruction.

6. Verify completion through state, not prose

Use task IDs and idempotency keys so retries do not create duplicate orders or messages. Keep the failed session for investigation, but never automatically repeat a risky write.

7. Manage async tools and steering as a state machine

Astra can continue while a slow tool runs and accept new instructions over WebSocket. A late result may conflict with a newer instruction.

queued → running → awaiting_approval → completed
                  ↘ cancelled
                  ↘ blocked
                  ↘ failed

Record the original call ID, instruction version, start time, cancellation capability and whether the output was accepted. Steering does not automatically cancel a tool already in progress. A practical policy displays a stale result for audit but prevents it from triggering a later change.

8. Build a minimum operational scorecard

Computer-use quality cannot be selected from one model benchmark. Test the exact workflow with its login method, page structure, permissions and network.

Pre-deployment checklist

• [ ] The runtime is separate from personal browsing.
• [ ] Hosts, tools and write actions are allow-listed.
• [ ] Secrets cannot appear in model output or logs.
• [ ] Actual state is checked after short action groups.
• [ ] A person approves transmission, payment and deletion.
• [ ] Step, time and cost limits plus cancellation exist.
• [ ] Retries cannot duplicate changes.
• [ ] Stale results after steering are identifiable.
• [ ] Failures and approvals are auditable.

Conclusion: observe broadly, act narrowly

Code-driven computer use lets Astra automate a larger unit of repetitive web work and UI testing. Greater capability should not produce broader permissions. Let the agent observe broadly only when execution remains inside a small sandbox and consequential boundaries retain explicit approval.

Start with one read-only task. Verify its result through Playwright, add failures to an evaluation set, and expand one verifiable write at a time.

Official sources

• OpenAI Developers: Computer use
• OpenAI Developers: GPT‑6 Astra model guidance
• OpenAI Developers: Async tool calling
• OpenAI Developers: Mid-turn steering
• OpenAI Developers: Misalignment monitoring