An agent tests the software behind an answer engine
OpenAI published a customer story on September 14 describing how Perplexity uses GPT‑6 Astra. Perplexity cofounder Johnny Ho says the model moves beyond writing search and summarization programs into changing and observing real systems. When there is not enough time to hand-build every integration test, the model creates a small test program that imitates responses from a language-model API or connector and checks how the application behaves across the complete workflow.
The notable shift is the scope of verification, not merely the amount of generated code. An agent that produces both implementation support and service doubles can explore failures across connected systems quickly. Yet using the same model to implement and judge a feature can reproduce the same mistaken assumption twice. Contract tests, fixed regression suites and results from a restricted sandbox therefore remain independent evidence.
A customer story is not an independent benchmark
The page is an OpenAI-authored customer account. It does not report a task count, failure rate, cost or controlled comparison with earlier models. Perplexity's statement that people can check in less often is operational experience, not a general performance guarantee. Editing and monitoring production software also require separate permissions for observation, proposed changes and approved deployment.
Teams should review the schemas and failure modes in model-generated mocks and avoid treating a passing synthetic test as the only success criterion. Production changes still need least privilege, approval, audit logs, rollback and monitoring that the agent cannot modify.