Production Personal AI Agent Platform

Overview

I designed and built the multi-agent, evaluation, and MCP integration layers of a personal AI platform used in production by developers across a large organization.

Each person works with a persistent agent rather than switching between disconnected assistants. The agent has a private workspace, retains relevant context, can create or propose reusable skills, and performs work inside its own sandbox with the user's permissions.

Autonomous work with explicit boundaries

The agent can use the capabilities available in its sandbox without asking for a separate approval on every step. Its reach is constrained by the user's permissions and the tools exposed to that sandbox.

This makes it possible to delegate work that takes more than a single chat turn while keeping the owner, the workspace, and the available capabilities separate from other users.

Multi-agent runtime and MCP integration

I built the multi-agent layer on top of Pi harness and connected the platform to external capabilities through MCPs. The gateway is implemented in Go, while the MCP integration layer is implemented in Python. MySQL stores user data used by the platform.

The architecture keeps runtime orchestration, tool access, and user context behind a consistent agent experience. I did not own the pod provisioning or underlying infrastructure.

Evals that test behavior, not just answers

I built the offline and online evaluation system for the agent. The challenge was to test every meaningful way the platform could be used: whether the agent calls the right tools, records information when it should, avoids recording it when it should not, and behaves reliably across real workflows.

The evaluation stack checks trajectories, memory behavior, and system effects in addition to the final response. It currently uses GLM 3.5 to keep evaluation runs cost-efficient.

Production status

The full platform is in production and used by developers across the organization. No user-count or business-impact figures are disclosed here, but the system operates as an internal daily-work platform rather than a prototype.

My role

I owned the multi-agent layer, the full offline and online evaluation system, and the MCP connection layer. My work focused on agent orchestration, quality evaluation, and safe access to capabilities within each user's sandbox and permission boundaries.