Prompt Laboratory Development Project

Anne07-Ai/prompt-laboratory — A Git-native platform for creating, versioning, testing and evaluating prompts before production release. · Apache-2.0 · 2 stars · 0 forks · 0 watching

Create, version, test, evaluate, compare, and approve prompts before they reach production.

The Phase 7 workbench makes the prompt catalog directly usable from the experiment console:

select any Git-versioned YAML prompt

enter variables through controls generated from the prompt contract

inspect the fully rendered prompt before execution

run an offline preview or a configured OpenAI, Anthropic, or Gemini model

inspect output, latency and token usage without exposing provider credentials to the browser

Prompt files remain the source of truth. The workbench does not silently edit YAML or bypass Git review. See the Phase 7 architecture decisions.

Phase 8 sends the same rendered prompt to two or more configured providers concurrently. The comparison view keeps outputs together with latency, total tokens, and estimated cost so model choices can be reviewed using evidence rather than isolated playground results. A provider failure is contained to its own result card and does not discard successful responses from other models.

See the Phase 8 comparison design.

Phase 8.1 adds Claude Haiku 4.5 and Gemini 3.5 Flash-Lite to the live comparison matrix while keeping every provider credential server-side. See the Phase 8.1 multi-provider design.

Phase 8.2 presents those results in a responsive two-column layout with friendly model names, status badges, readable metrics, and a compact comparison summary. See the Phase 8.2 dashboard polish notes.

Phase 9A adds email/password authentication, expiring access tokens, workspaces, memberships, and workspace-isolated run history. Phase 9A.1 adds versioned database migrations and automatically persists authenticated workbench experiments. See the Phase 9A authentication design and Phase 9A.1 persistence design.

Phase 9B encrypts per-user provider credentials at rest and keeps plaintext keys inside the API execution boundary. Phase 9C adds invitation-based collaboration, owner/admin/editor/viewer roles, member management, and permission-aware dashboard controls. See the Phase 9B credential design and Phase 9C collaboration design.

Prompts often begin as strings inside application code. As teams and models multiply, those strings become production dependencies without the controls expected for production software.

Prompt Laboratory moves quality decisions into a reproducible lifecycle:

prompt contracts live in reviewable YAML

test datasets make behaviour repeatable

model adapters make comparisons provider-neutral

deterministic and model-based evaluators produce measurable evidence

regression policies block unsafe changes in pull requests

the experiment console exposes quality, latency, and cost trends

Prompt Laboratory is deliberately a quality-control plane, not an agent orchestrator or production traffic router. Approved prompt versions can be consumed by any application, agent, RAG pipeline, or orchestration platform.

Paste the two generated values after PROMPT_LAB_AUTH_SECRET= and PROMPT_LAB_CREDENTIAL_KEY= in .env. Use different values, then start the stack:

Both server secrets are required and must contain at least 32 characters. Provider API keys are optional; users can store encrypted personal keys from the dashboard, and the deterministic mock provider works without any external credential.

Surface Address PurposeExperiment console http://localhost:8501 Quality, cost, latency, and run history OpenAPI http://localhost:8000/docs Explore the versioned run API Health http://localhost:8000/health Container/service readiness

Add a three-version demonstration:

Then refresh the experiment console. See the 90-second demonstration guide.

The experiment console turns evaluation evidence into a release decision: teams can inspect quality and pass-rate trends, compare efficiency, filter by prompt and provider, and identify regressions before a prompt reaches production.

Capability EvidencePrompt contracts Semantic versions, owners, variables, output type/schema Safe rendering Strict Jinja2 variables plus Pydantic input validation Evaluation Exact match, keywords, JSON Schema, similarity, and LLM-as-judge Model support OpenAI, Anthropic, Gemini, local OpenAI-compatible HTTP, and deterministic mock Observability Input/output tokens, latency, estimated cost, score, and pass rate Comparison Prompt versions, providers/models, and approved baselines Release control Configurable GitHub Actions regression gate and downloadable evidence Product surface FastAPI, PostgreSQL/SQLite, Streamlit, and Docker Compose Team collaboration Expiring invitations, workspace roles, member management, and isolation

The platform is industry-neutral. Synthetic fixtures demonstrate the same engine across:

healthcare — patient-friendly explanations

finance — document summaries

retail — customer-support responses

HR — structured CV extraction

legal — contract-clause identification

education — level-aware concept explanations

These fixtures are demonstrations, not professional advice or validated industry systems.

The command writes one JSON report per prompt, gate-results.json, and summary.md. It returns a non-zero status when any configured threshold fails.

Baselines never update themselves. Human review remains part of approval.

Phase 1 — prompt contracts and strict rendering

Phase 2 — offline evaluation engine and structured reports

Phase 3 — providers, advanced evaluation, pricing, and comparison

Phase 4 — pull-request automation and regression gates

Phase 5 — FastAPI, PostgreSQL, Streamlit, and Docker

Phase 6 — product identity, architecture, demo, and open-source MVP documentation

Phase 7 — interactive prompt workbench and credential-safe live execution

Phase 8 — concurrent side-by-side model comparison and safe provider failures

Phase 8.1 — OpenAI, Claude, and Gemini comparison with provider-specific cost estimates

Phase 8.2 — responsive comparison cards, status badges, and summary metrics

Phase 9A — secure authentication, memberships, and workspace-isolated history

Phase 9A.1 — Alembic migrations and workspace-scoped experiment persistence

Phase 9B — encrypted per-user provider credentials and runtime key isolation

Phase 9C — team invitations, role-based access, and collaboration dashboard

Detailed decisions: Phase 3 · Phase 4 · Phase 5 · Phase 7 · Phase 8 · Phase 8.1 · Phase 8.2 · Phase 9A · Phase 9A.1 · Phase 9B · Phase 9C

Provider secrets belong in environment/secret management, never prompt or dataset YAML.

LLM-as-judge supplements deterministic checks; it does not replace them.

The committed PostgreSQL password is local-development-only.

Provider keys may be deployment-level fallbacks or encrypted per-user credentials.

Plaintext provider credentials are never returned, logged, or stored in experiment evidence.

Hosted deployment, live A/B testing, and billing remain future scope.

Read the v0.10.0 release notes and the product roadmap. Licensed under Apache-2.0.