Jessica Zhan

Problem

Prompt iteration is often manual and subjective.

Teams generating repeated AI outputs (marketing, content, internal workflows) frequently adjust prompts without knowing whether quality is genuinely improving.

This project explored turning prompt improvement into a measurable, systematic process.

Approach

Built an AI evaluation platform that:

Accepts a prompt and target use case
Generates a custom evaluation rubric
Scores outputs against defined criteria
Produces multiple prompt variants and compares performance
Iteratively improves toward stronger outputs

Product Decisions & Tradeoffs

1. Reduce setup effort with templates and guided questions

Writing a good rubric from scratch creates friction. The product provides templates for common workflows while allowing users to customise evaluation depth for higher-stakes workflows.
2. Don't reinvent the wheel

Prompt optimisation had strong open-source approaches available. I focused on integrating, testing, and measuring whether those approaches produced meaningful improvements in my use cases.
3. Prove the system before polishing the experience

The core risk was not usability but whether systematic prompt optimisation produced better outcomes. I chose a lightweight interface first and focused on building confidence in the evaluation loop.

What I Learned

AI systems need gold standards — writing ideal outputs yourself first is what made the evaluation calibrate. Without a human-defined north star, the AI optimised toward its own scoring.
Human judgement is still necessary — letting users see the reasoning, compare before/after outputs, and choose the right prompt themselves matters more than automation alone.

Outcome

Created a repeatable evaluation process across several workflows — replacing ad-hoc prompt tweaking with a systematic method I can reuse and build on, leading to a 5–10 point increase on evaluated prompts.

What I'd Explore Next