PrizePilot benchmark method

A small, inspectable method for ranking cash-prize routes.

PrizePilot is evaluated as a route-selection and evidence-management agent. This page explains how the sample ranking is produced, what judges can reproduce without account access, and where the current evidence boundary is.

Current result: Splunk stays first because it is a near-term, multi-winner feedback route with limited build friction. Qwen is the stronger submitted project route because it has public assets, a verified Qwen/DashScope smoke proof, and USD 500 Blog Post Award / Honorable Mention upside, but it still needs live Alibaba Cloud endpoint proof before that claim can be made.

Measured Signals

Signal	Why It Matters	Public Evidence
Reachable value	A smaller multi-winner route can be more rational than a single grand prize.	`samples/*.json`, `src/prizepilot/agent.py`, and static plan JSON.
Evidence readiness	Public proof should exist before a public claim is made.	Judge pack, cloud readiness, and Qwen live proof.
Human-gate clarity	Account, API key, billing, payout, tax, and KYC steps must stay visible.	Approval queue in `/api/plan` and the dashboard.
Reproducibility	Judges should be able to inspect the plan without logging in.	Static plan JSON, local CLI, and local web/API service.

Sample Portfolio

Route	Target	Why It Is Included
Splunk Agentic Ops	Most Valuable Feedback	Near-term multi-winner feedback route with low build friction.
Qwen Cloud	Blog Post Award / Honorable Mention	Submitted Qwen project route with USD 500 upside and sponsor-tool evidence requirements.
Mind the Product World Product Day	Product analytics path	Example of a product route blocked by third-party account verification.
UiPath AgentHack	Agent/product feedback route	Higher prize pool but access and product setup gates.
Onyx Algora bounty	Public PR bounty	Public code contribution route with maintainer-review gate.
Arm AI Optimization	Cloud/hardware route	High technical upside with heavier setup cost.

Scoring Method

Prize amount contributes up to a capped value so a single long-shot grand prize does not dominate.
Winner count adds material weight because multi-winner categories are more reachable.
Feedback, blog, and honorable-mention tags receive bonuses because they match fast, evidence-rich routes.
Single-winner, public-PR, product-analytics, cloud-deployment, and non-cash tags apply penalties based on execution risk.
Participant count and human blockers reduce the route score.

The relevant implementation is _prize_score() and _route_score() in src/prizepilot/agent.py.

Reproduce Locally

$env:PYTHONPATH='src'
python -m unittest discover -s tests -v
python -m prizepilot plan samples/qwen_hackathon.json
python -m prizepilot portfolio samples/splunk_agentic_ops.json samples/qwen_hackathon.json samples/mindtheproduct_world_product_day.json samples/uipath_agenthack.json samples/algora_onyx_bounty.json samples/arm_ai_optimization.json
python -m prizepilot qwen-status
python -m prizepilot cloud-readiness
python -m prizepilot.webapp --host 127.0.0.1 --port 8000

Limits: the portfolio is a decision-support sample, not a statistical prediction of winning. Live Qwen/DashScope proof has been captured once; live Alibaba Cloud endpoint proof, prize awards, payout, tax, KYC, and bank steps are not claimed here.