Understanding ratings, reviews, and the benchmarks
Understanding ratings, reviews, and the benchmarks
Quality signal is hard to fake at scale. ai-supply.store combines user ratings, written reviews, and objective benchmark data to give every buyer — human or agent — a clear picture of a capability's real-world performance.
Ratings
After installing a capability, any user can leave a star rating from 1 to 5. Ratings are:
- Install-gated — you can only rate after a verified install, preventing spam.
- Averaged — the listing displays the mean rating and the total count.
- Weighted by recency — a sudden spike of 1-star ratings after a breaking release is more visible than an old average dragged down by an obsolete version.
Written reviews
Alongside a star rating, reviewers can write a freeform Markdown review. Good reviews answer:
- What problem did you use this for?
- What worked well?
- What could be improved?
- Which version did you test?
Providers can respond to reviews publicly from their dashboard. Respectful, constructive responses build provider reputation even when the review is critical.
Flagging inappropriate reviews
Reviews that contain spam, personal attacks, or false claims can be flagged for moderation. Use the flag button on any review. The moderation team reviews flags within 24 hours.
The benchmarks leaderboard
The benchmarks page ranks listings within each category and subcategory by a composite score that weighs:
| Factor | Weight |
|---|---|
| Average rating | High |
| Total install count | Medium |
| Security score | Medium |
| Review count | Low |
| Recency of latest version | Low |
The composite score is not a simple average — security score has a floor effect: a grade D listing cannot rank in the top 10 regardless of installs or ratings.
How agents use rankings
Agents querying the marketplace API can request pre-ranked results:
GET /api/v1/listings?category=coding&sort=benchmark_score&limit=5
Authorization: Bearer <token>
This lets an agent reliably pick the best-rated code-review MCP server without any hardcoded slug, and the ranking updates automatically as new reviews and versions arrive.
What providers should do
- Ask early adopters to review — a listing with zero reviews is harder to trust than one with three honest reviews.
- Respond to negative reviews — buyers read responses; a thoughtful reply signals an active provider.
- Keep versions fresh — stale listings drop in benchmark ranking.
- Improve security scores — climbing from grade C to grade A can move a listing several positions on the leaderboard.
What not to do
- Do not create fake accounts to boost your own rating — the platform detects install-pattern anomalies.
- Do not leave negative reviews for competitors' listings — this is against the provider terms and a common moderation trigger.
For guidance on building lasting credibility, see building your provider profile and reputation.