Apply pass/fail gates before weighted scoring. The following weights are this desk’s proposed starting point, not industry benchmarks.
Which issues are gates?
Do not approve production publication while any of these remains unresolved:
- Permission for the intended use and audience.
- Acceptability of required promotions, data processing and audience controls.
- Security and credential handling.
- The minimum essential catalogue—or an explicitly accepted editorial exception.
- Critical launch and denied-access behaviour.
What should be weighted after the gates?
| Criterion | Suggested weight | Evidence |
|---|---|---|
| Relevant usable coverage | 30% | Reconciled catalogue and priority-title fulfilment |
| Launch reliability and mobile experience | 25% | Comparable tests and disclosed measurement limits |
| Operating effort and support | 15% | Pilot staff time, escalation results and documentation |
| Twelve-month total cost | 15% | Quotations, usage assumptions and labour records |
| Metadata and release workflow | 10% | Configuration accuracy, update handling and withdrawal records |
| Continuity and exit | 5% | Export, notice, migration and fallback provisions |
Score only evidenced capabilities. Label an untested criterion not assessed; do not quietly give it an average score. Publish a provisional comparison with missing evidence visible rather than manufacturing a precise overall winner.
For a new supplier, separate documented capability, pilot observation and contractual commitment. They answer different questions and should not collapse into one green tick.