15DATA
Can the measurements themselves be trusted?
HowFar publishes the weaknesses of its own measurement system.
Not measured yet (8)
- Tracking coverageNot measured yet
- Missing dataNot measured yet
- Prospective measurementsNot measured yet
- Retrospective reconstructionsNot measured yet
- Corrections publishedNot measured yet
- Publication delay stateNot measured yet
- Evidence completenessNot measured yet
- Unknown rateNot measured yet
STATUS TAXONOMY
- Verified
- Measured
- Derived
- Self-reported
- Hypothesis
- Unknown
- Withheld
MEASUREMENT DEPTH
9 core measurements carry the public story (2 more are planned and not measured yet); 22 supporting measurements explain and audit them; 19 are held until a qualified population exists. Nothing is removed.
Supporting measurements (22)
- Measurement coverageNo data
- Money left overNo data
- Normalized founder costCalculated
- AI spendDeclared
- Infrastructure & toolingDeclared
- Updates releasedNo data
- Bugs confirmedMeasured
- Bugs fixedMeasured
- Pull requests mergedMeasured
- Typical time to merge a pull requestMeasured
- Manual nudgesNo data
- Problems resolved without human interventionNo data
- Longest run without human interventionNo data
- Share of time working wellNo data
- Monitoring without gapsNo data
- Ready for a major failureNo data
- Backups kept elsewhereNo data
- Backups proven to restoreNo data
- Rebuilt from scratchNo data
- Reliability goalNo data
- Room left for failuresNo data
- Verified outcomes per human hourNo data
Held for later measurement (19)
- Paying App Store subscribersNo data
- Decisions left to AINo data
- Guarded changes approved by AINo data
- Actions run by fixed rulesNo data
- Actions that needed AI judgmentNo data
- Important actions with a safety checkNo data
- Actions without a known categoryNo data
- Risky actions covered by rulesNo data
- Stops that were rightNo data
- Stops that were needlessNo data
- Stops that were missedNo data
- Stops not yet reviewedNo data
- Past results can be replayedNo data
- Works if the AI is swappedNo data
- Customer messagesNo data
- Customer issues solvedNo data
- Issues solved without human interventionNo data
- Issues needing human supportNo data
- Share solved without human interventionNo data