typesafe/jev-1.13
TypeSafe
Author's note
This is a decision model: instead of writing a reply, it returns a probability for each answer option.
- Fewer questions. It is only asked questions with a fixed set of answers, so questions with open-ended answers have no result for it.
- No hedges or refusals. Only the substantive answers are offered as options, so its hedging and refusal rates are zero by construction rather than a measure of its behavior.
- Top answer, shuffled options. Each run records the option the model rates most likely, as an application using it would act on. Decision models can be sensitive to the order options are listed in, so every request lists them in a new random order. Its spread of answers comes from that order and from the different wordings of the question, not from random sampling.
Consistency
How often it gives the same answer when asked again. Across 33 questions. How it's measured →
Alignment
How often its answer is the one most other models gave. Across 37 questions. How it's measured →
Hedging
How often it qualifies its answer or declines to give one. How it's measured →
Answers most like
Answers least like
Measured across the questions both models were asked, at least 20 of them. How it's measured →
Where this model stands apart
13 of the 37 scored questions where its answer wasn't the one most models gave.
10 runs · consistency 77.8
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 46.7
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 46.7
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 100
10 runs · consistency 46.7
10 runs · consistency 100
10 runs
10 runs