- Business consulting
fbeta
Finding the LLM whose answers come closest to the experts.
The challenge
fbeta needed to score how well machine-generated answers agree with the answers of subject-matter experts, to pick the LLM that gives the best answers on a set of 50+ benchmark questions.
What I did
I compared evaluation techniques. BERT sentence similarity and BERT embedding similarity were the two most promising.
I used Kendall tau to rank which LLM agreed best with the experts.
Results
BERT sentence similarity and BERT embedding similarity came out as the two most promising measures, and Kendall tau then ranked the LLMs by how well they agreed with the experts.
Contact
Bring me your analytics question
Send a short email. I reply within two working days with whether I can help, a rough plan and a fixed quote.