• Business consulting

fbeta

Finding the LLM whose answers come closest to the experts.

The challenge

fbeta needed to score how well machine-generated answers agree with the answers of subject-matter experts, to pick the LLM that gives the best answers on a set of 50+ benchmark questions.

What I did

I compared evaluation techniques. BERT sentence similarity and BERT embedding similarity were the two most promising.

I used Kendall tau to rank which LLM agreed best with the experts.

Results

BERT sentence similarity and BERT embedding similarity came out as the two most promising measures, and Kendall tau then ranked the LLMs by how well they agreed with the experts.

Contact

Bring me your analytics question


Send a short email. I reply within two working days with whether I can help, a rough plan and a fixed quote.