Could you give us some more info about the evaluation metric?
1 Like
The evaluation metric is Levenshtein Distance. There are lots of great articles about how this equation works. Here’s a link to a nice introduction:
Thanks for the reference. My question though is not on the understanding of the distance.
In the competition description I read:
“Submissions will be scored against the test answer key using a Levenshtein distance. A higher score is considered more successful.”
The problem is that a perfect match would produce a distance of 0 whereas if I submit the template submission I get a score of zero. Moreover, you say that a higher score is better in contrast to the distance definition.
My guess is, that you somehow modified the distance and I would like to know what is exactly the equation that you are using so that I can evaluate the performance of my solutions locally.
The final score utilizes the Levenshtein ratio and is applied to a 10-point scale.
score = (1 - leven/length) * 10
Which returns a score between 0 and 10. This scoring algorithm also prevents negative scores.
2 Likes
Great, thanks for the clarification!