SPARQL-based knowledge graph question answering is most often evaluated by computing a F1-score between the result sets of a predicted and groundtruth SPARQL query. While this is better than computing an exact match accuracy over the queries themselves, this metric still struggles when used in real-world scenarios. For example, the misuse of a single entity or property within the query can lead to an empty result set and therefore an F1-score of 0. This is too harsh for a small mistake. Or as another example, questions are often ambiguous (”Name some cities in Germany”) and there may be multiple valid predictions besides the groundtruth SPARQL query. The metric should also account for that.

This project is about:

• Coming up with a metric / evaluation protocol for SPARQL-based question answering that tackles the issues mentioned above; possible approaches could be
  • extend F1-score between result sets by a structural similarity measure between a pair of SPARQL queries (e.g. how many keywords/IRIs do they have in common?)
  • train / develop an evaluator that assigns a pair of SPARQL queries a score between 0 and 1 to indicate how well they match (possibly also adding the query as context)
• Implement the metric / evaluation protocol and demonstrate that it is a more accurate measure of real-world performance than the simple F1-score over result sets