Week 5 and 6

(5th July — 18th July)

Hey again!

So last week we implemented a new approach using Dbepdia lookup and comparing with the complete question. The comparison matrix we used was quite different from Jaccard, so my mentor asked me to try the same approach but using the Jaccard formula for comparison.

After implementation, the score fell from 0.72 to 0.56 just by changing the comparison formula to Jaccard. Here is an example dataset.

Now after this, we tried looking into results from Lookup Test 1 where we were scoring 0.72. After analysis the problems we found were -

1. when we have an incomplete or an ambiguous entity in question and there is more than one similar entity in the Dbpedia database, then while selection it creates a problem. An example will make it clear.

Eg-

question — “What are the regions in which the distributor of Secrets and Lies serves “
Benchmark-“http://dbpedia.org/resource/Secrets_and_Lies_(U.S._TV_series)"
What we choose- ”http://dbpedia.org/resource/Secrets_and_Lies”

Here we found that the entity we select is incomplete and they have a disambiguation section where our benchmarked entity is present.

so we solved this by checking if the entity we selected from the candidates, contains the benchmarked entity in their disambiguation section or not. For that, we used a query to retrieve all the links from the disambiguation section of the selected entity from candidates. Query -

SELECT ?wikiPageDisambiguates WHERE { <'''+Elink+'''> <http://dbpedia.org/ontology/wikiPageDisambiguates> ?wikiPageDisambiguates. }

Where Elink will be entities we selected from initial retrieved candidates from lookup.

2. The second problem was with the benchmarked entity. The entity mentioned in the benchmark and entity we retrieve by our approach is the same one and redirects to the same link but they are not exactly the same alphabetically.

Eg-

question — “List the affiliation of Indian general election, 2004 (Andhra Pradesh)” 
benchmark-
“http://dbpedia.org/resource/Indian_general_election,_2004_(Andhra_Pradesh)" 
Our answer- “http://dbpedia.org/resource/2004_Indian_general_election_in_Andhra_Pradesh” 

Now they both redirects to the same link.

so to solve this, we checked if the entities mentioned in the benchmark had any redirect links and if the entity we selected is present or not in those links. To retrieve the redirect links we used the query-

select ?sameAs where { { <'''+Elink+'''> <http://dbpedia.org/ontology/wikiPageRedirects> ?sameAs } UNION { ?sameAs <http://dbpedia.org/ontology/wikiPageRedirects> <'''+Elink+'''> }}

Where Elink will be entities given in the benchmark.

After solving both the problems we tested our improved approach, naming it as Lookup Test2 where the score increased from 0.72 to 0.78

After this, we again analyzed the dataset to find where are we still losing 0.22 score. After which we found some more problems which we solved -

• The 1st problem we solved was the minute spelling errors because of which while selecting entities from retrieved candidates we were choosing the wrong one. Eg.-

How many rivers are crossed by different Box Girder bridges
Benchmaked- ['http://dbpedia.org/resource/Box_girder_bridge']
We choosed- ['http://dbpedia.org/resource/Girder']

In this example when we were comparing the retrieved entity with the question, instead of counting an exact word-match as intersection, we compared words from question and candidate entities to find a similarity score using difflib similar function. Eg. if the “Bridge” in the candidate was compared to “Bridges” in question, it would not be an exact match, therefore it will not be counted as an intersection but they are the same. So with a difflib similar score, we check if any score >0.85 then we count it as an intersection. So then “bridge” and “brides” will be considered as intersections and therefore we will be able to choose our correct entity from the candidates even if there are minute spelling differences.

• The 2nd problem we solved was the entities that can’t be fetched using lookup because it doesn’t have labels for it. Therefore we checked if the entity has a label or not, if not then we don’t count it in the evaluation. The query we use for checking-

ask { <'''+Elink+'''> rdfs:label ?label. } Where Elink is the Benchmarked entity.

• The 3rd problem was because of the Benchmarked entities that are not present anymore in DBpedia Database. so for that, we checked if the entity in the benchmark is present in DBpedia or not, If not then we don’t count it in the evaluation. The query we use to check-

ask { <'''+Elink+'''> ?p ?o. } Where Elink is the Benchmarked entity.

• The 4th problem we solved was similar to the spelling error, but here the problem was the Latin character in the retrieved candidate and English character in question because of which we were not able to match them so we converted Latin character in candidates, to English alphabet so that we can compare and it worked.
• The 5th problem we solved was that some entities could not be retrieved by lookup. So we solved this by Taking candidates from both SPARQL ENDPOINT and LOOKUP.

After these improvements, we had a score of 0.86 and we named this as the Lookup test3.

Here is the Doc which has all the problems we analyzed and solved with proper problem examples.

All of these are properly documented with the dataset, code, and observations here — GitHub and all the problems and samples can be found in this Google doc.

Now for next week,

For the next week, we’re going to analyze our latest Lookup test3 approach to see where it fails and how we can improve the score further.

This was all for weeks 5 and 6.

See you next week !!

STAY TUNED!🙌.