Investigate non-exact / fuzzy matches #11

Activity

1. johann-petrak commented on Jun 18, 2018

   

   johann-petrak

   on Jun 18, 2018

   CollaboratorAuthor

   More actions

   From johann-petrak/gateplugin-StringAnnotation#13
   There are various possible approaches, several of them based on tries.
   See also http://dbgroup.cs.tsinghua.edu.cn/dd/projects/taste/index.html
2. thomas-heitz commented on Aug 26, 2018

   

   thomas-heitz

   on Aug 26, 2018

   Last edited by thomas-heitz

   More actions

   Hello Johann,

   I'm using your Extended Gazetteer with several millions of entries and it works very well, thanks for that!

   To contribute to the ideas about non-exact matches, I can share my method:

   • creation of different versions of gazetteer directories from a source gazetteer directory with a Java script
   • for example one is with normalized punctuation, then accent, stemmed, metaphone, etc.
   • point to the needed gazetteer in the .def file, for entities it will be punctuation and/or accent, for concepts it will be stemmed
   • add a Jape grammar with the same code as the previous Java script before the Extended Gazetteer to create as many Token features as type of gazetteer, for example: normalized, punctuation, stemmed, etc.
   • set the feature in the parameters of the Extended Gazetteer

   This method allows to choose the best normalization for each gazetteer and it's easy to change it as all the type of gazetteers are always created by the Java script and also the Token features by the Jape grammar.

   👍React with 👍1Reacted by Johann Petrak
3. johann-petrak commented on Aug 26, 2018

   

   johann-petrak

   on Aug 26, 2018

   CollaboratorAuthor

   More actions

   Thanks Thomas! Yes, this is a good method, when it is possible to generate most or all of the alternatives one wants to match automatically.

   However, sometimes users want to match in a way that cannot be predicted, e.g. based on Levenshtein distance, phonetic similarity or some such. If there is a well defined distance metric between the strings, it is possible to implement this as an extension to the trie matching algorithm but it is not easy to implement.
4. thomas-heitz commented on Aug 30, 2018

   

   thomas-heitz

   on Aug 30, 2018

   More actions

   I also plan to use the double metaphone then levenshtein for a phonetic match.

   • convert the gazetteer into double metaphone values
   • keep the original value of the gazetteer entry as a feature "string"
   • add a feature Token.metaphone for Tokens in the text that are not in my dictionary
   • apply the metaphone gazetteer on these Token.metaphone in the text
   • you get several Lookup.string for each Token
   • use levenshtein to choose the best candidate for each Token by comparing the Lookup.string and Token.string

   👍React with 👍1Reacted by Johann Petrak