I created Lemma, a Morphological Parser (Analyser) / Lemmatizer written in Elixir. It is implemented using a textbook classic method relying in an abstraction called Finite State Transducer
What is a morphological parser you may ask?
For grammatical reasons, documents are going to use different forms of a word, such as organize, organizes, and organizing. Additionally, there are families of derivationally related words with similar meanings, such as democracy, democratic, and democratization. In many situations, it seems as if it would be useful for a search for one of these words to return documents that contain another word in the set. The goal of both stemming and lemmatization is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form. For instance: am, are, is ⇒ be car, cars, car's, cars' ⇒ car The result of this mapping of text will be something like: the boy's cars are different colors ⇒ the boy car be differ color. --
What is so unique about this project is that we use Elixir's compile-time code execution to build out the lemmatization graph, and in the process we make the runtime execution so much faster.
Compile-time vs Runtime parser
To evaluate the difference between building the parser at compile time vs. building the parser at runtime, we have another benchmarking script. The script can be invoked with mix run benchmarks/runtime_vs_compiletime.exs
Sample output
