Skip to content

text-processing

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

text-processing

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
tests
tests
 
 
text_processing
text_processing
 
 
README.md
README.md
 
 
__init__.py
__init__.py
 
 

README.md

Text Processing Library

Gustavo Watanabe - [email protected]

Assumptions

  • Word: [az-AZ]
  • Case: insensitive
  • Input Text: words + separators
  • Memory: not a blocker
# following requirements
class WordFrequency
    word: str
    frequency: int


# following requirements
class WordFrequecyAnalyzer
    def calculate_highest_frequency(text: str) -> int:
        ...

    def calculate_frequency_for_word(text: str, word: str) -> int:
        ...

    def calculate_most_frequent_n_words(text: str, n: int) -> list[WordFrequency]:
        ...

Implementation

  • Python 3.9.5

  • No external libraries used. The native:

    • re (regular expressions)
      • used to extract words only from text
    • collections.Counter:
      • optimizing word frequency map generation
    • collections.namedtuple:
      • a decorator was created to ensure the correct argument types are being passed to the methods and return a default value in case of nullable arguments.
      • a namedtuple was created to facilitate the creation of a parameter set (arg types and nullable return) for that decorator.
    • functools.wraps:
      • bind original decorated function name to the decorator
    • functools.cache:
      • Due to the nature of the WordFrequecyAnalyzer being static (once it's receiving text as an argument on its method level) I've decided to use a cache strategy keeping in memory the lastest computed counter_map for that text, avoiding to reprocess this hasmap in every subsequent WordFrequecyAnalyzer method call.
    • unittest
      • self explanatory ;)

  • Extra assumption on calculate_most_frequent_n_words

    • if N is greater than len(text) we will return all WordFrequencies bound to the text firstly per frequency and then alphabetically sorted instead of raising an error such as "out of bounds".

  • Running tests

    • If using and IDE/Text editor, the tests should be automatically loaded

    • If using a Terminal to run the tests, you may need to firstly export the PYTHONPATH:

      # bash
cd <root folder> # where this README file is located
export PYTHONPATH=.

# or Windows
set PYTHONPATH=.

python tests/test_word_frequency_analyzer.py
# or
python -m unittest tests/test_word_frequency_analyzer.py