o Mf@s`ddlZddlmZddlmZddlmZddlmZGdddej Z Gdd d ej Z dS) N)closing)data) PorterStemmer)SnowballStemmerc@s4eZdZddZddZddZddZd d Zd S) SnowballTestcCsRtdd}|ddksJ|ddksJ|ddks J|dd ks)J|d d ks2J|d d ks;J|d d ksDJ|d d ksMJ|ddksVJtdd}|d dksdJ|dd ksmJ|ddksvJtd}|ddksJ|ddksJ|ddksJ|dd ksJ|ddksJdS)z this unit testing for test the snowball arabic light stemmer this stemmer deals with prefixes and suffixes arabicTu&الْعَرَبِــــــيَّةuعربuالعربيةu فقالواuقالuالطالباتuطالبuفالطالباتuوالطالباتuالطالبونu اللذانuمنFuاللذuالكلماتuكلمNrstem)self ar_stemmerr Z/var/www/html/analyze/labelStudio/lib/python3.10/site-packages/nltk/test/unit/test_stem.py test_arabic s(  zSnowballTest.test_arabiccCtd}|ddks JdS)Nrussianuавантненькаяuавантненькr)r stemmer_russianr r r test_russian'zSnowballTest.test_russiancCs`td}tddd}|ddksJ|ddksJ|ddks%J|ddks.JdS)NgermanT)ignore_stopwordsu Schränkeschrankkeinenkeinr)r stemmer_germanstemmer_german2r r r test_german+s  zSnowballTest.test_germancCs0td}|ddks J|ddksJdS)Nspanish Visionadovisionalguealgurr stemmerr r r test_spanish5szSnowballTest.test_spanishcCr)Nenglishzy'syrr!r r r test_short_strings_bug=rz#SnowballTest.test_short_strings_bugN)__name__ __module__ __qualname__rrrr#r&r r r r r s  rc@sDeZdZddZddZddZddZd d Zd d Zd dZ dS) PorterTestcCsFttdjdd}|WdS1swYdS)Nz*stemmers/porter_test/porter_vocabulary.txtutf-8encoding)rrfindopenread splitlinesr fpr r r _vocabularyCs  $zPorterTest._vocabularycCsLt|d}t||D]\}}||}||ks#Jd||||q dS)N)modez*{} should stem to {} in {} mode but got {})rzipr4r format)r stemmer_modeexpected_stemsr"word true_stemour_stemr r r _test_against_expected_outputKs   z(PorterTest._test_against_expected_outputcCsRttdjdd}|tj|WddS1s"wYdS)azTests all words from the test vocabulary provided by M Porter The sample vocabulary and output were sourced from https://tartarus.org/martin/PorterStemmer/voc.txt and https://tartarus.org/martin/PorterStemmer/output.txt and are linked to from the Porter Stemmer algorithm's homepage at https://tartarus.org/martin/PorterStemmer/ z-stemmers/porter_test/porter_martin_output.txtr+r,N) rrr.r/r=rMARTIN_EXTENSIONSr0r1r2r r r test_vocabulary_martin_modeXs "z&PorterTest.test_vocabulary_martin_modecCsRttdjdd}|tj|WddS1s"wYdS)Nz+stemmers/porter_test/porter_nltk_output.txtr+r,) rrr.r/r=rNLTK_EXTENSIONSr0r1r2r r r test_vocabulary_nltk_modejs "z$PorterTest.test_vocabulary_nltk_modecCstttdjdd}|tj|Wdn1s!wY|tjtdjdddS)Nz/stemmers/porter_test/porter_original_output.txtr+r,) rrr.r/r=rORIGINAL_ALGORITHMr0r1r2r r r test_vocabulary_original_modets$  z(PorterTest.test_vocabulary_original_modecCstddks JdS)zTest for bug https://github.com/nltk/nltk/issues/1581 Ensures that 'oed' can be stemmed without throwing an error. oedoNrr )r r r r test_oed_bugszPorterTest.test_oed_bugcCslt}|ddks J|ddksJ|jddddks J|ddks)J|jddddks4Jd S) zTest for improvement on https://github.com/nltk/nltk/issues/2507 Ensures that stems are lowercased when `to_lowercase=True` OnonIiF) to_lowercaseGithubgithubNrF)r porterr r r test_lowercase_options z PorterTest.test_lowercase_optionN) r'r(r)r4r=r?rArCrGrPr r r r r*Bs   r*) unittest contextlibrnltkrnltk.stem.porterrnltk.stem.snowballrTestCaserr*r r r r s    9