R.E, Text Normalization, Tokenization ALgs, BPE.pdf

About core NLP

1. Foundations of NLP
CS3126
Lecture-1
1.1 Regular Expressions
1.2 TextNormalization
1.3 Tokenization algorithms
1

2. Acknowledgments
These slides were adapted from the book
SPEECH and LANGUAGE PROCESSING: An Introduction to Natural
Language Processing, Computational Linguistics, andSpeech
Recognition and
Some modifications from presentations and resources found inthe
WEB by several scholars.
2

3. Recap
•Regular Expressions
•Applications
•Regex rules
•Class activity
3

4. Class Activity
Goal: To match regular expressions
Task: To generate a large dataset of student records and write a Python program to match
specific regular expressions against the data.
You will generate a dataset consisting of 100,000 student records. Each record will contain the
following fields:
•Student Name: A random name composed of a first and last name.
•Roll Number: A unique identifier, such as SE22UARI001 (where "SE22UARI" is a batch code
and the digits are a sequence).
•Courses Taken: A list of 3-5 courses represented by course codes like CS3126, CS3202, etc.
•Email: A randomly generated email address associated with the student with domain name
@mahindrauniversity.edu.in.
•Section: The section AI1,AI2, AI3 etc.
Outcome: By the end of this activity, you should be able to: 1. Understand the use of regular
expressions in data filtering. 2. Generate large datasets programmatically. 3. Apply regular
expressions effectively to extract meaningful information from datasets.

5. Text Normalization
5
Slide Reference:2_TextProc_Mar_25_2021.pdf (stanford.edu)

6. Real world issues in text that need Normalization
6
Industry examples:
Recruitment Domain
E-commerce
and many more........

7. Research- Real-world/Industry use-case
7
https://cdn.iiit.ac.in/cdn/precog.iiit.ac.in/p
ubs/2021_July_KCNet-slides.pdf

8. Basic Normalization steps:
1.Segmenting/tokenizing words in running text
2.Normalizing word formats
3.Segmenting sentences in running text
8
Slide Reference:2_TextProc_Mar_25_2021.pdf (stanford.edu)

9. How many words in a sentence?
they lay back on the San Francisco grass and looked at the stars
and their
Type: an element of the vocabulary.
Token: an instance of that type in running text.
How many?

10. How many words in a sentence?
they lay back on the San Francisco grass and looked at the stars and
their
Type: an element of the vocabulary.
Token: an instance of that type in running text.
How many?
◦ 15 tokens (or 14)
◦ 13 types (or 12) (or 11?)

11. How many words in a corpus?
N = number of tokens
V = vocabulary = set of types
|V| is size of vocabulary

12. Heaps law/Herden's law
N = number of tokens
V = vocabulary = set of types
|V| is size of vocabulary
Heaps Law = Herdan's Law = |V|= = kN
b
where often .67 < β < .75
i.e., vocabulary size grows with > square root of the number of word
tokens
Heaps' law: Estimating the number of terms (stanford.edu)

13. Heaps law/Herden's law
Heaps' law - Wikipedia

14. How many words in a corpus?
Tokens = NTypes = |V|
Switchboard phone
conversations
2.4 million20 thousand
Shakespeare 884,00031 thousand
COCA 440 million2 million
Google N-grams 1 trillion13+ million

15. Corpora
Wordsdon'tappearoutofnowhere! Atextisproducedby
•aspecificwriter(s),
•ataspecifictime,
•inaspecificvariety,
•ofaspecificlanguage,
•foraspecificfunction.

16. Corporavaryalongdimensionlike
◦Language:7097languagesintheworld
◦Variety,likeAfricanAmericanLanguagevarieties.
◦AAETwitterpostsmightincludeformslike"iont"(Idon't)
◦Codeswitching,e.g.,Spanish/English,Hindi/English:
S/E:Porprimeravezveoa@usernameactuallybeinghateful!Itwasbeautiful:)
[ForthefirsttimeIgettosee@usernameactuallybeinghateful!itwasbeautiful:)]
H/E:dostthaorra-hega...dontwory...butdheryarakhe
[“hewasandwillremainafriend...don’tworry...buthavefaith”]
◦Genre:newswire,fiction,scientific articles,Wikipedia
◦AuthorDemographics:writer'sage,gender,ethnicity,SES

17. Tokenization
Input: Mahindra university department
Tokens:
Mahindra
University
Department
A token is a sequence of characters in a document
17

18. Space-based tokenization
A very simple way to tokenize
•For languages that use space characters between words Arabic,
Cyrillic, Greek, Latin, etc., based writing systems
•Segment off a token between instances of spaces

19. SimpleTokenizationinUNIX
(InspiredbyKenChurch’sUNIXforPoets.)
Givenatextfile,outputthewordtokensandtheirfrequencies
tr-sc’A-Za-z’’\n’<shakes.txt
|sort
|uniq–c
1945A
72AARON
19ABBESS
5ABBOT
......
25Aaron
6Abate
1Abates
5Abbess
6Abbey
3Abbot
....…
Changeallnon-alphatonewlines
Sortinalphabeticalorder
Mergeandcounteachtype

20. Thefirststep:tokenizing
tr-sc’A-Za-z’
THE
SONNETS
by
William
Shakespeare
From
fairest
creatures
We
...
’\n’<shakes.txt|head

21. Thesecondstep:sorting
tr-sc’A-Za-z’’\n’<shakes.txt|sort|head
A
A
A
A
A
A
A
A
A
...

22. More counting
Mergingupperandlowercase
tr‘A-Z’‘a-z’<shakes.txt|tr–sc‘A-Za-z’‘\n’|sort|uniq–c
Sortingthecounts
tr‘A-Z’‘a-z’<shakes.txt|tr–sc‘A-Za-z’‘\n’|sort|uniq–c|sort–n–r
23243the
22225i
18618and
16339to
15687of
12780a
12163you
10839my
10005in
8954d
Whathappenedhere?

23. IssuesinTokenization
Can'tjustblindlyremovepunctuation:
◦m.p.h.,Ph.D.,AT&T,cap’n
◦prices($45.55)
◦dates(01/02/06)
◦URLs(http://www.stanford.edu)
◦hashtags(#nlproc)
◦emailaddresses([email protected])
Clitic:awordthatdoesn'tstandonitsown
◦"are"inwe're,French"je"inj'ai, "le"inl'honneur
Whenshouldmultiwordexpressions(MWE)bewords?
◦NewYork,rock’n’roll

24. What are valid tokens?
Hewlett-Packard Company
Are these two tokens "Hewlett" or "Packard" or one token?
Mahindra university -> 1 token or two?
State-of-the-art-> how many tokens?
Language issues-> Left-right or right-left (For example: Arabic)
24

25. Simple Code Example (Python NLTK)
25
Source: https://web.stanford.edu/~jurafsky/slp3/2.pdf

26. Tokenizationinlanguageswithoutspaces
Manylanguages(likeChinese,Japanese,Thai)don't
usespacestoseparatewords!
Howdowedecidewherethetokenboundaries
shouldbe?

27. Wordtokenization inChinese
•Chinesewordsarecomposedofcharacterscalled
"hanzi"(orsometimesjust"zi")
•Eachonerepresentsameaningunitcalledamorpheme. Each
wordhasonaverage2.4ofthem.
•Butdecidingwhatcountsasawordiscomplexandnot agreed
upon.

28. HowtodowordtokenizationinChinese?
姚明进入总决赛“YaoMingreachesthefinals”
3words?
姚明 进入总决赛
YaoMingreachesfinals
5words?
姚 明进入 总 决赛
YaoMingreachesoverallfinals
7characters?(don'tusewordsatall):
姚明 进入 总 决 赛
YaoMingenterenteroveralldecisiongame

29. HowtodowordtokenizationinChinese?
姚明进入总决赛“YaoMingreachesthefinals”
3words?
姚明 进入总决赛
YaoMingreachesfinals
5words?
姚 明进入 总 决赛
YaoMingreachesoverallfinals
7characters?(don'tusewordsatall):
姚明 进入 总 决 赛
YaoMingenterenteroveralldecisiongame

30. HowtodowordtokenizationinChinese?
姚明进入总决赛“YaoMingreachesthefinals”
3words?
姚明 进入总决赛
YaoMingreachesfinals
5words?
姚 明
YaoMing
进入
reaches
总
overall
决赛
finals
7characters?(don'tusewordsatall):
姚明 进入 总 决 赛
YaoMingenterenteroveralldecisiongame

31. HowtodowordtokenizationinChinese?
姚明进入总决赛“YaoMingreachesthefinals”
3words?
姚明 进入总决赛
YaoMingreachesfinals
5words?
姚 明
YaoMing
进入
reaches
总
overall
决赛
finals
7characters?(don'tusewordsatall):
姚明 进入 总 决 赛
YaoMingenterenteroveralldecisiongame

32. Wordtokenization /segmentation
•SoinChineseit'scommontojusttreateachcharacter (zi)
asatoken.
•Sothesegmentationstepisverysimple
•Inotherlanguages(likeThaiandJapanese),more
complexwordsegmentationisrequired.
•Thestandardalgorithmsareneuralsequencemodels
trainedbysupervisedmachinelearning.

33. Anotheroptionfortexttokenization
Insteadof
•white-spacesegmentation
•single-charactersegmentation
Usethedatatotellushowtotokenize.
Subwordtokenization(becausetokenscanbeparts
ofwordsaswellaswholewords)

34. Complexity in Word tokenization
Word tokenization is more complex in languages like written
Chinese, Japanese, and Thai, which do not use spaces to mark
potential word-boundaries
Another Solution-
Byte-pair encoding – [Read the example from book]
34

35. Subwordtokenization
Threecommonalgorithms:
◦Byte-PairEncoding(BPE)(Sennrichetal.,2016)
◦Unigramlanguagemodelingtokenization(Kudo,2018)
◦WordPiece(SchusterandNakajima,2012)
Allhave2parts:
◦Atokenlearnerthattakesarawtrainingcorpusand
induces a vocabulary (asetof tokens).
◦Atokensegmenterthattakesarawtestsentence
and tokenizesitaccordingtothatvocabulary

36. BytePairEncoding(BPE)tokenlearner
Letvocabularybethesetofallindividualcharacters
= {A,B,C,D,…,a,b,c,d….}
Repeat:
◦Choosethetwosymbolsthataremostfrequently
adjacentinthetrainingcorpus(say'A','B')
◦Addanewmergedsymbol'AB'tothe vocabulary
◦Replaceeveryadjacent'A''B'inthecorpuswith'AB'.
Untilkmergeshavebeendone.

37. BPEtoken learneralgorithm
functionBYTE-PAIRENCODING(stringsC,numberofmergesk)returnsvocabV
V←alluniquecharactersinC
fori=1tokdo
#initialsetoftokensischaracters
#mergetokenstilktimes
tL,tR←Mostfrequentpairof adjacenttokensinC
tNEW←tL+tR #makenewtokenbyconcatenating
V←V+tNEW # update the vocabulary
Replaceeachoccurrenceof tL,tRinCwithtNEW # and update the corpus
returnV

38. BytePairEncoding(BPE)Addendum
Mostsubwordalgorithmsareruninsidespace-
separatedtokens.
Sowecommonlyfirstaddaspecialend-of-word
symbol''beforespaceintrainingcorpus
Next,separateintoletters.

39. BPEtoken learner
Original(veryfascinating)corpus:
lowlowlowlowlowlowestlowestnewernewernewer
newernewernewerwiderwiderwidernewnew
Addend-of-wordtokens,resultinginthisvocabulary:
vocabulary
,d,e,i,l,n,o,r,s,t,w
corpusrepresentation
5low
2lowest
6newer
3wider
2new

40. BPEtoken learner
corpus vocabulary
5low ,d,e,i,l,n,o,r,s,t,w
2lowest
6newer
3wider
2new
Mergeertoer
corpus vocabulary
5 low , d,e,i,l,n,o,r,s,t,w,er
2 lowest
6newer
3wider
2new

41. BPE
corpus vocabulary
5 low , d,e,i,l,n,o,r,s,t,w,er
2 lowest
6newer
3wider
2new
Mergeer_toer_
corpus
5l o w
2l o w e s t
6newer
3wider
2n e w
vocabulary
,d,e,i,l,n,o,r,s,t,w,er,er

42. BPE
Mergenetone
corpus vocabulary
5 l o w ,d,e,i,l,n,o,r,s,t,w,er,er ,ne
2 lowes t
6 newer
3 wider
2 ne w
corpus vocabulary
5 low ,d,e,i,l,n,o,r,s,t,w,er,er
2 lowest
6 newer
3 wider
2 new

43. BPE
Thenextmergesare:
CurrentVocabulary
,ne,
new,ne, new, lo
,ne, new,lo, low
,ne, new,lo, low, newer
Merge
(ne, w)
(l, o)
(lo, w)
(new, er
(low,)
,d, e,i, l,n, o,r, s,t, w,er, er
,d, e,i, l,n, o,r, s,t, w,er, er
,d, e,i, l,n, o,r, s,t, w,er, er
),d, e,i, l,n, o,r, s,t, w,er, er
,d, e,i, l,n, o,r, s,t, w,er, er,ne, new,lo, low, newer, low

44. BPEtokensegmenteralgorithm
Onthetestdata,runeachmergelearnedfromthe
trainingdata:
◦Greedily
◦Intheorderwelearnedthem
◦(testfrequenciesdon'tplayarole)
So:mergeeveryertoer,thenmergeer_toer_,etc.
Result:
◦Testset"ne we r_"wouldbetokenizedasafullword
◦Testset"low er_"would betwotokens:"lower_"

45. PropertiesofBPEtokens
Usuallyincludefrequentwords
Andfrequentsubwords
•Whichareoftenmorphemeslike-estor–er
Amorphemeisthesmallestmeaning-bearingunitofa
language
•unlikeliesthas3morphemesun-,likely,and-est

46. SentenceSegmentation
!,?mostlyunambiguousbutperiod“.”isveryambiguous
◦Sentenceboundary
◦AbbreviationslikeInc.orDr.
◦Numberslike.02%or4.3
Commonalgorithm:Tokenizefirst:userulesorMLtoclassifyaperiodaseither
(a)partofthewordor(b)asentence-boundary.
◦Anabbreviationdictionarycanhelp
Sentencesegmentationcanthenoftenbedonebyrulesbasedonthistokenization.

47. Implementation/Tokenization
https://huggingface.co/learn/nlp-course/en/chapter6/5
https://github.com/SumanthRH/tokenization
47

48. Class Activity
•Implement Byte Pair Encoding Algorithm from scratch and use the
below corpus:
fast-bpe/tinyshakespeare.txt at main · IAmPara0x/fast-bpe
(github.com)

49. Other tokenizers
49
•Word piece tokenizers [https://huggingface.co/learn/nlp-course/chapter6/6]
•Sentence piece tokenizers [https://github.com/google/sentencepiece]

50. WordNormalization
Puttingwords/tokensinastandardformat
◦U.S.A.orUSA
◦uhhuhoruh-huh
◦Fedorfed
◦am, is,be, are

51. Case folding
•Applications such as Information retrieval: reduce all letters to
lower case
oSince all users tend to use lower case
oPossible exceptions: uppercase in mid-sentence?

52. Case folding
•Applications such as Information retrieval: reduce all letters to
lower case
oSince all users tend to use lower case
oPossible exceptions: uppercase in mid-sentence?
oGeneral Motors
oFed vs. Fed
oSAIL vs. sail

53. Case folding
•Applications such as Information retrieval: reduce all letters to
lower case
oSince all users tend to use lower case
oPossible exceptions: uppercase in mid-sentence?
oGeneral Motors
oFed vs. Fed
oSAIL vs. Sail
•Sentiment analysis (US and us have different meanings)

54. Morphology
•Morphemes:
◦The small meaningful units that make up words
◦Stems: The core meaning-bearing units
◦Affixes: Parts that adhere to stems, often with grammatical
functions
•Morphological Parsers:
◦Parse cats into two morphemes cat and s
◦Parse Spanish amaren (‘if in the future they would love’) into
morpheme amar ‘to love’, and the morphological features 3PL and
future subjunctive.

55. Dealingwithcomplexmorphologyisnecessary
formanylanguages
◦e.g.,theTurkishword:
◦Uygarlastiramadiklarimizdanmissinizcasina
◦`(behaving)asifyouareamongthosewhomwecouldnotcivilize’
◦Uygar`civilized’+las`become’
+tir`cause’+ama`notable’
+dik`past’+lar‘plural’
+imiz‘p1pl’+dan‘abl’
+mis‘past’+siniz‘2pl’+casina‘asif’

56. Objective
•The goal of both stemming and lemmatization is to reduce
inflectional forms and sometimes derivationally related forms of a
word to a common base form.
For instance:
•am, are, isbe
car, cars, car's, cars'car
•The result of this mapping of text will be something like:
othe boy's cars are different colors
othe boy car be differ color

57. Stemming in NLP
57

58. Stemming
58
https://en.wikipedia.org/wiki/Stemming#

59. Stemming
Reducetermstostems,choppingoffaffixescrudely
Thiswasnotthemapwe
foundinBillyBones’s
chest,butanaccurate
copy,completeinall
things-namesandheights
andsoundings-withthe
singleexceptionofthe
redcrossesandthe
writtennotes.
Thiwanotthemapwe
foundinBilliBoneschest
butanaccurcopicomplet
inallthingnameand
heightandsoundwiththe
singlexceptofthered
crossandthewrittennote
.

60. Stemming
Stemming suggests crude affix chopping
•language dependent
•automation, automatic, automate ---(automat)
Stemming programs are called as Stemmers or Stemming
algorithms
Porter Stemming Algorithm (tartarus.org)
60

61. The Porter Stemmer (Porter, 1980)
•Common Algorithm for English language
•A simple rule-based algorithm for stemming
•An example of a HEURISTIC method
•Based on rules like:
•ATIONAL -> ATE (e.g., relational -> relate)
•The algorithm consists of 7 sets of rules, applied in order
61
tartarus.org/martin/PorterStemmer/def.txt

62. The Porter Stemmer: definitions
•Definitions:
•CONSONANTS: a letter other than A, E, I, O, U, and Y preceded by
consonant
•VOWEL: any other letter (if the letter is not a consonant)
•With this definition, all words are of the form: (C)(VC)
m
(V)
•C: string of one or more consonants (con+)
•V: string of one or more vowels
•m: measure of word or word part which is represented in form of VC
•E.g.
•Troubles
•C (VC)
m
V
62
tartarus.org/martin/PorterStemmer/def.txt

63. Measure of the word
•M=0 TREE, BY, TR
•M=1 TROUBLE, OATS, TREES, IVY
•M=2 TROUBLES, PRIVATE, OATEN
63
tartarus.org/martin/PorterStemmer/def.txt

64. The Porter Stemmer: Rule format
•The rules are of the form:
•(condition) S1 -> S2 where S1 and S2 are suffixes
•If the rule (m>1) EMENT->
•In this S1 is EMENT and S2 is NULL
•So, this would map REPLACEMENT with REPLAC
64
tartarus.org/martin/PorterStemmer/def.txt

65. Conditions
m The measure of the stem
*S The stem ends with S
*v* The stem contains a vowel
*d The stem ends with a double consonant (TT,SS)
*o The stem ends in CV C (second C not W, X, or Y) Ex: WIL, HOP
65
The condition may also contains expressions with and, or, or not
Example ( (m>1) and (*s or*t)) -tests for a stem with m>1 ending in s or t
tartarus.org/martin/PorterStemmer/def.txt

66. The Porter Stemmer: Step 1
•SSES -> SS
•caresses -> caress
•IES -> I
•ponies -> poni
•ties -> ti
•SS -> SS
•caress -> caress
•S ->€
•cats -> cat
66
tartarus.org/martin/PorterStemmer/def.txt

67. The Porter Stemmer: Step 2a (past tense,
progressive)
•(m>0) EED -> EE
•Condition verified: agreed -> agree
•Condition not verified: feed -> feed
•(*V*) ED -> €
•Condition verified: plastered -> plaster
•Condition not verified: bled -> bled
•(*V*) ING ->€
•Condition verified: motoring -> motor
•Condition not verified: sing -> sing
67
tartarus.org/martin/PorterStemmer/def.txt

68. The Porter Stemmer: Step 2b (cleanup)
•(These rules are ran if second or third rule in 2a apply)
•AT -> ATE
•Conflat(ed) -> conflate
•BL -> BLE
•Troubl(ing) - > trouble
•( *d & ! (*L or *S or *Z)) -> single letter
•Condition verified: hopp(ing) -> hop, tann(ed) -> tan
•Condition not verified: fall(ing) -> fall
•(m=1 & *o) -> E
•Condition verified: fil(ing) -> file
•Condition not verified: fail -> fail
68
tartarus.org/martin/PorterStemmer/def.txt

69. The Porter Stemmer: step 3 and 4
•Step 3: Y elimination (*V*) Y -> I
•Condition verified:happy -> happi
•Condition not verified: sky -> sky
•Step 4: Derivational Morphology, I
•(m>0) ATIONAL -> ATE
•Relational -> relate
•(m>0) IZATION -> IZE
•Generalization -> generalize
•(m>0) BILITI -> BLE
•Sensibiliti -> sensible
69tartarus.org/martin/PorterStemmer/def.txt

70. Porter Stemmer Step 5 and Step 6
•Derivational Morphology II
•(m>0) ICATE-> IC
•Triplicate-> Triplic
•(m>0) FUL ->€
•hopeful-> hope
•(m>0) NESS->€
•goodness->good
•Derivational Morphology III
•(m>1) ANCE-> €
•allowance->allow
•(m>1) ENT ->€
•dependent-> depend
•(m>1) IVE->€
•effective->effect
70tartarus.org/martin/PorterStemmer/def.txt

71. The porter stemmer Step 7 (cleanup)
•Step 7a
•(m>1) E ->€
•Probate ->probat
•(m=1 & !*o) NESS ->€
•Goodness -> good
•Step 7 b
•(m>1 & *d & *L) -> single letter
•Condition verified: controll -> control
•Condition not verified: roll -> roll
71
tartarus.org/martin/PorterStemmer/def.txt

72. Advantages
•Speed and efficiency: Stemming algorithms are generally faster
as they follow simple rule-based approaches.
•Simplicity: The algorithms for stemming use simple heuristic
rules, so they are less complex to implement and understand than
other methods.
•Improved search performance: In search engines and
information retrieval systems, stemming helps connect different
word forms, potentially increasing the breadth of search results.

73. Disadvantages
•Over-stemming and under-stemming:
Stemming can often be imprecise, leading to over-stemming (where
words are overly reduced and unrelated words are conflated) and
under-stemming (where related words don’t appear related).
•Language limitations:
The effectiveness of a stemming algorithm reduces if words appear
in irregular formats (i.e., irregular conjugated forms).

74. Lemmatization
•Lemmatization goes beyond truncating words and analyzes the
context of the sentence, considering the word's use in the larger
text and its inflected form.
•After determining the word's context, the lemmatization algorithm
returns the word's base form (lemma) from a dictionary reference.
•Task of determining whether two words have same root despite
surface differences
74

75. Lemmatization
75
https://en.wikipedia.org/wiki/Lemmatization

76. Lemmatization
•The most sophisticated methods
forlemmatizationinvolvecompletemorphologicalparsing of the word.
•Morphology is the study of morphemethe way words are built up from
smaller meaning-bearing units calledmorphemes.
•Two broad classes of morphemes can be distinguished:
•stems—the centralmorphemeof the word, supplying the mainmeaning— and
affixes—adding “additional” meanings of various kinds.
•So,for example, the word fox consists of one morpheme (the morpheme
fox)andthe word cats consists of two: the morpheme cat and the morpheme -
s.
76

77. Lemmatization
Representallwordsastheirlemma, theirsharedroot=dictionary
headwordform:
•am,are,isbe
•car,cars,car's,cars'car
•Spanishquiero (‘Iwant’),quieres(‘youwant’)querer‘want'
•HeisreadingdetectivestoriesHebereaddetectivestory

78. Lemmatization: Advantages
Accuracy and contextual understanding: Lemmatization is more
accurate as it considers words' context and the morphological
analysis. It can distinguish between different word uses based on
its part of speech.
Reduced ambiguity: By converting words to their dictionary form,
lemmatization reduces ambiguity and enhances the clarity of text
analysis.
Language and grammar compliance: Lemmatization adheres
more closely to the grammar and vocabulary of the target
language, leading to linguistically meaningful outputs.

79. Lemmatization: Disadvantages
Computational complexity:
Lemmatization algorithms are more complex and computationally
intensive than stemming. They require more processing power
and time.
Dependency on language resources:
Lemmatization depends on extensive language-specific resources
like dictionaries and morphological analyzers, making it less flexible
for use with certain languages, such as Arabic.

80. 80

81. In-class activity
Exercise1:
•Convert these list of words into base form using Stemming and
Lemmatization and observe the transformations
•['running', 'painting', 'walking', 'dressing', 'likely', 'children', 'whom', 'good',
'ate', 'fishing']
•Write a short note on the words that have different base words
using stemming and Lemmatization
81

82. In class activity
Write a python code to use NLTK library and convert the base forms
using different Stemmers and Lemmatizers
•use different stemmers and lemmatizers provided by NLTK
•seehttps://www.nltk.org/howto/stem.htmlfor full NLTK
stemmer module documentations
82

83. Reference materials
•https://vlanc-lab.github.io/mu-nlp-
course/teachings/fall-2024-AI-nlp.html
•Lecture notes
•(A) Speech and Language Processing
by Daniel Jurafsky and James H. Martin
•(B) Natural Language Processing with
Python. (updated edition based on
Python 3 and NLTK
•3) Steven Bird et al. O’Reilly Media
83