UNIT 1- B.pptx natural language processing

Notes of Natural language processing

1. NLP Basics of word level analysis

2. Word Level Analysis in NLP Introduction Word Level Analysis is the process of examining the structure, meaning, and relationships of words in text, crucial for NLP tasks. Goals include understanding and processing language at the most granular level.

3. Key Components Morphology:  Study of word structure and formation (prefixes, suffixes, roots). Tokenization:  Splitting text into words/tokens for further analysis. Part-of-Speech Tagging:  Assigning grammatical roles to each word.

4. Morphological Analysis Inflectional Morphology:  Changes for grammatical info (tense, number, case). Derivational Morphology:  Formation of new words by adding affixes. Examples: "Walked" (inflection), "Happiness" (derivation).

5. Tokenization Process: Break sentences into tokens (words/punctuation). Example: "NLP is fun!" → ["NLP", "is", "fun", "!"].

6. Part-of-Speech(POS) Tagging Assigns tags like noun (NN), verb (VB), adjective (AJ) Example: "She runs fast." → She (PR), runs (VB), fast (AJ).

7. Stemming and Lemmatization Stemming:  Cutting words to root (e.g., "playing" → "play"). Lemmatization:  Reduces words to their dictionary form considering context ("better" → "good").

8. Applications Used in search engines, sentiment analysis, chatbots, machine translation. Improves accuracy, data structure, reduces complexity.

9. Example Workflow Input text → Tokenization → Morphology → POS tagging → Stemming/Lemmatization → Application (e.g., translation)

10. Challenges & Future Trends Ambiguity:  Same word, multiple meanings. Handling  multi-lingual text , informal writing , evolving language . Advances: Deep learning integration, context-aware models.