Relationship Among Token, Lexeme & Pattern

Relationship among Token, Lexeme and Pattern
Outline
Token
Lexeme
Pattern
Relationship

Tokens : A token is sequence of characters that can be treated
as a unit/single logical entity.

Examples

Keywords
Examples : for, while, if etc.
Identifier
Examples : Variable name, function name, etc.
Operators
Examples : '+', '++', '-' etc.
Separators
Examples : ',' ';' etc.


Pattern
Pattern is a rule describing all those lexemes that can represent a particular token in a source language.

Lexeme
It is a sequence of characters in the source program that is matched by the pattern for a token.

Example : “float”, “=“, “223”, “;”

1. Subject: Compiler Design Presentation On Relationship Among Tokens, Pattern & Lexemes Submitted To : Mr. Gaurav Dubey (Asst. Prof. , Department of Computer Science & Engineering) Submitted By : Bharat Rathore 0905CS191062 Activity - 1

2. OUTLINES Token Lexeme Pattern Relationship

3. Tokens Tokens : A token is sequence of characters that can be treated as a unit/single logical entity. Examples Keywords Examples : for, while, if etc. Identifier Examples : Variable name, function name, etc. Operators Examples : '+', '++', '-' etc. Separators Examples : ',' ';' etc.

4. Example on Tokens int a = 10 ; Keyword Constant Identifier Separator Operator

5. Lexeme It is a sequence of characters in the source program that is matched by the pattern for a token. Example : “float”, “=“, “223”, “;”

6. Pattern Pattern is a rule describing all those lexemes that can represent a particular token in a source language.

7. Relationship Relationship Between Token, Pattern & Lexeme