Compiler Design

Waqar Ahmed
i am computer system Engineering form BUET khuzdar pakistan

1. Compiler Design Lec # 3 Token, Pattern and Lexems Tokens . Tokens are terminal symbols of the source language. e.g. Identifiers, number, key words, punctuation and symbols etc. Patten is a rule describing all those lexemes that represent a particular token in a source language. Ids: start with an alphabet or -, followed by any alphanumeric char. Lexemes are matched against the pattern ---Specific instance of a token . Count = count + temp; ( there are three variable count , count and temp) ( count is token as a identifier ) (= assignment Operator , + addition operator and punctuation ;) There are three tokens Tokens: identifier, operator and punctuation;

2. Tokens A  token  is a pair consisting of a  token  name and an optional attribute value. The  token  name is an abstract symbol representing a kind of lexical unit, e.g., a particular keyword, or sequence of input characters denoting an identifier. The  token  names are the input symbols that the parser processes. Pattern pg. 111.

3. What is lexemes A  Lexeme  is a string of characters that is a lowest-level syntactic unit in the programming language. These are the "words" and punctuation of the programming language. A  Token  is a syntactic category that forms a class of  lexemes .

4. What is lexeme with example? It is a basic abstract unit of meaning, a unit of morphological analysis in linguistics that roughly corresponds to a set of forms taken by a single root word. For  example , in English, run, runs, ran and running are forms of the same  lexeme , which can be represented as RUN.

5. what is pattern in compiler Pattern : A set of strings in the input for which the same token is produced as output. This set of strings is described by a rule called a  pattern  associated with the token. Lexeme: A lexeme is a sequence of characters in the source program that is matched by the  pattern  for a token

6. Token:  Token is a sequence of characters that can be treated as a single logical entity. Typical tokens are, 1) Identifiers 2) keywords 3) operators 4) special symbols 5)constants Pattern:  A set of strings in the input for which the same token is produced as output. This set of strings is described by a rule called a pattern associated with the token. Lexeme:  A lexeme is a sequence of characters in the source program that is matched by the pattern for a token.