Chapter 33

Chapter 3
Scanning - Theory and Practice
1 5/3/2018
Outlines
 3.1 Overview
 3.2 Regular Expressions
 3.4 Finite Automata and Scanners
 3.5 Using a Scanner Generator
 LEX --- Introduce in TA course: LEX introduction
 3.7 Practical Considerations
 3.8 Translating Regular Expressions into Finite Automata
 3.9 Summary
2
2 5/3/2018
Overview(1)
 Formal notations
 For specifying the precise structure of tokens are necessary
 Quoted string in Pascal
 Can a string split across a line?
 Is a null string allowed?
 Is .1 or 10. ok?
 The 1..10 problem
 Scanner generators
 Tables, Programs
 What formal notations to use?
3 5/3/2018
Overview(2)
 Lexical analyzer (scanner) role
 Produce a sequence of (tokens) for parser
 Stripe out comments and whitespaces
 Associate a line number with each error message
 Expand macros
source Lexical token to semantic

program Parser analysis
Analyzer
getNextToken
Symbol
Table
4 5/3/2018
Regular Expressions (1)
 Tokens
 built from symbols of a finite vocabulary.
 Structures of tokens
 use regular expressions to define
 Set Definition
 The sets of strings defined by regular expressions are termed
  is a regular expression denoting the empty set
  is a regular expression
denoting the set that contains only the empty string
 A string s is a regular expression denoting a set containing only s
5 5/3/2018
Regular Expression (2)
 if A and B are regular expressions, so are
 A | B (alternation)
 A regular expression formed by A or B
 (a)|(b) = {a, b}
 AB or A•B (concatenation)
 A regular expression formed by A followed by B
 (a)(b) = {ab}
 A* (Kleene closure)
 A regular expression formed by zero or more repetitions of A
 a* = {, a, aa, aaa, …}
More Complex Example
(a|b|c)* = {, a, b, c, aa, ab, ac, ba, bb, bc, ca, cb, cc …}
6 5/3/2018
Regular Expression (3)
 Some notational convenience
 P+
 PP* (at least one)
More Complex Example
 Not(A)  Let D = (0 | 1 | 2 | 3 | 4 | ... | 9 )
 Let L = (A | B | ... | Z | a | b |... | z)
 V-A
 comment = -- not(EOL)* EOL
 Not(S)  ex: --hello12_34 \n
 V* - S
 decimal = D+ · D+
 ex: 123.456
 AK
 AA …A (k copies)  ident = L (L | D | _)*
 ex: A1a5_6
 A?  comments = ((#|  ) not(#))*

 Optional, zero or one occurrence of A  ex:#A435#3
7 5/3/2018
Regular Expressions (4)
 Is regular expression as power as CFG?
 { [i ]i | i1}
 Regular grammar
Aa Aa
or
AaB ABa
8 5/3/2018
Finite Automata and Scanners (1)
 Finite automaton (FA)
 can be used to recognize the tokens specified by a regular expression
 A FA consists of
 A finite set of states S
 A set of input symbols  (the input symbol alphabet)
 A set of transitions (or moves) from one state to another,
labeled with characters in V
 A special start state s0 (only one)
 A set of final, or accepting, states F
FA = {S, , s0, F, move }
9 5/3/2018
is a state
is a transition
is the start state
is a final state
Example at next page….

10 5/3/2018
 Example
 A transition diagram
 This machine accepts (abc+)+
( a b c +) +
a b c
11 5/3/2018
 Other Example
 (0|1)*0(0|1)(0|1)
0 0,1 0,1
1 2 3 4
0,1 0
5
0,1
(0|1) * 0 (0|1)(0|1)
12 5/3/2018
 Other Example
 ID = L(L|D)*(_(L|D)+)* Final for two * symbol
L (L|D)* (_(L|D)+)*
L|D
L -
What difference?
L|D Answer : “_” by times
item 2 = item 3
 A data structure can be translated for many REs or FAs

13 5/3/2018
 Other Example
 RealLit = (D+(|.))|(D*.D+)
14 5/3/2018
 Two kinds of FA:
 Deterministic: next transition is unique
 Non-deterministic: otherwise
a ...
a
...
Which path we should select?

15 5/3/2018
 A transition diagram Not(Eol)
/ / Eol
1 2 3 4
 A transition table
State Character
- Eol a b …
1 2
2 3
3 3 4 3 3 3
4
16 5/3/2018
 Any regular expression
 can be translated into a DFA
that accepts the set of strings denoted by the regular expression
 The transition can be done

 Automatically by a scanner generator : LEX (TA course)
 Manually by a programmer :
 Coding the DFA in two form
 1. Table-driven, commonly produced by a scanner generator
 2. Explicit control, produced automatically or by hand
17 5/3/2018
 Scanner Driver Interpreting a Transition Table
/* Note: CurrentChar is already set to the current input character. */
State = StartState;
while (TRUE) {
NextSate = T[State , CurrentChar];
if (NextSate == ERROR)
Table-driven
break;
State = NextState;
CurrentChar = getchar();
}
If(is_final_state(State))
/* Return or process valid token. */
else
lexical_error(CurrentChar);
18 5/3/2018
 Scanner with Fixed Token Definition
if (CurrentChar == ‘/') {
CurrentChar = getchar();
if (CurrentChar == ‘/') {
do {
CurrentChar = getchar(); Explicit
} while (CurrentChar != '\n'); control
} else {
ungetc(CurrentChar, stdin);
}
}
else
/* Return or process valid token. */
19 5/3/2018
 Transducer
 We may perform some actions during state transition.
 A scanner can be turned into a transducer by the

appropriate insertion of actions based on state
transitions
20 5/3/2018
Using a Scanner Generator
 By TA….
21 5/3/2018
Practical Considerations (1)
 Reserved Words
 Usually, all keywords are reserved in order to simplify parsing.
 In Pascal, we could even write
 begin
 begin; end; end; begin;
 end
 if else then if = else;
 The problem
 with reserved words is that they are too numerous.
 COBOL has several hundreds of reserved words!
 ZEROS
 ZERO
 ZEROES
22 5/3/2018
 Compiler Directives and Listing Source Lines
 Compiler options e.g. optimization, profiling, etc.
 handled by scanner or semantic routines
 Complex pragmas are treated like other statements.
 Source inclusion
 e.g. #include in C
 handled by preprocessor or scanner
 Conditional compilation
 e.g. #if, #endif in C
 useful for creating program versions
23 5/3/2018
 Entry of Identifiers into the Symbol Table
 Who is responsible for entering symbols into symbol table?
 Scanner?
 Consider this example:

 { int abc;
 …
 { int abc; }
 }
24 5/3/2018
 How to handle end-of-file?
 Create a special EOF token.
 EOF token is useful in a CFG
 Multicharacter Lookahead
 Blanks are not significant in Fortran
 DO 10 I= 1,100
 Beginning of a loop
 DO 10 I = 1.100
 An assignment statement DO 10I=1.100
 A Fortran Scanner
 can determine whether the O is the last character of a DO token only
after reading as far as the comma
25 5/3/2018
 Multicharacter Lookahead (Cont’d)
 In Ada and Pascal
 To scan 10..100
 There are three token
 10
 ..
 100
 Two-character (..) lookahead after the 10
 It is easy to build a scanner that can perform general backup.

 If we reach a situation Example at next page
 in which we are not in final state and
cannot scan any more characters, we extract characters from the right end
of the buffer and queue them fir rescanning
 Until we reach a prefix of the scanned characters flagged as a valid token
26 5/3/2018
 An FA That Scans Integer and Real Literals and the
Subrange Operator
D D
D ● D
Buffered Token Token Flag

● ● 1 Integer Literal
12 Integer Literal
12. Invalid
12.3 Real Literal
Detail Operation of each case at next page 12.3e Invalid
27 12.3e+ Invalid
5/3/2018
Subrange Operator
Input string: 12.3e+q
Input Token Buffered Token Token Flag

1 1 Integer Literal
28 5/3/2018
Subrange Operator
D

1 2 1 2 Integer Literal
29 5/3/2018
Subrange Operator
D
D ●

1 2 . 1 2 . Invalid
30 5/3/2018
Subrange Operator
D
D ● D

1 2 . 3 1 2 . 3 Real Literal
31 5/3/2018
Subrange Operator
D
?
D ● D
Backup is invoked!

1 2 . 3 e 1 2 . 3 e Invalid
32 5/3/2018
Subrange Operator
D
?
D ● D
Backup is invoked!

1 2 . 3 e 1 2 . 3 e Invalid
33 5/3/2018
Subrange Operator
D
? ?
D ● D

+ +
1 2 . 3 e 1 2 . 3 e Invalid
34 5/3/2018
cannot scan any more characters, and not in accept state
 Backup is invoked !
D
? ?
D ● D

+ +
1 2 . 3 e 1 2 . 3 e Invalid
35 5/3/2018
Outlines
 3.1 Over View
 3.2 Regular Expression
 Creating Deterministic Automata
 Optimizing Finite Automata
 3.7 Tracing Example
36 5/3/2018
Translating Regular Expressions
into Finite Automata(1)
 Regular expressions are equivalent to FAs
 The main job of a scanner generator

 To transform a regular expression definition into an equivalent FA
Importance in NFA->DFA
A regular expression Nondeterministic FA Deterministic FA
minimize # of states
Optimized Deterministic FA
37 5/3/2018
 We can transform any regular expression
into an NFA with the following properties:
 There is an unique final state
 The final state has no successors
 Every other state has at least one successors
 Example : A Nondeterministic Finite Automaton (NFA)

 Input : babb
 Regular Expressions : (a|b)*abb
a
a b b
0 1 2 3
b
either one or two successors Unique final state Final S has no successor
38 5/3/2018
 We need to review the definition of regular expression
 Item 1: 
 It is null string
 Item 2: a
 It is a char of the vocabulary
 Item 3 : |
 It is “or” operation.
 Example : A|B
 Item 4 : ●
 It is the operation of catenation
 Example : AB
 Item 4 : *
 It is the operation of repetition
 Example : A*
More Example at Next Page
39 5/3/2018
NFA :  (null string)

Processing Token 

 NFA : a (1string) Processing Token a

 A char of the vocabulary
40 5/3/2018
NFA : A|B

Processing Token A| B
NFA 
For A

 NFA 
For B
41 5/3/2018
NFA : A●B

Processing Token A● B
 NFA 
For A
NFA
For B
42 5/3/2018
NFA : A*

A
*
Processing Token
 > 1 times
 NFA 
For A

= 0 times
43 5/3/2018
 Construct an NFA for Regular Expression 1*
 01* | 1  (0(1*)) |1
 1 
Start

*
Processing Token 1
44 5/3/2018
 Construct an NFA for Regular Expression 01*
 01*|1  (0(1*)) |1

Start
For Connection
 0   1 
*
Processing Token 01
45 5/3/2018
 Construct an NFA for Regular Expression 01*|1
 01*+1  (0(1*))+1

 0   1 


 
1 
Start
*
Processing Token 01 | 1
46 5/3/2018
 What’s problem about NFA?
 Ans: It may be ambiguous that difficult to program!!!
 A Nondeterministic Finite Automaton (NFA):(a|b)*abb
a b b
Start 0 1 2 3
b Ambiguous!!!
Which one should we select?
Input : babb Processing Token b a
47 5/3/2018
 What’s problem about NFA?
 Ans: It may be ambiguous that difficult to program!!!
 A deterministic Finite Automaton (NFA): b*abb
a b b
Start 0 1 2 3
b No Ambiguous!!!
It have unique path!
Input : babb Processing Token b a b b
48 5/3/2018
Creating Deterministic Automata(1)
 The transformation
 from an NFA N to an equivalent DFA M works
by what is sometimes called the subset construction
 An Example for each step…

 Initial NFA : 01*|1  (0(1*)) |1
 1 
Start 1 8 9 10

 
0   1 
2 3 4 5 6 7

49 More Detail operation at next page… 5/3/2018

 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7, 10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
More Detail operation at next page…

50 5/3/2018
 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
51 5/3/2018
 Step 1:
1
 8
1
 0
1. -closure(1) ={1, 2, 8}
52 5/3/2018
 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
53 5/3/2018
 Step 1:
2. -closure(2) ={2}
54 5/3/2018
 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
55 5/3/2018
 Step 1:
10

  1
3 4 5 7
3. -closure(3) ={3 ,4 ,5 ,7 ,10}
56 5/3/2018
 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
57 5/3/2018
 Step 1:
10

 1
4 5 7
4. -closure(4) ={4 ,5 ,7 ,10}
58 5/3/2018
 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
59 5/3/2018
 Step 1:
1
5
5. -closure(5) ={5}
60 5/3/2018
 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
61 5/3/2018
 Step 1:
10



5 6 7
This point line not be computed!!
6. -closure(6) ={5 ,6 ,7 ,10}
62 5/3/2018
 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
63 5/3/2018
 Step 1:
10


1
5 7
7. -closure(7) ={5, 7 ,10}
64 5/3/2018
 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
65 5/3/2018
 Step 1:
1
8
8. -closure(8) ={8}
66 5/3/2018
 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
67 5/3/2018
 Step 1:
9
 10
9. -closure(9) ={9 ,10}
68 5/3/2018
 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
69 5/3/2018
 Step 1:
10
10. -closure(10) ={10}
70 5/3/2018
 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
Total closures, but…..

71 5/3/2018
 Step 1:
Start 1
 8
1
9
 10

 0 
  1 
2 3 4 5 6 7
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={5, 7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
Delete Sub Set ...

72 5/3/2018
 Step 1:
Start 1
 8
1
9
 10 State 3
 0 state 3
 0  1 state 4
  1  2 state 5,7
2 3 4 5 6 7
3 state 10
 4 empty
1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

2. -closure(2) ={2} 7. -closure(7) ={7 ,10}
3. -closure(3) ={3 ,4 ,5 ,7 ,10} 8. -closure(8) ={8}
4. -closure(4) ={4 ,5 ,7 ,10} 9. -closure(9) ={9 ,10}
5. -closure(5) ={5} 10. -closure(10) ={10}
Now Closures, No Sub Set ...

73 5/3/2018
 Step 1:
 The initial state of M is the set of states reachable
from the initial state of N by -transitions
 Usually called l-closure or ε-closure
Algorithm for example at upside

74 5/3/2018
 Step 2:
 To create the successor states
 Take any state S of M and any character c, and compute S’s successor under c
 S is identified with some set of N’s states, {n1, n2,…}
 Find all possible successor states to {n1, n2,…} under c
 Obtain a set {m1, m2,…}
 T=close({m1, m2,…})
S T
{n1, n2,…} close({m1, m2,…})
75 5/3/2018
 Step 2:
void make_deterministic( nondeterministic_fa N , deterministic *M)
{
set_of_fa_states T;
M->initial_state = SET_OF(N.initial_state) ;
close (& M->initial_state );
Add M-> initial_state to M->states;
while( states or transitions can be added)

{
choose S in M->states and c in Alphabet;
T=SET_OF (y in N . states SUCH THAT x->y under c for some x in S);
close(& T);
if(T not in M->states)
add T to M->states;
Add the transition to M->transitions:

S->T under c;
}
M->final_states = SET_OF(S in M->states
SUCH_THAT N.final_state in S);
} Example at next page…
76 5/3/2018
 Step 2:
Start 1  8
1
9  10

 0 
2 3
 4
 5 1 6
 7
 First Re-Number for simplifying the work flow

1. -closure(1) ={1, 2, 8}  A = {1, 2, 8}
3. -closure(3) ={3 ,4 ,5 ,7 ,10}  B = {3 ,4 ,5 ,7 ,10}
6. -closure(6) ={5 ,6 ,7 ,10}  C = {5 ,6 ,7 ,10}
9. -closure(9) ={9 ,10}  D = {9, 10}
More Operation at next page ……

77 5/3/2018
Start 1  8
1
9  10
A : {1, 2, 8}
 B : {3 ,4 ,5 ,7 ,10}
 0 
2
3
 4
 5 1 6
 7
C : {5 ,6 ,7 ,10}
D : {9, 10}

A D
1
Start {1,2,8} {9, 10}
No
Out-Degree
0 1
B C
{3 ,4 ,5 1 {5 ,6 ,7
,7 ,10} ,10}
Final
78 5/3/2018
 Step 2:
void make_deterministic( nondeterministic_fa N , deterministic *M)
{
set_of_fa_states T;
M->initial_state = SET_OF(N.initial_state) ;
close (& M->initial_state );
Add M-> initial_state to M->states;
while( states or transitions can be added)

{
choose S in M->states and c in Alphabet;
T=SET_OF (y in N . states SUCH THAT x->y under c for some x in S);
close(& T);
if(T not in M->states)
add T to M->states;
Add the transition to M->transitions:

S->T under c;
}
M->final_states = SET_OF(S in M->states
SUCH_THAT N.final_state in S);
} Example at next page…
79 5/3/2018
Optimizing Finite Automata(1)
 Minimize number of states
 Every DFA has a unique smallest equivalent DFA
 Given a DFA M
we use Transition Table to construct the equivalent minimal DFA.
 Initially, we draw a transition table from DFA diagram.
Character
DFA State
0 1
1
Start A D
A B D
1
0
1
B C
Table C C
B C
A: Start State
80 5/3/2018
B,C,D: Final State
Optimizing Finite Automata(2)
 Minimize number of states
DFA New DFA

1
Start A D
1
1 Start A D
0
1
B C 0
1
B,C
Character
State Character
0 1 State
0 1
A B D
A {B, C} D
B C
{B, C} {B, C}
C C Optimize
D
D B is equal C
A: Start State
Special81
: 5/3/2018
B,C,D: Final State
B can merge into C, Because the B and C are final state.
Additional
 Simplifying rules (removing parentheses)
 “*” has highest precedence and is left associative
 Concatenation has 2nd highest precedence and is left associative

 “| “has lowest precedence and is left associative
 E.g., (a)|((b)*(c)) == a|b*c
82 5/3/2018
Outlines
 3.1 Over View
 3.2 Regular Expression
 3.7 Tracing Example
 Modify form http://www.cs.ualberta.ca/~amaral/courses/680/
83 5/3/2018
Tracing Example(1)
 Review Steps of Scanner Generator
Importance in NFA->DFA
A regular expression Nondeterministic FA Deterministic FA
Optimized Deterministic FA
84 5/3/2018
Tracing Example(2)
 Regular Expression
if {return IF;}
[a - z] [a – z|0 - 9 ] * {return ID;}
[0 - 9] + {return NUM;}
. {error ();}
 IF and IFA
85 5/3/2018
Tracing Example(3)
 Translate from RE to NFA
A regular Nondeterministic Deterministic

expression FA FA
Optimized
Deterministic
FA
86 5/3/2018
Tracing Example(4)
if {return IF;}
The NFA for a symbol i is: start 1 i 2
The NFA for a symbol f is: start 1 f 2
The NFA for the regular expression if is:
start i f
1 2 3
87
IF 5/3/2018
Tracing Example(5)
[a-z] [a-z|0-9 ] * {return ID;}
a-z 
start 1 a-z 2 3 4
0-9
 ID
88 5/3/2018
Tracing Example(6)
[0 – 9]+ {return NUM;}
start 1 0-9 2 0-9

5
3 4 

NUM
89 5/3/2018
Tracing Example(9)

i f
a-z 
1 2 3 1 a-z 2 3 4
0-9
ID 
IF
0-9 0-9
1 2 3 4 5 1 2

any but \n

NUM error
90 5/3/2018
Tracing Example(10)
 Translate from NFA to DFA

expression FA FA
Optimized
Deterministic
FA
91 5/3/2018
Tracing Example(11)
Full NFA Diagram

a-z
f IF  ID
2 3 a-z 
4 5 6 7 8
i 
0-9

1 
 0-9  0-9
NUM
14
any
character 15 9 10 11 12  13
error

Special case :Handle in Final
92 5/3/2018
Tracing Example(12)

IF
a-z
f ID
2 3 a-z 
i  4 5 6 7 8
0-9 
1  
NUM
 0-9 0-9
14
any
15 9 10 11 12  13
character
error

1. -closure(1) ={ 1, 4, 9, 14} 6. -closure(8) ={ 6, 8}
2. -closure(2) ={ 2} 7. -closure(10) ={ 10, 11, 13}
3. -closure(3) ={ 3} 8. -closure(12) ={12, 13}
4. -closure(5) ={ 5, 6, 8} 9. -closure(13) ={11, 13}
5.
93 -closure(7) ={ 7, 8} 10. -closure(15) ={15} 5/3/2018
Tracing Example(13)

IF
a-z
f ID
2 3 a-z 
i  4 5 6 7 8
0-9
1  

NUM
 0-9 0-9
14
any
15 9 10 11 12  13
character
error

DFA States = {1-4-9-14}
a-h Now we need to compute:
5-6-8-15
move(1-4-9-14,a-h) = {5,15}
1-4-9-14
-closure({5,15}) = {5,6,8,15}
94 5/3/2018
Tracing Example(16)

IF
a-z
f
2 3 a-z
6 7  8
i  4 5
0-9 
1  
 0-9 0-9
14
any
15 9 10 11 12  13
character
error
DFA States = {1-4-9-14}

move(1-4-9-14, i) = {2,5,15}
a-h 5-6-8-15
-closure({2,5,15}) = {2,5,6,8,15}
1-4-9-14
i 2-5-6-8-15
95 5/3/2018
Tracing Example(21)

IF
a-z
f ID
2 3 a-z
6 7  8
i  4 5
0-9

1  
NUM
 0-9
10
0-9
13
14
any
character 15 9 11 12 
error
DFA States = {1-4-9-14}

move(1-4-9-14, j-z) = {5,15}
a-h 5-6-8-15
j-z -closure({5,15}) = {5,6,8,15}
1-4-9-14
i 2-5-6-8-15
96 5/3/2018
Tracing Example(22)

IF
a-z
f ID
2 3 a-z
6 7  8
i  4 5
0-9
1  

NUM
 0-9 0-9
14
any
15 9 10 11 12  13
character
error

DFA States = {1-4-9-14}
move(1-4-9-14, 0-9) ={10,15}
10-11-13-15 a-h 5-6-8-15
j-z -closure({10,15}) ={10,11,13,15}
0-9 1-4-9-14
i 2-5-6-8-15
97 5/3/2018
Tracing Example(23)

IF
a-z
f ID
2 3 a-z
6 7  8
i  4 5
0-9 
1  
NUM
 0-9 0-9
14
any
15 9 10 11 12  13
character
error

DFA States = {1-4-9-14}
move(1-4-9-14, other) = {15}
10-11-13-15 a-h 5-6-8-15
j-z -closure({15}) = {15}
0-9 1-4-9-14
i 2-5-6-8-15
98 other 5/3/2018
15
Tracing Example(24)
a-z
2
f
3
IF
a-z  
ID
4 5 6 7 8
i 
0-9

1 
 0-9  0-9
NUM
14
any
character 15 9 10 11 12  13
error
DFA states = {1-4-9-14} 
10-11-13-15 a-h 5-6-8-15 The analysis for 1-4-9-14

j-z is complete. We mark it and
0-9 1-4-9-14 pick another state in the DFA
i 2-5-6-8-15 to analysis. (Practice)
99 other 5/3/2018
15
Tracing Example(25)
a-e, g-z, 0-9

ID
2-5-6-8-15 f
IF
i 3-6-7-8
a-z,0-9
ID ID
a-h 5-6-8-15
1-4-9-14 j-z a-z,0-9 6-7-8
0-9 NUM NUM a-z,0-9

0-9
10-11-13-15 11-12-13
other error 0-9

15 See pp. 118 of Aho-Sethi-Ullman
and pp. 29 of Appel.
100 5/3/2018
Tracing Example(26)
Minimize DFA

expression FA FA
Optimized
Deterministic
FA
101 5/3/2018
DFA B
Tracing Example(27) C
F
A G
Transition Table D H
character E
Stat
e oth
0-9 a-e f g-h i j-z er
A D C C C B C E
B G G F G G G -
C G G G G G G -
D H - - - - - -
E - - - - - - -
F G G G G G G -
G G G G G G G -
H H - - - - - -
102 5/3/2018
DFA B
F
A G
Transition Table D H
character E
Stat
e oth
New Transition Table-1
A D C C C B C E
B G G F G G G - character
Sta
te oth
C G G G G G G - 0-9 a-e f g-h i j-z er
D H - - - - - - A D C C C B C E
E - - - - - - - B C C C C C C -
F G G G G G G - C C C C C C C -
G G G G G G G - D D - - - - - -
H H - - - - - - E - - - - - - -
103 5/3/2018
DFA B
F
A G
New Transition Table-1 D H
character New Transition Table-2

Sta
te oth
0-9 a-e f g-h i j-z er character
Sta
A D C C C B C E te oth
B C C C C C C - A D B B B B B E
C C C C C C C - B B B B B B B -
D D - - - - - - D D - - - - - -
E - - - - - - - E - - - - - - -
104 5/3/2018
New Transition Table-2 IF can be handled by look-ahead
Tracing Example(30) programming
character
Sta New DFA f
te oth IF
A D B B B B B E i B
B B B B B B B -
a-z a-z,0-9
D D - - - - - - ID
E - - - - - - -
A
B
DFA 0-9 NUM
F
C
D
A G
0-9
D H
other error
B=C=F=G E
E D=H
105 5/3/2018
隨堂考試(1+)
What is the optimized DFA for 1+?
Chapter 3 End
Any Question?
106 5/3/2018
1*
= 1 . 1+ (Can use this method)
 1
A B
1 {3,4
1  2 1 3  4 Start {1,2} ,2}
1. -closure(1) ={1, 2}
2. -closure(2) ={2}
3. -closure(3) ={3 ,4 ,2} Character
State
4. -closure(4) ={4 ,2} 0 1
A B
1. -closure(1) ={1, 2}  A
B B
3. -closure(3) ={3 ,4 ,2}  B
1 Can Not Optimized, (Merge)

1 For A is Start State, B is Final State!
A B

Chapter 33

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Chapter 33

Enviado por

Direitos autorais:

Formatos disponíveis

Chapter 3

Scanning - Theory and Practice

 What formal notations to use?

source Lexical token to semantic

 A?  comments = ((#|  ) not(#))*

FA = {S, , s0, F, move }

is the start state

Example at next page….

 A data structure can be translated for many REs or FAs

Which path we should select?

 The transition can be done

 A scanner can be turned into a transducer by the

 Consider this example:

 It is easy to build a scanner that can perform general backup.

Buffered Token Token Flag

Input Token Buffered Token Token Flag

Input Token Buffered Token Token Flag

Input Token Buffered Token Token Flag

Input Token Buffered Token Token Flag

Input Token Buffered Token Token Flag

Input Token Buffered Token Token Flag

Input Token Buffered Token Token Flag

Input Token Buffered Token Token Flag

 The main job of a scanner generator

 Example : A Nondeterministic Finite Automaton (NFA)

 NFA : a (1string) Processing Token a

 A Nondeterministic Finite Automaton (NFA):(a|b)*abb

 A deterministic Finite Automaton (NFA): b*abb

 An Example for each step…

49 More Detail operation at next page… 5/3/2018

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

More Detail operation at next page…

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

3. -closure(3) ={3 ,4 ,5 ,7 ,10}

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

4. -closure(4) ={4 ,5 ,7 ,10}

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

This point line not be computed!!

6. -closure(6) ={5 ,6 ,7 ,10}

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

7. -closure(7) ={5, 7 ,10}

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

9. -closure(9) ={9 ,10}

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

10. -closure(10) ={10}

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

Total closures, but…..

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

Delete Sub Set ...

1. -closure(1) ={1, 2, 8} 6. -closure(6) ={5 ,6 ,7 ,10}

Now Closures, No Sub Set ...

 Usually called l-closure or ε-closure

Algorithm for example at upside

while( states or transitions can be added)

Add the transition to M->transitions:

 First Re-Number for simplifying the work flow

More Operation at next page ……

while( states or transitions can be added)

Add the transition to M->transitions: