lestes::lang::cplus::lex::pre_lex

lestes::lang::cplus::lex::pre_lex Class Reference

Character level processing. More...

#include <pre_lex.hh>

Inheritance diagram for lestes::lang::cplus::lex::pre_lex:

lestes::std::object lestes::std::mem::keystone List of all members.

Public Member Functions

ptr< ucn_token > read (void)
 Reads single token.

Static Public Member Functions

static ptr< pre_lex > create (const ptr< data_source > &a_src, const ptr< encoder > &a_enc)
 Returns new instance, initializes with data source and encoder.

Protected Member Functions

 pre_lex (const ptr< data_source > &a_src, const ptr< encoder > &a_enc)
 Creates new object, initializes with data source and encoder.
virtual void gc_mark (void)
 Marks the object.

Private Types

enum  state_type {
  START, CONT, NEWLINE, ERROR,
  END
}
 Type of state. More...

Private Member Functions

void connect (void)
 Connect all filters.
 pre_lex (const pre_lex &copy)
 Hides copy constructor.
pre_lex & operator= (const pre_lex &rhs)
 Hides assignment operator.

Private Attributes

state_type state
 State of the object.
srp< ucn_token > saved
 Saved token.
srp< data_source > src
 Data source.
srp< encoder > enc
 Source encoder.
srp< special_tokens > stok
 Special tokens translator.
srp< line_numbers > lnum
 Line numbering.
srp< trigraphs > trig
 Trigraph translator.
srp< line_join > ljoin
 Line joining.

Detailed Description

Character level processing.

Performs operations done prior to lexical analysis.

Definition at line 57 of file pre_lex.hh.

Member Enumeration Documentation

enum lestes::lang::cplus::lex::pre_lex::state_type [private]

Type of state.

Enumerator:
START  Starting state.
CONT  Continuing state.
NEWLINE  Newline was read.
ERROR  Error was returned.
END  EOF was read.

Definition at line 72 of file pre_lex.hh.

00072                      {
00073                 //! Starting state.
00074                 START,
00075                 //! Continuing state.
00076                 CONT,
00077                 //! Newline was read.
00078                 NEWLINE,
00079                 //! Error was returned.
00080                 ERROR,
00081                 //! EOF was read.
00082                 END,
00083         } state_type;

Constructor & Destructor Documentation

lestes::lang::cplus::lex::pre_lex::pre_lex ( const ptr< data_source > &  a_src,
const ptr< encoder > &  a_enc 
) [protected]

Creates new object, initializes with data source and encoder.

Creates the object.

Postcondition:
state == START
Precondition:
a_src != NULL

a_enc != NULL

Parameters:
a_src The source stream of data.
a_enc The encoder to apply on the source.
Returns:
New instance of the class.

Definition at line 55 of file pre_lex.cc.

References connect().

Referenced by create().

00055                                                                         :
00056         state(START),
00057         saved(),
00058         src((lassert(a_src),a_src)),
00059         enc((lassert(a_enc),a_enc)),
00060         stok(special_tokens::create()),
00061         lnum(line_numbers::create()),
00062         trig(trigraphs::create()), 
00063         ljoin(line_join::create())
00064 {
00065         connect();
00066 }

lestes::lang::cplus::lex::pre_lex::pre_lex ( const pre_lex &  copy  )  [private]

Hides copy constructor.

Member Function Documentation

ptr< ucn_token > lestes::lang::cplus::lex::pre_lex::read ( void   ) 

Reads single token.

Returns next token passed through all actions before lex. Inserts fake newline, if there is none at the end of file. Saves the EOF token to be returned forever.

Returns:
Next token with all fields set to proper values.

Definition at line 86 of file pre_lex.cc.

References CONT, lestes::lang::cplus::lex::ucn_token::create(), lestes::lang::cplus::lex::ucn_token::create_error(), END, lestes::msg::eolog, ERROR, lassert2, ljoin, lestes::lang::cplus::lex::missing_newline, NEWLINE, saved, START, state, lestes::lang::cplus::lex::ucn_token::TOK_BASIC, lestes::lang::cplus::lex::ucn_token::TOK_EOF, lestes::lang::cplus::lex::ucn_token::TOK_ERROR, and lestes::lang::cplus::lex::ucn_token::TOK_TRANSLATED.

00087 {
00088         pre_lex_logger << "pre_lex::read()\n" << msg::eolog;
00089         pre_lex_logger << "state = " << state["scn#$"] << '\n' << msg::eolog;
00090 
00091         ptr<ucn_token> t;
00092         
00093         switch (state) {
00094                 case END:
00095                         t = saved;
00096                         break;
00097                 case ERROR:
00098                         state = END;
00099                         // return fake newline
00100                         t = ucn_token::create(ucn_token::TOK_BASIC,character::ascii_new_line,saved->location_get());
00101                         break;
00102                 case START:
00103                 case CONT:
00104                 case NEWLINE:
00105                         t = ljoin->read();
00106                         break;
00107                 default:
00108                         lassert2(false,"You should never get here");
00109         }
00110         
00111         ucn_token_type utt = t->type_get();
00112 
00113         switch (state) {
00114                 case START:
00115                         if (utt == ucn_token::TOK_EOF) {
00116                                 // save the EOF
00117                                 saved = t;
00118                                 state = END;
00119                         } else if (utt == ucn_token::TOK_BASIC && t->value_get() == character::ascii_new_line) {
00120                                 // save token's location
00121                                 saved = t;
00122                                 state = NEWLINE;
00123                         } else {
00124                                 // continuing
00125                                 state = CONT;
00126                                 saved = NULL;
00127                         }
00128                         break;
00129                 case CONT:
00130                         if (utt == ucn_token::TOK_EOF) {
00131                                 // save the EOF
00132                                 saved = t;
00133                                 // report error: missing newline at the end of file
00134                                 t = ucn_token::create_error(missing_newline->format(),t->location_get());
00135                                 state = ERROR;
00136                         } else if (utt == ucn_token::TOK_BASIC && t->value_get() == character::ascii_new_line) {
00137                                 // save token's location
00138                                 saved = t;
00139                                 state = NEWLINE;
00140                         } else {
00141                                 saved = NULL;
00142                         }
00143                         break;
00144                 case NEWLINE:
00145                         switch (utt) {
00146                                 case ucn_token::TOK_EOF:
00147                                         // move the location to that of the newline
00148                                         t = saved = t->clone_location(saved->location_get());
00149                                         state = END;
00150                                         break;
00151                                 case ucn_token::TOK_ERROR:
00152                                         break;
00153                                 case ucn_token::TOK_BASIC:
00154                                         if (t->value_get() == character::ascii_new_line)
00155                                                 break;
00156                                         // fall through
00157                                 case ucn_token::TOK_TRANSLATED:
00158                                         state = CONT;
00159                                         break;
00160                                 default:
00161                                         lassert2(false,"You should never get here");
00162                         }
00163                         break;
00164                 case END:
00165                 case ERROR:
00166                         break;
00167                 default:
00168                         lassert2(false,"You should never get here");
00169         }
00170 
00171         pre_lex_logger << "returning " << static_cast<ulint>(t->type_get()) << '\n' << msg::eolog;
00172         pre_lex_logger << "pre_lex::read() end\n" << msg::eolog;
00173         return t;
00174 }

ptr< pre_lex > lestes::lang::cplus::lex::pre_lex::create ( const ptr< data_source > &  a_src,
const ptr< encoder > &  a_enc 
) [static]

Returns new instance, initializes with data source and encoder.

Returns new instance, initializes with source and encoding.

Parameters:
a_src The source stream of data.
a_enc The encoder to apply on the source.
Returns:
New instance of the class.

Definition at line 197 of file pre_lex.cc.

References pre_lex().

00198 {
00199         return new pre_lex(a_src,a_enc);
00200 }

void lestes::lang::cplus::lex::pre_lex::gc_mark ( void   )  [protected, virtual]

Marks the object.

Marks the object.

Reimplemented from lestes::std::mem::keystone.

Definition at line 179 of file pre_lex.cc.

References enc, lestes::std::mem::keystone::gc_mark(), ljoin, lnum, saved, src, stok, and trig.

00180 {
00181         saved.gc_mark();
00182         src.gc_mark();
00183         enc.gc_mark();
00184         stok.gc_mark();
00185         lnum.gc_mark();
00186         trig.gc_mark();
00187         ljoin.gc_mark();
00188 	::lestes::std::object::gc_mark();
00189 }

void lestes::lang::cplus::lex::pre_lex::connect ( void   )  [private]

Connect all filters.

Connects all filters together.

Definition at line 71 of file pre_lex.cc.

References enc, ljoin, lnum, src, stok, and trig.

Referenced by pre_lex().

00072 {
00073         enc->input_set(src);
00074         stok->input_set(enc);
00075         lnum->input_set(stok);
00076         trig->input_set(lnum);
00077         ljoin->input_set(trig);
00078 }

pre_lex& lestes::lang::cplus::lex::pre_lex::operator= ( const pre_lex &  rhs  )  [private]

Hides assignment operator.

Member Data Documentation

state_type lestes::lang::cplus::lex::pre_lex::state [private]

State of the object.

Definition at line 85 of file pre_lex.hh.

Referenced by read().

srp<ucn_token> lestes::lang::cplus::lex::pre_lex::saved [private]

Saved token.

Definition at line 87 of file pre_lex.hh.

Referenced by gc_mark(), and read().

srp<data_source> lestes::lang::cplus::lex::pre_lex::src [private]

Data source.

Definition at line 89 of file pre_lex.hh.

Referenced by connect(), and gc_mark().

srp<encoder> lestes::lang::cplus::lex::pre_lex::enc [private]

Source encoder.

Definition at line 91 of file pre_lex.hh.

Referenced by connect(), and gc_mark().

srp<special_tokens> lestes::lang::cplus::lex::pre_lex::stok [private]

Special tokens translator.

Definition at line 93 of file pre_lex.hh.

Referenced by connect(), and gc_mark().

srp<line_numbers> lestes::lang::cplus::lex::pre_lex::lnum [private]

Line numbering.

Definition at line 95 of file pre_lex.hh.

Referenced by connect(), and gc_mark().

srp<trigraphs> lestes::lang::cplus::lex::pre_lex::trig [private]

Trigraph translator.

Definition at line 97 of file pre_lex.hh.

Referenced by connect(), and gc_mark().

srp<line_join> lestes::lang::cplus::lex::pre_lex::ljoin [private]

Line joining.

Definition at line 99 of file pre_lex.hh.

Referenced by connect(), gc_mark(), and read().

The documentation for this class was generated from the following files:
Generated on Mon Feb 12 18:24:18 2007 for lestes by doxygen 1.5.1-20070107