lestes::lang::cplus::lex::token_sequence

lestes::lang::cplus::lex::token_sequence Class Reference

#include <token_sequence.hh>

Inheritance diagram for lestes::lang::cplus::lex::token_sequence:

lestes::lang::cplus::lex::token_input lestes::lang::cplus::lex::pp_filter lestes::std::object lestes::std::mem::keystone List of all members.

Public Types

typedef sequence_type::size_type size_type
 The type for size of the sequence.

Public Member Functions

ptr< pp_token > read (void)
 Reads first token.
ptr< pp_token > peek_front (void)
 Returns front token.
ptr< pp_token > peek_back (void)
 Returns back token.
ptr< pp_token > read_front (void)
 Reads front token, squeezing whitespace.
ptr< pp_token > read_front_skip_ws (void)
 Reads front token, skipping front whitespace, but not newline.
ptr< pp_token > read_back (void)
 Reads back token, squeezing whitespace.
ptr< pp_token > read_back_skip_ws (void)
 Reads back token, skipping back whitespace, but not newline.
bool skip_front_ws (void)
 skips back whitespace
bool skip_back_ws (void)
 skips back whitespace
void add_back (const ptr< pp_token > &t)
 Adds token to back.
void prepend (const ptr< token_sequence > &ts)
 Prepends another sequence.
void append (const ptr< token_sequence > &inserted)
 Appends another sequence.
void taboo_extend (const ptr< taboo_macros > &tm, const ptr< macro_storage > &macros)
 Extends taboo set for tokens in sequence.
ptr< token_sequence > expand_all (const ptr< macro_storage > &macros)
 completely expands all macros within sequence
ptr< token_sequence > clone (void) const
 Clones the sequence.
ptr< token_sequence > clone (const ptr< source_location > &a_location) const
 clones the sequence with new location.
size_type length (void) const
 Returns length of the sequence.
bool equals (const ptr< token_sequence > &other) const
 Tests equality.
bool congruent (const ptr< token_sequence > &other) const
 Tests congruence.
void debug_print (::std::ostream &o) const
 TODO prints debug dump.

Static Public Member Functions

static ptr< token_sequence > create (void)
 Returns empty sequence.

Protected Member Functions

 token_sequence (void)
 Creates empty sequence.
virtual void gc_mark (void)
 Marks the object.

Private Types

typedef ::lestes::std::list<
srp< pp_token > > 		sequence_type
 The type of sequence container.

Private Attributes

srp< sequence_type > sequence
 The sequence container.

Friends

class token_stream
 For direct acces to sequence in prepend().

Detailed Description

Represents sequence of tokens, with access at both ends.

Definition at line 61 of file token_sequence.hh.

Member Typedef Documentation

typedef ::lestes::std::list< srp<pp_token> > lestes::lang::cplus::lex::token_sequence::sequence_type [private]

The type of sequence container.

Definition at line 64 of file token_sequence.hh.

typedef sequence_type::size_type lestes::lang::cplus::lex::token_sequence::size_type

The type for size of the sequence.

Author:
TMA

Definition at line 70 of file token_sequence.hh.

Constructor & Destructor Documentation

lestes::lang::cplus::lex::token_sequence::token_sequence ( void   )  [protected]

Creates empty sequence.

Constructs new empty sequence.

Postcondition:
length() == 0

Definition at line 59 of file token_sequence.cc.

Referenced by create().

00059                                   :
00060         sequence(sequence_type::create())
00061 {
00062 }

Member Function Documentation

ptr< pp_token > lestes::lang::cplus::lex::token_sequence::read ( void   )  [virtual]

Reads first token.

Reads front token, does no transformation.

Returns:
The first token in the sequence.

Implements lestes::lang::cplus::lex::pp_filter.

Definition at line 68 of file token_sequence.cc.

References length(), sequence, and lestes::lang::cplus::lex::pp_token::terminator().

00069 {
00070         if (length() == 0) return pp_token::terminator();
00071         ptr<pp_token> t = sequence->front();
00072         sequence->pop_front();
00073         return t;
00074 }

ptr< pp_token > lestes::lang::cplus::lex::token_sequence::peek_front ( void   )  [virtual]

Returns front token.

Returns front token, which is kept in the sequence.

Returns:
The first token in the sequence, or TOK_TERMINATOR.

Implements lestes::lang::cplus::lex::token_input.

Definition at line 80 of file token_sequence.cc.

References length(), sequence, and lestes::lang::cplus::lex::pp_token::terminator().

Referenced by expand_all().

00081 {
00082         if (length() != 0) return sequence->front();
00083         return pp_token::terminator();
00084 }

ptr< pp_token > lestes::lang::cplus::lex::token_sequence::peek_back ( void   ) 

Returns back token.

Returns back token, which is kept in the sequence.

Returns:
The last token in the sequence, or TOK_TERMINATOR.

Definition at line 90 of file token_sequence.cc.

References length(), sequence, and lestes::lang::cplus::lex::pp_token::terminator().

00091 {
00092         if (length() == 0) return pp_token::terminator();
00093         return sequence->back();
00094 }

ptr< pp_token > lestes::lang::cplus::lex::token_sequence::read_front ( void   )  [virtual]

Reads front token, squeezing whitespace.

Reads front token, squeezing multiple blanks into one.

Returns:
The first token in the sequence.

Implements lestes::lang::cplus::lex::token_input.

Definition at line 100 of file token_sequence.cc.

References length(), sequence, lestes::lang::cplus::lex::pp_token::terminator(), lestes::lang::cplus::lex::pp_token::TOK_BLANK, and u.

00101 {
00102         ulint len = length();
00103         if (!len) return pp_token::terminator();
00104         
00105         ptr<pp_token> t = sequence->front();
00106         sequence->pop_front();
00107         len--;
00108         
00109         if (t->type_get() == pp_token::TOK_BLANK) {
00110                 ptr<pp_token> u;
00111                 while (len--) {
00112                         u = sequence->front();
00113                         if (u->type_get() != pp_token::TOK_BLANK) break;
00114                         sequence->pop_front();
00115                 }
00116         }
00117         
00118         return t;
00119 }

ptr< pp_token > lestes::lang::cplus::lex::token_sequence::read_front_skip_ws ( void   )  [virtual]

Reads front token, skipping front whitespace, but not newline.

Reads front token, skipping front whitespace, but not newline.

Returns:
The first non-whitespace token in the sequence, or TOK_TERMINATOR.

Implements lestes::lang::cplus::lex::token_input.

Definition at line 125 of file token_sequence.cc.

References length(), sequence, lestes::lang::cplus::lex::pp_token::terminator(), and lestes::lang::cplus::lex::pp_token::TOK_BLANK.

00126 {
00127         ptr<pp_token> t;
00128         ulint len = length();
00129         
00130         while (len--) {
00131                 t = sequence->front();
00132                 sequence->pop_front();
00133                 if (t->type_get() != pp_token::TOK_BLANK)
00134                         return t;
00135         }
00136         
00137         return pp_token::terminator();
00138 }

ptr< pp_token > lestes::lang::cplus::lex::token_sequence::read_back ( void   ) 

Reads back token, squeezing whitespace.

Reads back token, squeezing blanks.

Returns:
The last token in the sequence, or TOK_TERMINATOR.

Definition at line 144 of file token_sequence.cc.

References length(), sequence, lestes::lang::cplus::lex::pp_token::terminator(), lestes::lang::cplus::lex::pp_token::TOK_BLANK, and u.

00145 {
00146         ulint len = length();
00147         
00148         if (!len) return pp_token::terminator();
00149         
00150         ptr<pp_token> t = sequence->back();
00151         sequence->pop_back();
00152         len--;
00153         
00154         if (t->type_get() == pp_token::TOK_BLANK) {
00155                 ptr<pp_token> u;
00156                 while (len--) {
00157                         u = sequence->back();
00158                         if (u->type_get() != pp_token::TOK_BLANK) break;
00159                         sequence->pop_back();
00160                 }
00161         }
00162         
00163         return t;
00164 }

ptr< pp_token > lestes::lang::cplus::lex::token_sequence::read_back_skip_ws ( void   ) 

Reads back token, skipping back whitespace, but not newline.

Reads back token, skipping back whitespace.

Precondition:
The sequence contains non-whitespace tokens.
Returns:
The last non-whitespace token in the sequence.

Definition at line 171 of file token_sequence.cc.

References length(), sequence, lestes::lang::cplus::lex::pp_token::terminator(), and lestes::lang::cplus::lex::pp_token::TOK_BLANK.

00172 {
00173         ptr<pp_token> t;
00174         ulint len = length();
00175         
00176         while (len--) {
00177                 t = sequence->back();
00178                 sequence->pop_back();
00179                 if (t->type_get() != pp_token::TOK_BLANK)
00180                         return t;
00181         }
00182         
00183         return pp_token::terminator();
00184 }

bool lestes::lang::cplus::lex::token_sequence::skip_front_ws ( void   )  [virtual]

skips back whitespace

Skips front whitespace.

Returns:
true If there are some tokens left in the sequence.

Implements lestes::lang::cplus::lex::token_input.

Definition at line 190 of file token_sequence.cc.

References sequence, and lestes::lang::cplus::lex::pp_token::TOK_BLANK.

00191 {
00192         sequence_type::size_type len = sequence->size();
00193         while (len && sequence->front()->type_get() == pp_token::TOK_BLANK) {
00194                 sequence->pop_front();
00195                 len--;
00196         }
00197         return len;
00198 }

bool lestes::lang::cplus::lex::token_sequence::skip_back_ws ( void   ) 

skips back whitespace

Skips back whitespace.

Returns:
true If there are some tokens left in the sequence.

Definition at line 204 of file token_sequence.cc.

References sequence, and lestes::lang::cplus::lex::pp_token::TOK_BLANK.

00205 {
00206         sequence_type::size_type len = sequence->size();
00207         while (len && sequence->back()->type_get() == pp_token::TOK_BLANK) {
00208                 sequence->pop_back();
00209                 len--;
00210         }
00211         return len;
00212 }

void lestes::lang::cplus::lex::token_sequence::add_back ( const ptr< pp_token > &  a_token  ) 

Adds token to back.

Adds token to back of the sequence.

Precondition:
a_token != NULL
Parameters:
a_token The token to add.

Definition at line 219 of file token_sequence.cc.

References lassert, and sequence.

00220 {
00221         lassert(a_token);
00222         sequence->push_back(a_token);
00223 }

void lestes::lang::cplus::lex::token_sequence::prepend ( const ptr< token_sequence > &  inserted  ) 

Prepends another sequence.

Inserts another token sequence before the beginning of the sequence.

Precondition:
inserted != NULL
Postcondition:
inserted->length() == 0
Parameters:
inserted The sequence to insert, the content is moved into new place.

Definition at line 343 of file token_sequence.cc.

References lassert, and sequence.

Referenced by expand_all().

00344 {
00345         lassert(inserted);
00346         sequence->splice(sequence->begin(),*(inserted->sequence));
00347 }

void lestes::lang::cplus::lex::token_sequence::append ( const ptr< token_sequence > &  inserted  ) 

Appends another sequence.

Inserts another token sequence after the end of the sequence.

Precondition:
inserted != NULL
Postcondition:
inserted->length() == 0
Parameters:
inserted The sequence to insert, the content is moved into new place.

Definition at line 355 of file token_sequence.cc.

References lassert, and sequence.

00356 {
00357         lassert(inserted);
00358         sequence->splice(sequence->end(),*(inserted->sequence));
00359 }

void lestes::lang::cplus::lex::token_sequence::taboo_extend ( const ptr< taboo_macros > &  tm,
const ptr< macro_storage > &  macros 
)

Extends taboo set for tokens in sequence.

Extends taboo set for possibly expandable tokens in sequence by a macro.

Precondition:
a_macro != NULL

macros != NULL

Parameters:
a_macro The macro to add to the taboo set.
macros The defined macros.

Definition at line 280 of file token_sequence.cc.

References lassert, and sequence.

00281 {
00282         lassert(tm);
00283         lassert(macros);
00284 
00285         ptr<pp_token> t;
00286         
00287         for (sequence_type::iterator it = sequence->begin(),
00288                   end = sequence->end(); it != end; ++it) {
00289                 t = *it;
00290                 
00291                 // extend only currently defined macros
00292                 if (t->is_name() &&
00293                          macros->lookup(t->value_get())) {
00294                         // extend taboo for token by mac
00295                         
00296                         *it = t->clone(tm);
00297                 }
00298         }
00299 }

ptr< token_sequence > lestes::lang::cplus::lex::token_sequence::expand_all ( const ptr< macro_storage > &  macros  ) 

completely expands all macros within sequence

Expands macros within sequence. The sequence is discarded.

Precondition:
macros != NULL
Postcondition:
length() == 0
Parameters:
macros The macros for expansion.
Returns:
The expanded tokens in a new sequence.

Definition at line 232 of file token_sequence.cc.

References create(), lassert, length(), peek_front(), prepend(), sequence, and lestes::lang::cplus::lex::pp_token::TOK_LEFT_PAR.

00233 {
00234         lassert(macros);
00235 
00236         ptr<token_sequence> result = token_sequence::create();
00237 
00238         if (!length()) return result;
00239         
00240         ptr<pp_token> t = sequence->front();
00241         pp_token_type ptt = t->type_get();
00242 
00243         ptr<macro> mac;
00244         ptr<token_sequence> expanded;
00245         
00246         while (length()) {
00247                 t = sequence->front();
00248                 sequence->pop_front();
00249                 
00250                 ptt = t->type_get();
00251                 
00252                 // check whether the token is defined as macro
00253                 // whether the macro is allowed in the context of the token
00254                 // and whether a funlike macro is called with '('
00255                 if (t->is_name() &&
00256                          (mac = macros->lookup(t->value_get())) &&
00257                          !t->taboo_get()->contains(mac) &&
00258                          (!mac->funlike_get() || peek_front()->type_get() == pp_token::TOK_LEFT_PAR)) {
00259 
00260                         ptr<token_sequence> expanded = mac->expand(t,this,macros);
00261 
00262                         expanded->taboo_extend(t->taboo_get()->extend(mac),macros);
00263 
00264                         // prepend in O(1) to allow rescanning
00265                         prepend(expanded);
00266                 } else {
00267                         result->add_back(t);
00268                 }
00269         }
00270         return result;
00271 }

ptr< token_sequence > lestes::lang::cplus::lex::token_sequence::clone ( void   )  const

Clones the sequence.

Returns copy of the sequence.

Postcondition:
is_equal(returned,this)
Returns:
New sequence with equal values.

Definition at line 306 of file token_sequence.cc.

References create(), and sequence.

00307 {
00308         ptr<token_sequence> nju = token_sequence::create();
00309         copy(sequence->begin(),sequence->end(),back_inserter(*(nju->sequence)));
00310         return nju;
00311 }

ptr< token_sequence > lestes::lang::cplus::lex::token_sequence::clone ( const ptr< source_location > &  a_location  )  const

clones the sequence with new location.

Returns a copy of the sequence with new location.

Precondition:
a_location != NULL
Parameters:
a_location The location for the copied tokens.
Returns:
The copy of the sequence.

Definition at line 319 of file token_sequence.cc.

References create(), lassert, and sequence.

00320 {
00321         lassert(a_location);
00322         
00323         ptr<pp_token> t;
00324         ptr<token_sequence> nju = token_sequence::create();
00325 
00326         sequence_type::iterator it = sequence->begin();
00327         sequence_type::iterator end = sequence->end();
00328         
00329         for (; it != end; ++it) {
00330                 t = *it;
00331                 nju->add_back(t->clone(a_location));
00332         }
00333 
00334         return nju;
00335 }

token_sequence::size_type lestes::lang::cplus::lex::token_sequence::length ( void   )  const

Returns length of the sequence.

Returns length of the sequence.

Returns:
The length of the sequence.

Definition at line 365 of file token_sequence.cc.

References sequence.

Referenced by congruent(), equals(), expand_all(), peek_back(), peek_front(), read(), read_back(), read_back_skip_ws(), read_front(), and read_front_skip_ws().

00366 {
00367         return sequence->size();
00368 }

bool lestes::lang::cplus::lex::token_sequence::equals ( const ptr< token_sequence > &  other  )  const

Tests equality.

Tests equality to other token sequence.

Parameters:
other The token sequence to compare with.
Returns:
true If both sequences contain equal tokens.

Definition at line 384 of file token_sequence.cc.

References lestes::is_equal(), length(), and sequence.

00385 {
00386         if (!other || length() != other->length()) return false;
00387         
00388         sequence_type::iterator it = sequence->begin();
00389         sequence_type::iterator end = sequence->end();
00390 
00391         sequence_type::iterator ot = other->sequence->begin();
00392         
00393         for (; it != end; ++it, ++ot) {
00394                 ptr<pp_token> pt = *it;
00395                 ptr<pp_token> qt = *ot;
00396                 
00397                 if (!is_equal(pt,qt)) return false;
00398         }
00399         
00400         return true;
00401 }

bool lestes::lang::cplus::lex::token_sequence::congruent ( const ptr< token_sequence > &  other  )  const

Tests congruence.

Tests whether sequences have the same type and spelling of tokens.

Parameters:
other The token sequence to compare with.
Returns:
true If both sequences contain interchangeable tokens.

Definition at line 408 of file token_sequence.cc.

References length(), and sequence.

00409 {
00410         // TODO pt ::std::cerr << "token_sequence::resembles\n";
00411         // TODO pt debug_print( ::std::cerr); ::std::cerr << '\n';
00412         // TODO pt ::std::cerr << other << '\n';
00413         
00414         if (!other || length() != other->length()) return false;
00415         
00416         // TODO pt ::std::cerr << "length ok\n";
00417         
00418         sequence_type::iterator it = sequence->begin();
00419         sequence_type::iterator end = sequence->end();
00420 
00421         sequence_type::iterator ot = other->sequence->begin();
00422         
00423         for (; it != end; ++it, ++ot) {
00424                 ptr<pp_token> pt = *it;
00425                 ptr<pp_token> qt = *ot;
00426                 
00427                 if (!pt->congruent(qt)) return false;
00428                 // TODO pt ::std::cerr << pt->description_get() << " is congruent to " << qt->description_get() << '\n';
00429         }
00430         
00431         return true;
00432 }

void lestes::lang::cplus::lex::token_sequence::debug_print ( ::std::ostream &  o  )  const

TODO prints debug dump.

ptr< token_sequence > lestes::lang::cplus::lex::token_sequence::create ( void   )  [static]

Returns empty sequence.

Returns empty token sequence.

Returns:
Empty token sequence.

Definition at line 374 of file token_sequence.cc.

References token_sequence().

Referenced by clone(), lestes::lang::cplus::lex::macro_body::expand(), expand_all(), lestes::lang::cplus::lex::token_stream::expand_line(), lestes::lang::cplus::lex::macro_body::parse(), lestes::lang::cplus::lex::macro_argument::parse(), lestes::lang::cplus::lex::concat::process(), lestes::lang::cplus::lex::token_stream::read_line(), and lestes::lang::cplus::lex::macro_argument::stringified_get().

00375 {
00376         return new token_sequence();
00377 }

void lestes::lang::cplus::lex::token_sequence::gc_mark ( void   )  [protected, virtual]

Marks the object.

Marks the object.

Reimplemented from lestes::lang::cplus::lex::pp_filter.

Definition at line 437 of file token_sequence.cc.

References lestes::lang::cplus::lex::pp_filter::gc_mark(), and sequence.

00438 {
00439         sequence.gc_mark();
00440         token_input::gc_mark();
00441 }

Friends And Related Function Documentation

friend class token_stream [friend]

For direct acces to sequence in prepend().

Definition at line 121 of file token_sequence.hh.

Member Data Documentation

srp<sequence_type> lestes::lang::cplus::lex::token_sequence::sequence [private]

The sequence container.

Definition at line 123 of file token_sequence.hh.

Referenced by add_back(), append(), clone(), congruent(), equals(), expand_all(), gc_mark(), length(), peek_back(), peek_front(), prepend(), read(), read_back(), read_back_skip_ws(), read_front(), read_front_skip_ws(), skip_back_ws(), skip_front_ws(), and taboo_extend().

The documentation for this class was generated from the following files:
Generated on Mon Feb 12 18:24:19 2007 for lestes by doxygen 1.5.1-20070107