lestes::lang::cplus::lex::ucn_token_buffer Class Reference

Token buffer. More...

#include <ucn_token_buffer.hh>

Inheritance diagram for lestes::lang::cplus::lex::ucn_token_buffer:

List of all members.

Public Types
typedef buffer_type::size_type	size_type
	Type for size of the buffer.
Public Member Functions
void	add_back (const ptr< ucn_token > &item)
	Adds single item to the end.
void	advance (size_type len)
	Discards items from the beginning.
ptr< ucn_token >	peek_front (void) const
	Returns item at the beginning.
ptr< token_value >	extract_until (ucn stop)
	Extracts value until stop character without interpreting.
ptr< token_value >	extract_ordinary (size_type len)
	Extracts value without interpreting.
ptr< token_value >	extract_simple_ucn (size_type len, bool identifier)
	Extracts identifiers and numbers with ucn.
ptr< token_value >	extract_invalid_ucn (size_type len, bool identifier)
	Extracts identifiers and numbers with bad ucn.
ptr< token_value >	extract_ucn_literal (size_type len)
	Extracts literal with escaped ucn.
ptr< token_value >	extract_bad_literal (size_type len)
	Extracts malformed literal.
size_type	length (void) const
	Returns length of the buffer.
Static Public Member Functions
static ptr< ucn_token_buffer >	create (const ptr< line_control > &a_lines)
	Returns new buffer.
Protected Member Functions
	ucn_token_buffer (const ptr< line_control > &a_lines)
	Creates empty buffer.
virtual void	gc_mark (void)
	Marks the object.
Private Types
typedef list< srp< ucn_token > >	buffer_type
	Type of buffer to hold stored data.
Private Member Functions
	ucn_token_buffer (const ucn_token_buffer &)
	Hides copy constructor.
ucn_token_buffer &	operator= (const ucn_token_buffer &)
	Hides assignment operator.
Private Attributes
srp< buffer_type >	buffer
	Buffer to hold stored data.
srp< line_control >	lines
	Line control to transform locations.

---

Detailed Description

Token buffer.

Represents flexible buffer holding ucn_token items. Items are added mostly one by one at the end and afterwards removed from the beginning in longer runs, forming literals and identifiers.

Definition at line 59 of file ucn_token_buffer.hh.

---

Member Typedef Documentation

typedef list< srp<ucn_token> > lestes::lang::cplus::lex::ucn_token_buffer::buffer_type [private]

Type of buffer to hold stored data.

Definition at line 62 of file ucn_token_buffer.hh.

typedef buffer_type::size_type lestes::lang::cplus::lex::ucn_token_buffer::size_type

Type for size of the buffer.

Author:

TMA

Definition at line 68 of file ucn_token_buffer.hh.

---

Constructor & Destructor Documentation

lestes::lang::cplus::lex::ucn_token_buffer::ucn_token_buffer

(

const ptr< line_control > &

a_lines

)

[protected]

Creates empty buffer.

Creates empty ucn token buffer.

Postcondition:

length() == 0

Precondition:

a_lines != NULL

Parameters:

a_lines

The associated line control.

Definition at line 56 of file ucn_token_buffer.cc.

Referenced by create().

                                                                  :
        buffer(buffer_type::create()),
        lines(checked(a_lines))
{
}

lestes::lang::cplus::lex::ucn_token_buffer::ucn_token_buffer

(

const ucn_token_buffer &

)

[private]

Hides copy constructor.

---

Member Function Documentation

void lestes::lang::cplus::lex::ucn_token_buffer::add_back

(

const ptr< ucn_token > &

item

)

Adds single item to the end.

Adds single item to the end of the buffer.

Parameters:

item

The item to be added.

Definition at line 66 of file ucn_token_buffer.cc.

References buffer.

{
        buffer->push_back(item);
}

void lestes::lang::cplus::lex::ucn_token_buffer::advance

(

ucn_token_buffer::size_type

len

)

Discards items from the beginning.

Discards len items from the beginning. Used for skipping unimportant values.

Precondition:

len <= length()

Parameters:

len

The length of the discarded sequence.

Definition at line 77 of file ucn_token_buffer.cc.

References buffer, lassert, and length().

{
        lassert(len <= length());
        while (len != 0) {
                buffer->pop_front();
                len--;
        }
}

ptr< ucn_token > lestes::lang::cplus::lex::ucn_token_buffer::peek_front

(

void

)

const

Returns item at the beginning.

Returns item currently at the beginning of the buffer.

Precondition:

buffer.size() != 0

Returns:

The item at the beginning of the buffer.

Definition at line 91 of file ucn_token_buffer.cc.

References buffer, lassert, and length().

{
        lassert(length() != 0);
        return buffer->front();
}

ptr< token_value > lestes::lang::cplus::lex::ucn_token_buffer::extract_until

(

ucn

stop

)

Extracts value until stop character without interpreting.

Removes items from the beginning of the buffer until the stop value is encountered. That value is not removed. Returns token value representing the removed items. Performs no scanning for ucn or ucn escape sequences. Used for hchar and qchar sequences.

Precondition:

The stop value is present in the sequence.

Parameters:

stop

The value to stop scanning the sequence.

Returns:

The representation of the items' values.

Definition at line 135 of file ucn_token_buffer.cc.

References buffer, lestes::lang::cplus::lex::token_value::create(), lassert2, length(), and u.

{
        ucn_token_buffer::size_type len = length();
        // reserve space
        ucn_string us(len,0xbeef);
        ucn u;
        
        buffer_type::iterator bit = buffer->begin();
        ucn_string::iterator sit = us.begin();
        for (ucn_token_buffer::size_type i = 0; i < len; i++, ++bit, ++sit) {
                u = (*bit)->value_get();
                if (u == stop) {
                        // erase removed items
                        buffer->erase(buffer->begin(),bit);
                        // return only the valid part of the string
                        return token_value::create(ucn_string(us.begin(),sit));
                }
                *sit = u;
        }

        lassert2(false,"The stop value was not found");
        return NULL; 
}

ptr< token_value > lestes::lang::cplus::lex::ucn_token_buffer::extract_ordinary

(

ucn_token_buffer::size_type

len

)

Extracts value without interpreting.

Removes len items from the beginning of the buffer. Returns token value representing the removed items. Performs no scanning for ucn or ucn escape sequences. Used for identifiers with no ucn in any form and literals without escaped ucn.

Precondition:

len <= length()

Parameters:

len

The length of the removed sequence.

Returns:

The representation of the items' values.

Definition at line 106 of file ucn_token_buffer.cc.

References buffer, lestes::lang::cplus::lex::token_value::create(), lassert, and length().

{
        lassert(len <= length());
        
        // reserve space
        ucn_string us(len,0xbeef);
        
        buffer_type::iterator bit = buffer->begin();
        ucn_string::iterator sit = us.begin();
        for (ucn_token_buffer::size_type i = 0; i < len; i++, ++bit, ++sit) {
                *sit = (*bit)->value_get();
        }
        
        // erase removed items
        buffer->erase(buffer->begin(),bit);

        return token_value::create(us);
}

ptr< token_value > lestes::lang::cplus::lex::ucn_token_buffer::extract_simple_ucn	(	ucn_token_buffer::size_type	len,
		bool	identifier
	)

Extracts identifiers and numbers with ucn.

Removes len items from the beginning of the buffer. Returns token value representing the removed items. Performs scanning for ucn escape sequences, issues errors for ucn invalid in identifiers. Useful for identifiers and preprocessing numbers containing escaped or translated ucn characters.

Precondition:

len <= length()

Only contains backslash in ucn escape sequences.

The contained ucn escape sequences are well-formed.

Parameters:

	len	The length of the removed sequence.
	identifier	Flag set to true if extracting identifier.

Returns:

The representation of the items' values, with converted ucn escape sequences.

Definition at line 291 of file ucn_token_buffer.cc.

References BEGIN, buffer, lestes::lang::cplus::lex::token_value::create(), lassert, length(), lines, lestes::report, lestes::lang::cplus::lex::ucn_token::TOK_BASIC, lestes::lang::cplus::lex::ucn_token::TOK_TRANSLATED, u, lestes::lang::cplus::lex::ucn_escape_value_invalid, and lestes::lang::cplus::lex::ucn_escape_value_invalid_in_identifier.

{
        lassert(len <= length());
        
        // state of the function
        enum {
                BEGIN,
                BACK,
                UCN
        } fstate = BEGIN;

        ulint count = 0xbad, value = 0xbad;
        ptr<ucn_token> t;
        ptr<simple_location> loc;
        ucn_token_type utt;
        ucn u;
        
        // reserve space, the representation can only be shorter
        ucn_string us(len,0xbeef);
        
        buffer_type::iterator bit = buffer->begin();
        ucn_string::iterator sit = us.begin();
        for (ucn_token_buffer::size_type i = 0; i < len; i++, ++bit) {
                t = *bit;
                loc = t->location_get();
                utt = t->type_get();
                u = t->value_get();

                switch (fstate) {
                        case BEGIN:
                                if (utt == ucn_token::TOK_BASIC) {
                                        if (u == character::ascii_backslash) {
                                                fstate = BACK;
                                                break;
                                        }
                                } else {
                                        lassert(utt == ucn_token::TOK_TRANSLATED);
                                        if (identifier && !character::is_translated_identifier(u)) {
                                                // value out of range for identifier
                                                report << ucn_escape_value_invalid_in_identifier << lines->translate_location(loc);
                                        }
                                }
                                *sit = u;
                                ++sit;
                                break;
                        case BACK:
                                // expecting only ucn escape sequences
                                lassert(utt == ucn_token::TOK_BASIC && 
                                                (u == character::ascii_lower_u || u == character::ascii_upper_u));
                                count = u == character::ascii_lower_u ? 4 : 8;
                                value = 0;
                                fstate = UCN;
                                break;
                        case UCN:
                                // expecting only hexa digits in the ucn escape sequence
                                lassert(utt == ucn_token::TOK_BASIC && character::is_xdigit(u));
                                value = (value << 4) | (character::extract_xdigit(u) & 0x0f);
                                if (--count == 0) {
                                        u = character::create_internal(value);
                                  
                                        if (!character::is_translated(u)) {
                                                // ucn escape sequence value out of range
                                                report << ucn_escape_value_invalid << lines->translate_location(loc);
                                                // substitute for harmless
                                                u = character::ascii_underscore;
                                        } else if (identifier && !character::is_translated_identifier(u)) {
                                                // value out of range for identifier
                                                report << ucn_escape_value_invalid_in_identifier << lines->translate_location(loc);
                                                // keep the character, even if out of range
                                        }
                                        *sit = u;
                                        ++sit;
                                        fstate = BEGIN;
                                }
                                break;
                }
        }

        // shall not be partial ucn escape sequence
        lassert(fstate == BEGIN);
        
        // erase removed items
        buffer->erase(buffer->begin(),bit);

        // use only filled part of the string
        return token_value::create(ucn_string(us.begin(),sit));
}

ptr< token_value > lestes::lang::cplus::lex::ucn_token_buffer::extract_invalid_ucn	(	ucn_token_buffer::size_type	len,
		bool	identifier
	)

Extracts identifiers and numbers with bad ucn.

Removes len items from the beginning of the buffer. Returns token value representing the removed items. Performs scanning for ucn escape sequences, issues errors for invalid ucn and ucn invalid in identifiers. Used for identifiers and preprocessing numbers containing translated or (possibly unterminated) escaped ucn characters.

Precondition:

len <= length()

Only contains backslash in ucn escape sequences.

Backslash is always followed by `U' or `u'.

Parameters:

	len	The length of the removed sequence.
	identifier	Flag set to true if extracting identifier.

Returns:

The representation of the items' values, with converted ucn escape sequences.

Definition at line 173 of file ucn_token_buffer.cc.

References BEGIN, buffer, lestes::lang::cplus::lex::token_value::create(), lassert, length(), lines, lestes::report, lestes::lang::cplus::lex::ucn_token::TOK_BASIC, lestes::lang::cplus::lex::ucn_token::TOK_TRANSLATED, u, lestes::lang::cplus::lex::ucn_escape_insufficient_digits, lestes::lang::cplus::lex::ucn_escape_value_invalid, and lestes::lang::cplus::lex::ucn_escape_value_invalid_in_identifier.

{
        lassert(len <= length());
        
        // state of the function
        enum {
                BEGIN,
                BACK,
                UCN
        } fstate = BEGIN;

        ulint count = 0xbad, value = 0xbad;
        ptr<ucn_token> t;
        ptr<simple_location> loc;
        ucn_token_type utt;
        ucn u;
        
        // reserve space, the representation can only be shorter
        ucn_string us(len,0xbeef);
        
        buffer_type::iterator bit = buffer->begin();
        ucn_string::iterator sit = us.begin();
        for (ucn_token_buffer::size_type i = 0; i < len; i++, ++bit) {
                t = *bit;
                utt = t->type_get();
                u = t->value_get();
                loc = t->location_get();
                switch (fstate) {
                        case BACK:
                                // expecting only ucn escape sequences
                                lassert(utt == ucn_token::TOK_BASIC && 
                                                (u == character::ascii_lower_u || u == character::ascii_upper_u));
                                count = u == character::ascii_lower_u ? 4 : 8;
                                value = 0;
                                fstate = UCN;
                        break;
                        case UCN:
                                if (utt == ucn_token::TOK_BASIC && character::is_xdigit(u)) {
                                        value = (value << 4) | (character::extract_xdigit(u) & 0x0f);
                                        if (--count == 0) {
                                                u = character::create_internal(value);
                                          
                                                if (!character::is_translated(u)) {
                                                        // ucn escape sequence value out of range
                                                        report << ucn_escape_value_invalid << lines->translate_location(loc);
                                                        // substitute for harmless
                                                        u = character::ascii_underscore;
                                                } else if (identifier && !character::is_translated_identifier(u)) {
                                                        // value out of range for identifier
                                                        report << ucn_escape_value_invalid_in_identifier << lines->translate_location(loc);
                                                        // keep the character, even if out of range
                                                }

                                                *sit = u;
                                                ++sit;
                                                fstate = BEGIN;
                                        }
                                        break;
                                } 
                                
                                // malformed ucn escape sequence
                                report << ucn_escape_insufficient_digits << lines->translate_location(loc);
                                
                                // substitute the character and terminate the sequence
                                *sit = character::ascii_underscore;
                                ++sit;
                                fstate = BEGIN;

                                // fall through
                        case BEGIN:
                                if (utt == ucn_token::TOK_BASIC) {
                                        if (u == character::ascii_backslash) {
                                                fstate = BACK;
                                                break;
                                        }
                                } else {
                                        lassert(utt == ucn_token::TOK_TRANSLATED);
                                        if (identifier && !character::is_translated_identifier(u)) {
                                                // value out of range for identifier
                                                report << ucn_escape_value_invalid_in_identifier << lines->translate_location(loc);
                                        }
                                }
                                *sit = u;
                                ++sit;
                                break;
                }
        }

        // shall not end with backslash
        lassert(fstate != BACK);

        if (fstate == UCN) {
                // malformed ucn escape sequence
                report << ucn_escape_insufficient_digits << lines->translate_location(loc);
                // substitute the character and terminate the sequence
                *sit = character::ascii_underscore;
                ++sit;
        }
        
        // erase removed items
        buffer->erase(buffer->begin(),bit);

        // use only filled part of the string
        return token_value::create(ucn_string(us.begin(),sit));
}

ptr< token_value > lestes::lang::cplus::lex::ucn_token_buffer::extract_ucn_literal

(

ucn_token_buffer::size_type

len

)

Extracts literal with escaped ucn.

Removes len items from the beginning of the buffer. Returns token value representing the removed items. Performs scanning for ucn escape sequences, issues errors for invalid ucn. Useful for well-formed string and character literals.

Precondition:

len <= length()

Only contains well-formed escape sequences.

The contained ucn escape sequences are well-formed.

Parameters:

len

The length of the removed sequence.

Returns:

The representation of the items' values, with converted ucn escape sequences.

Definition at line 390 of file ucn_token_buffer.cc.

References BEGIN, buffer, lestes::lang::cplus::lex::token_value::create(), lassert, length(), lines, lestes::report, lestes::lang::cplus::lex::ucn_token::TOK_BASIC, u, and lestes::lang::cplus::lex::ucn_escape_value_invalid.

{
        lassert(len <= length());
        
        // state of the function
        enum {
                BEGIN,
                BACK,
                UCN
        } fstate = BEGIN;
        ulint count = 0xbad, value = 0xbad;
        ptr<ucn_token> t;
        ucn_token_type utt;
        ptr<simple_location> loc;
        ucn u;
        
        // reserve space, the representation can only be shorter
        ucn_string us(len,0xbeef);
        
        buffer_type::iterator bit = buffer->begin();
        ucn_string::iterator sit = us.begin();
        for (ucn_token_buffer::size_type i = 0; i < len; i++, ++bit) {
                 t = *bit;
                 loc = t->location_get();
                 utt = t->type_get();
                 u = t->value_get();
                 
                 switch (fstate) {
                         case BEGIN:
                                 if (utt == ucn_token::TOK_BASIC && u == character::ascii_backslash) {
                                         fstate = BACK;
                                 } else {
                                        *sit = u;
                                        ++sit;
                                 }
                                 break;
                         case BACK:
                                 // expecting only basic characters after backslash
                                 lassert(utt == ucn_token::TOK_BASIC);
                                 if (u == character::ascii_lower_u || u == character::ascii_upper_u) {
                                        count = u == character::ascii_lower_u ? 4 : 8;
                                        value = 0;
                                        fstate = UCN;
                                 } else fstate = BEGIN;
                         break;
                         case UCN:
                                 // expecting only hexa digits in the ucn escape sequence
                                 lassert(utt == ucn_token::TOK_BASIC && character::is_xdigit(u));
                                 value = (value << 4) | (character::extract_xdigit(u) & 0x0f);
                                 if (--count == 0) {
                                         u = character::create_internal(value);
                                        
                                         if (!character::is_translated(u)) {
                                                 // ucn escape sequence value out of range
                                                 report << ucn_escape_value_invalid << lines->translate_location(loc);
                                                 // substitute for harmless
                                                 u = character::ascii_underscore;
                                         }
                                         
                                         *sit = u;
                                         ++sit;
                                         fstate = BEGIN;
                                 }
                                 break;
                 }
        }

        // shall not be partial escape sequence
        lassert(fstate == BEGIN);
        
        // erase removed items
        buffer->erase(buffer->begin(),bit);

        // use only filled part of the string
        return token_value::create(ucn_string(us.begin(),sit));
}

ptr< token_value > lestes::lang::cplus::lex::ucn_token_buffer::extract_bad_literal

(

ucn_token_buffer::size_type

len

)

Extracts malformed literal.

Removes len items from the beginning of the buffer. Returns token value representing the removed items. Performs scanning for escape sequences, issues errors for invalid. Used for character and string literals with ill-formed escape sequences.

Precondition:

len <= length()

The sequence does not end with backslash.

Parameters:

len

The length of the removed sequence.

Returns:

The representation of the items' values, with converted ucn escape sequences.

Definition at line 477 of file ucn_token_buffer.cc.

References BEGIN, buffer, lestes::lang::cplus::lex::token_value::create(), lestes::lang::cplus::lex::invalid_escape_sequence, lassert, lassert2, length(), lines, lestes::lang::cplus::lex::missing_hexadecimal_digits, lestes::report, lestes::lang::cplus::lex::ucn_token::TOK_BASIC, u, lestes::lang::cplus::lex::ucn_escape_insufficient_digits, and lestes::lang::cplus::lex::ucn_escape_value_invalid.

{
        lassert(len <= length());
        
        // state of the function
        enum {
                BEGIN,
                PASS,
                BACK,
                UCN,
                OCT,
                HEX
        } fstate = BEGIN;
        
        ulint count = 0xbad, value = 0xbad;
        ptr<ucn_token> t;
        ptr<simple_location> loc;
        ucn_token_type utt = 0xbad;
        ucn u = 0xbad;
        
        // reserve space
        ucn_string us(len,0xbeef);
        
        buffer_type::iterator bit = buffer->begin();
        ucn_string::iterator sit = us.begin();
        ucn_token_buffer::size_type i = 0;
        while (true) {

                if (fstate == PASS) {
                        fstate = BEGIN;
                } else if (i < len) {
                        t = *bit;
                        ++bit;
                        ++i;
                        loc = t->location_get();
                        utt = t->type_get();
                        u = t->value_get();
                } else break;

                switch (fstate) {
                        case BEGIN:
                                if (utt == ucn_token::TOK_BASIC && u == character::ascii_backslash) {
                                        fstate = BACK;
                                } else {
                                  *sit = u;
                                  ++sit;
                                }
                                break;
                        case BACK:
                                if (utt == ucn_token::TOK_BASIC) {
                                        switch (u) {
                                                case character::ascii_lower_u:
                                                        count = 4;
                                                        fstate = UCN;
                                                        break;
                                                case character::ascii_upper_u:
                                                        count = 8;
                                                        fstate = UCN;
                                                        break;
                                                case character::ascii_lower_x:
                                                        count = 1;
                                                        fstate = HEX;
                                                        break;
                                                case character::ascii_quote:
                                                case character::ascii_dquote:
                                                case character::ascii_qmark:
                                                case character::ascii_backslash:
                                                case character::ascii_lower_a:
                                                case character::ascii_lower_b:
                                                case character::ascii_lower_f:
                                                case character::ascii_lower_n:
                                                case character::ascii_lower_r:
                                                case character::ascii_lower_t:
                                                case character::ascii_lower_v:
                                                        *sit = character::ascii_backslash;
                                                        ++sit;
                                                        *sit = u;
                                                        ++sit;
                                                        fstate = BEGIN;
                                                        break;
                                                default:
                                                  if (character::is_odigit(u)) {
                                                          *sit = character::ascii_backslash;
                                                          ++sit;
                                                          *sit = u;
                                                          ++sit;
                                                          count = 2;
                                                          fstate = OCT;
                                                  } else {
                                                          // unknown escape sequence
                                                          report << invalid_escape_sequence << lines->translate_location(loc);
                                                          *sit = character::ascii_underscore;
                                                          ++sit;
                                                          fstate = BEGIN;
                                                  }
                                        }
                                } else {
                                        // bad character after backslash
                                        report << invalid_escape_sequence << lines->translate_location(loc);
                                        *sit = character::ascii_underscore;
                                        ++sit;
                                        fstate = BEGIN;
                                }
                                break;
                        case UCN:
                                // expecting only hexa digits in the ucn escape sequence
                                if (utt != ucn_token::TOK_BASIC || !character::is_xdigit(u)) {
                                        // bad ucn escape sequence
                                        report << ucn_escape_insufficient_digits << lines->translate_location(loc);
                                        *sit = character::ascii_underscore;
                                        ++sit;
                                        // keep the character for parsing
                                        fstate = PASS;
                                        break;
                                }

                                value = (value << 4) | (character::extract_xdigit(u) & 0x0f);
                                if (--count == 0) {
                                        u = character::create_internal(value);
                                        if (!character::is_translated(u)) {
                                                // disallowed ucn range
                                                report << ucn_escape_value_invalid << lines->translate_location(loc);
                                                *sit = character::ascii_underscore;
                                                ++sit;
                                        } else {
                                                *sit = u;
                                                ++sit;
                                        }
                                        fstate = BEGIN;
                                }
                                break;
                        case OCT:
                                if (utt == ucn_token::TOK_BASIC && character::is_odigit(u)) {
                                        *sit = u;
                                        ++sit;
                                        if (--count == 0) fstate = BEGIN;
                                } else fstate = PASS;
                                break;
                        case HEX:
                                if (utt == ucn_token::TOK_BASIC && character::is_xdigit(u)) {
                                        if (count) {
                                                count = 0;
                                                *sit = character::ascii_backslash;
                                                ++sit;
                                                *sit = character::ascii_lower_x;
                                                ++sit;
                                        }
                                        *sit = u;
                                        ++sit;
                                } else {
                                        if (count) {
                                                // \x with no xdigits
                                                report << missing_hexadecimal_digits << lines->translate_location(loc);
                                                *sit = character::ascii_underscore;
                                                ++sit;
                                        }
                                        // keep the character for parsing
                                        fstate = PASS;
                                }
                                break;
                        default:
                                lassert2(false,"You should never get here");
                }
        }

        switch (fstate) {
                case BEGIN:
                case OCT:
                        break;
                case HEX:
                        if (count) {
                                // \x with no xdigits
                                report << missing_hexadecimal_digits << lines->translate_location(loc);
                                *sit = character::ascii_underscore;
                                ++sit;
                        }
                        break;
                case UCN:
                        // unterminated \uU
                        report << ucn_escape_insufficient_digits << lines->translate_location(loc);
                        *sit = character::ascii_underscore;
                        ++sit;
                        break;
                default:
                        lassert2(false,"You should never get here");
        }
        
        // erase removed items
        buffer->erase(buffer->begin(),bit);

        // use only filled part of the string
        return token_value::create(ucn_string(us.begin(),sit));
}

ucn_token_buffer::size_type lestes::lang::cplus::lex::ucn_token_buffer::length

(

void

)

const

Returns length of the buffer.

Returns length of the buffer.

Returns:

The current length of the buffer.

Definition at line 675 of file ucn_token_buffer.cc.

References buffer.

Referenced by advance(), extract_bad_literal(), extract_invalid_ucn(), extract_ordinary(), extract_simple_ucn(), extract_ucn_literal(), extract_until(), and peek_front().

{
        return buffer->size();
}

ptr< ucn_token_buffer > lestes::lang::cplus::lex::ucn_token_buffer::create

(

const ptr< line_control > &

a_lines

)

[static]

Returns new buffer.

Retruns new empty ucn token buffer.

Postcondition:

length() == 0

Parameters:

a_lines

The associated line control.

Returns:

The new ucn token buffer.

Definition at line 695 of file ucn_token_buffer.cc.

References ucn_token_buffer().

{
        return new ucn_token_buffer(a_lines);
}

void lestes::lang::cplus::lex::ucn_token_buffer::gc_mark

(

void

)

[protected, virtual]

Marks the object.

Marks the object.

Reimplemented from lestes::std::mem::keystone.

Definition at line 683 of file ucn_token_buffer.cc.

References buffer, and lines.

{
        buffer.gc_mark();
        lines.gc_mark();
}

ucn_token_buffer& lestes::lang::cplus::lex::ucn_token_buffer::operator=

(

const ucn_token_buffer &

)

[private]

Hides assignment operator.

---

Member Data Documentation

srp<buffer_type> lestes::lang::cplus::lex::ucn_token_buffer::buffer [private]

Buffer to hold stored data.

Definition at line 98 of file ucn_token_buffer.hh.

Referenced by add_back(), advance(), extract_bad_literal(), extract_invalid_ucn(), extract_ordinary(), extract_simple_ucn(), extract_ucn_literal(), extract_until(), gc_mark(), length(), and peek_front().

srp<line_control> lestes::lang::cplus::lex::ucn_token_buffer::lines [private]

Line control to transform locations.

Definition at line 100 of file ucn_token_buffer.hh.

Referenced by extract_bad_literal(), extract_invalid_ucn(), extract_simple_ucn(), extract_ucn_literal(), and gc_mark().

---

The documentation for this class was generated from the following files:

• ucn_token_buffer.hh
• ucn_token_buffer.cc

---