ACE_Tokenizer Class Reference

Tokenizer. More...

#include <SString.h>

Collaboration diagram for ACE_Tokenizer:

[legend]List of all members.

Public Types
	MAX_DELIMITERS = 16
	MAX_PRESERVES = 16
enum	{ MAX_DELIMITERS = 16, MAX_PRESERVES = 16 }
Public Member Functions
	ACE_Tokenizer (ACE_TCHAR *buffer)
int	delimiter (ACE_TCHAR d)
int	delimiter_replace (ACE_TCHAR d, ACE_TCHAR replacement)
int	preserve_designators (ACE_TCHAR start, ACE_TCHAR stop, int strip=1)
ACE_TCHAR *	next (void)
	Returns the next token.
Protected Member Functions
int	is_delimiter (ACE_TCHAR d, int &replace, ACE_TCHAR &r)
int	is_preserve_designator (ACE_TCHAR start, ACE_TCHAR &stop, int &strip)
Protected Attributes
ACE_TCHAR *	buffer_
int	index_
Preserve_Entry	preserves_ [MAX_PRESERVES]
	The application can specify MAX_PRESERVES preserve designators.
int	preserves_index_
	Pointer to the next free spot in preserves_.
Delimiter_Entry	delimiters_ [MAX_DELIMITERS]
	The tokenizer allows MAX_DELIMITERS number of delimiters.
int	delimiter_index_
	Pointer to the next free space in delimiters_.
Classes
class	Delimiter_Entry
	Delimiter Entry. More...
class	Preserve_Entry
	Preserve Entry. More...

---

Detailed Description

Tokenizer.

Tokenizes a buffer. Allows application to set delimiters and preserve designators. Does not allow special characters, yet (e.g., printf ("\"like a quoted string\"")).

Definition at line 271 of file SString.h.

---

Member Enumeration Documentation

anonymous enum

Enumerator:

MAX_DELIMITERS
MAX_PRESERVES

Definition at line 381 of file SString.h.

       {
    MAX_DELIMITERS=16,
    MAX_PRESERVES=16
  };

---

Constructor & Destructor Documentation

ACE_Tokenizer::ACE_Tokenizer

(

ACE_TCHAR *

buffer

)

buffer will be parsed. Notice that ACE_Tokenizer will modify buffer if you use delimiter_replace or preserve_designators to do character substitution.

Note:

You should NOT pass a constant string or string literal to this constructor, since ACE_Tokenizer will try to modify the string.

See also:

preserve_designators

preserve_designators

Definition at line 339 of file SString.cpp.

  : buffer_ (buffer),
    index_ (0),
    preserves_index_ (0),
    delimiter_index_ (0)
{
}

---

Member Function Documentation

int ACE_Tokenizer::delimiter

(

ACE_TCHAR

d

)

d is a delimiter.

Returns:

Returns 0 on success, -1 if there is no memory left.

Example:

     char buf[30];
     ACE_OS::strcpy(buf, "William/Joseph/Hagins");

     ACE_Tokenizer tok (buf);
     tok.delimiter ('/');
     for (char *p = tok.next (); p; p = tok.next ())
      cout << p << endl;
    

This will print out:

     William/Joseph/Hagins
      Joseph/Hagins
      Hagins 

Definition at line 348 of file SString.cpp.

References ACE_Tokenizer::Delimiter_Entry::delimiter_, delimiter_index_, delimiters_, MAX_DELIMITERS, and ACE_Tokenizer::Delimiter_Entry::replace_.

{
  if (delimiter_index_ == MAX_DELIMITERS)
    return -1;

  delimiters_[delimiter_index_].delimiter_ = d;
  delimiters_[delimiter_index_].replace_ = 0;
  delimiter_index_++;
  return 0;
}

int ACE_Tokenizer::delimiter_replace	(	ACE_TCHAR	d,
		ACE_TCHAR	replacement
	)

d is a delimiter and, when found, will be replaced by replacement.

Returns:

0 on success, -1 if there is no memory left.

Example:

     char buf[30];
     ACE_OS::strcpy(buf, "William/Joseph/Hagins");

     ACE_Tokenizer tok (buf);
     tok.delimiter_replace ('/', 0);
     for (char *p = tok.next (); p; p = tok.next ())
       cout << p << endl;
    

This will print out:

       William
       Joseph
       Hagins 

Definition at line 360 of file SString.cpp.

References ACE_Tokenizer::Delimiter_Entry::delimiter_, delimiter_index_, delimiters_, MAX_DELIMITERS, ACE_Tokenizer::Delimiter_Entry::replace_, and ACE_Tokenizer::Delimiter_Entry::replacement_.

Referenced by ACE_Process_Options::command_line_argv().

{
  // Make it possible to replace delimiters on-the-fly, e.g., parse
  // string until certain token count and then copy rest of the
  // original string.
  for (int i = 0; i < delimiter_index_; i++)
    if (delimiters_[i].delimiter_ == d)
      {
        delimiters_[i].replacement_ = replacement;
        delimiters_[i].replace_ = 1;
        return 0;
      }

  if (delimiter_index_ >= MAX_DELIMITERS)
    return -1;

  delimiters_[delimiter_index_].delimiter_ = d;
  delimiters_[delimiter_index_].replacement_ = replacement;
  delimiters_[delimiter_index_].replace_ = 1;
  delimiter_index_++;
  return 0;
}

int ACE_Tokenizer::is_delimiter	(	ACE_TCHAR	d,
		int &	replace,
		ACE_TCHAR &	r
	)			[protected]

Returns 1 if <d> is a delimiter, 0 otherwise. If <d> should be replaced with r, <replace> is set to 1, otherwise 0.

Definition at line 400 of file SString.cpp.

References delimiter_index_, and delimiters_.

{
  replace = 0;

  for (int x = 0; x < delimiter_index_; x++)
    if (delimiters_[x].delimiter_ == d)
      {
        if (delimiters_[x].replace_)
          {
            r = delimiters_[x].replacement_;
            replace = 1;
          }
        return 1;
      }

  return 0;
}

int ACE_Tokenizer::is_preserve_designator	(	ACE_TCHAR	start,
		ACE_TCHAR &	stop,
		int &	strip
	)			[protected]

If <start> is a start preserve designator, returns 1 and sets <stop> to the stop designator. Returns 0 if <start> is not a preserve designator.

Definition at line 421 of file SString.cpp.

References preserves_, preserves_index_, and ACE_Tokenizer::Preserve_Entry::strip_.

{
  for (int x = 0; x < preserves_index_; x++)
    if (preserves_[x].start_ == start)
      {
        stop = preserves_[x].stop_;
        strip = preserves_[x].strip_;
        return 1;
      }

  return 0;
}

ACE_TCHAR * ACE_Tokenizer::next

(

void

)

Returns the next token.

Definition at line 437 of file SString.cpp.

References buffer_, and index_.

Referenced by ACE_Process_Options::command_line_argv().

{
  // Check if the previous pass was the last one in the buffer.
  if (index_ == -1)
    {
      index_ = 0;
      return 0;
    }

  ACE_TCHAR replacement = 0;
  int replace;
  ACE_TCHAR *next_token;

  // Skip all leading delimiters.
  for (;;)
    {
      // Check for end of string.
      if (buffer_[index_] == '\0')
        {
          // If we hit EOS at the start, return 0.
          index_ = 0;
          return 0;
        }

      if (this->is_delimiter (buffer_[index_],
                              replace,
                              replacement))
        index_++;
      else
        break;
    }

  // When we reach this point, buffer_[index_] is a non-delimiter and
  // not EOS - the start of our next_token.
  next_token = buffer_ + index_;

  // A preserved region is it's own token.
  ACE_TCHAR stop;
  int strip;
  if (this->is_preserve_designator (buffer_[index_],
                                    stop,
                                    strip))
    {
      while (++index_)
        {
          if (buffer_[index_] == '\0')
            {
              index_ = -1;
              goto EXIT_LABEL;
            }

          if (buffer_[index_] == stop)
            break;
        }

      if (strip)
        {
          // Skip start preserve designator.
          next_token += 1;
          // Zap the stop preserve designator.
          buffer_[index_] = '\0';
          // Increment to the next token.
          index_++;
        }

      goto EXIT_LABEL;
    }

  // Step through finding the next delimiter or EOS.
  for (;;)
    {
      // Advance pointer.
      index_++;

      // Check for delimiter.
      if (this->is_delimiter (buffer_[index_],
                              replace,
                              replacement))
        {
          // Replace the delimiter.
          if (replace != 0)
            buffer_[index_] = replacement;

          // Move the pointer up and return.
          index_++;
          goto EXIT_LABEL;
        }

      // A preserve designator signifies the end of this token.
      if (this->is_preserve_designator (buffer_[index_],
                                        stop,
                                        strip))
        goto EXIT_LABEL;

      // Check for end of string.
      if (buffer_[index_] == '\0')
        {
          index_ = -1;
          goto EXIT_LABEL;
        }
    }

EXIT_LABEL:
  return next_token;
}

int ACE_Tokenizer::preserve_designators	(	ACE_TCHAR	start,
		ACE_TCHAR	stop,
		int	strip = 1
	)

Extract string between a pair of designator characters. For instance, quotes, or '(' and ')'. start specifies the begin designator. stop specifies the end designator. strip If strip == 1, then the preserve designators will be stripped from the tokens returned by next.

Returns:

0 on success, -1 if there is no memory left.

Example with strip = 0:

     char buf[30];
     ACE_OS::strcpy(buf, "William(Joseph)Hagins");

     ACE_Tokenizer tok (buf);
     tok.preserve_designators ('(', ')', 0);
     for (char *p = tok.next (); p; p = tok.next ())
       cout << p << endl;
    

This will print out:

      William(Joseph)Hagins
      (Joseph)Hagins
      )Hagins 

Example with strip = 1:

     char buf[30];
     ACE_OS::strcpy(buf, "William(Joseph)Hagins");

     ACE_Tokenizer tok (buf);
     tok.preserve_designators ('(', ')', 1);
     for (char *p = tok.next (); p; p = tok.next ())
       cout << p << endl;
    

This will print out:

      William
      Joseph
      Hagins 

Definition at line 385 of file SString.cpp.

References MAX_PRESERVES, preserves_, preserves_index_, ACE_Tokenizer::Preserve_Entry::start_, ACE_Tokenizer::Preserve_Entry::stop_, and ACE_Tokenizer::Preserve_Entry::strip_.

Referenced by ACE_Process_Options::command_line_argv().

{
  if (preserves_index_ == MAX_PRESERVES)
    return -1;

  preserves_[preserves_index_].start_ = start;
  preserves_[preserves_index_].stop_ = stop;
  preserves_[preserves_index_].strip_ = strip;
  preserves_index_++;
  return 0;
}

---

Member Data Documentation

ACE_TCHAR* ACE_Tokenizer::buffer_ [protected]

Definition at line 398 of file SString.h.

Referenced by next().

int ACE_Tokenizer::delimiter_index_ [protected]

Pointer to the next free space in delimiters_.

Definition at line 458 of file SString.h.

Referenced by delimiter(), delimiter_replace(), and is_delimiter().

Delimiter_Entry ACE_Tokenizer::delimiters_[MAX_DELIMITERS] [protected]

The tokenizer allows MAX_DELIMITERS number of delimiters.

Definition at line 455 of file SString.h.

Referenced by delimiter(), delimiter_replace(), and is_delimiter().

int ACE_Tokenizer::index_ [protected]

Definition at line 399 of file SString.h.

Referenced by next().

Preserve_Entry ACE_Tokenizer::preserves_[MAX_PRESERVES] [protected]

The application can specify MAX_PRESERVES preserve designators.

Definition at line 427 of file SString.h.

Referenced by is_preserve_designator(), and preserve_designators().

int ACE_Tokenizer::preserves_index_ [protected]

Pointer to the next free spot in preserves_.

Definition at line 430 of file SString.h.

Referenced by is_preserve_designator(), and preserve_designators().

---

The documentation for this class was generated from the following files:

• SString.h
• SString.cpp

---