Tokenizes a paragraph separating words from spaces. More...

Public Types
typedef std::list< std::pair < std::string, std::string > >	WordTokenizerResults
	Results of the tokenizer; each element is a pair where the first string represents a possible space and the second string a possible word. More...

Static Public Member Functions
static void	tokenize (const std::string &s, WordTokenizerResults &results)
	Tokenizes the passed string and stores the results. More...

Detailed Description

Tokenizes a paragraph separating words from spaces.

Member Typedef Documentation

typedef std::list<std::pair<std::string, std::string> > srchilite::WordTokenizer::WordTokenizerResults

Results of the tokenizer; each element is a pair where the first string represents a possible space and the second string a possible word.

The two elements are mutually exclusive

void srchilite::WordTokenizer::tokenize	(	const std::string &	s,
		WordTokenizerResults &	results
	)

static

Tokenizes the passed string and stores the results.

Parameters

s	the string to tokenize
results	where to store the results

The documentation for this class was generated from the following files:

/home/bettini/work/source-highlight/src-highlite/lib/srchilite/wordtokenizer.h
/home/bettini/work/source-highlight/src-highlite/lib/srchilite/wordtokenizer.cpp