tokenise-string

Description

Tokenises a string, breaking at arbitrary character classes

Argument list

str = String (Given)

String to be tokenised

boundary-char? = function (Given), keyword

Character-class function, which takes a single character as argument, and returns true if it should be categorised as whitespace, and false otherwise. If this is insufficiently flexible, then the isbdy? function can be used. The default is the function isspace?

isbdy? = function (Given), keyword

General character-class function, which takes as argument a list of characters, and returns #f if the string should not be broken here (ie, if the string does not begin with separator characters). If the string should be broken here, it returns the list of characters which consists of the remainder of the string with the leading separators removed. For example, the function (lambda (l) (if (and (char=? (car l) #\/) (char=? (cadr l) #\/)) (cddr l) #f)) breaks strings at double-slashes, but not single ones. By default, the function removes strings of characters for which boundary-char? is true.

max = integer (Given), keyword

Integer which specifies the maximum number of splits which should be made. Thus, for example, if max is passed as 1, the string will be split into a maximum of two pieces; if this is negative (the default) the string will be completely tokenised. Note that removal of initial whitespace counts as a `split'.

Return value

Type: list

List of strings, each containing a single token