uk.ac.bham.gloss
Class Token

java.lang.Object
  uk.ac.bham.gloss.Token

All Implemented Interfaces:

Locator

public class Token
extends java.lang.Object
implements Locator

A token is a unit of textual information from an input file. This class represents such tokens. There are around 17 types of tokens used here, with different uses. Each has a String value as well as its type. Each type is documented fully below. As well as storing a type and a string, the token also contains the line and column number from the source stream, and a "depth" which is a measure of how deep into the source (in terms of element-nesting) the token is. Copyright Richard Kaye 2007-8 for GLOSS, http://gloss.bham.ac.uk Usage permitted according to the GPL. No waranty.

Constructor Summary

Token(int type, int depth, java.lang.String data, int l, int c, java.lang.String d)
Constructor for a token with given type, depth, data, and location.

Token(int type, int depth, java.lang.String data, Locator l)
Constructor for a token with given type, depth, data, and locator.

Token(java.lang.String type, int depth, java.lang.String data, Locator l)
Construct a token with type, depth and data and locator as given.

Method Summary

int	col() Return the column number of this token.
java.lang.String	data() Return the data of this token as a string.
int	depth() Returns the depth of this token as an int.
java.lang.String	doc() Returns the document name of this token.
boolean	equals(java.lang.Object t) equality method for Tokens: two tokens are equal by this method if they have the same type and data and exactly the same depth and coordinates.
java.lang.String	fullName() Returns the "full name" part of this token's data.
boolean	hasType(java.lang.String t) Returns true if the type of this token is one of those listed in the string t.
static int	intType(java.lang.String item) Returns the int value of the input which is a string representation of a Token type; eg returns Token.UCTYPE on input "uc".
static int	intTypes(java.lang.String t) Returns a | separated int of all types that appear in the string t.
int	itype() Returns the type of this token as an int.
int	line() Returns the line number of this token.
java.lang.String	localName() Returns the local-name part of this token's name.
java.lang.String	prefix() Returns the prefix or namespace part of this token's name.
java.lang.String	stype() Returns the type of this token as a String.
java.lang.String	toString() Returns a string representation of this token.
static java.lang.String	toString(Token t) static version of toString that checks for t==null
java.lang.String	type() Returns the type of this token as a String.
protected static java.lang.String	unescape(java.lang.String x) Unescapes a string where: \n->newline \r->cr, \t->tab, \c->c, for any other c and c->c otherwise.
static java.lang.String	validTypes(int allowedtypes) Returns a string of the form "type|type|...|type" representing all the allowed types and allowed punctuation.
static java.lang.String	validTypes(int allowedtypes, java.lang.String punctvals) Returns a string of the form "type|type|...|type|p|p" representing all the allowed types and allowed punctuation.
java.lang.String	value() Returns the value part of this token's data.
protected static java.lang.String	xmlescape(java.lang.String x) escapes a string where: newline-> cr-> , tab-> , "->", '->' &->& etc and c->c otherwise.

Methods inherited from class java.lang.Object

clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Token

public Token(int type,
             int depth,
             java.lang.String data,
             int l,
             int c,
             java.lang.String d)

Constructor for a token with given type, depth, data, and location.

Token

public Token(int type,
             int depth,
             java.lang.String data,
             Locator l)

Constructor for a token with given type, depth, data, and locator.

Token

public Token(java.lang.String type,
             int depth,
             java.lang.String data,
             Locator l)

Construct a token with type, depth and data and locator as given.

Method Detail

intType

public static int intType(java.lang.String item)

Returns the int value of the input which is a string representation of a Token type; eg returns Token.UCTYPE on input "uc". Returns Token.PUNCTYPE for any input string not recognised.

validTypes

public static java.lang.String validTypes(int allowedtypes)

Returns a string of the form "type|type|...|type" representing all the allowed types and allowed punctuation. Eg on input Token.LABELTYPE|Token.EREFTYPE returns "label|eref".

validTypes

public static java.lang.String validTypes(int allowedtypes,
                                          java.lang.String punctvals)

Returns a string of the form "type|type|...|type|p|p" representing all the allowed types and allowed punctuation. Eg on input Token.LABELTYPE|Token.EREFTYPE,"[|]" returns "label|eref|[|]". The string "punc" for punctation will not appear in the output but punctvals will be ignored unless (allowedtypes&PUNCTTYPE)!=0.

equals

public boolean equals(java.lang.Object t)

equality method for Tokens: two tokens are equal by this method if they have the same type and data and exactly the same depth and coordinates.

Overrides:

equals in class java.lang.Object

hasType

public boolean hasType(java.lang.String t)

Returns true if the type of this token is one of those listed in the string t. Here t is a |-separated list of short names.

intTypes

public static int intTypes(java.lang.String t)

Returns a | separated int of all types that appear in the string t. Here t is a |-separated list of short names.

itype

public int itype()

Returns the type of this token as an int.

stype

public java.lang.String stype()

Returns the type of this token as a String.

type

public java.lang.String type()

Returns the type of this token as a String.

depth

public int depth()

Returns the depth of this token as an int.

data

public java.lang.String data()

Return the data of this token as a string. The data is a single string representation of the data that made up the token in the first place. For elt, pelt, attr, eref, pi, label, pref it is "NAME" or "PREFIX:LNAME" For uc it is the character itself encoded in UTF-16 For fp, hex, int it is a string containing the number For str it is the string itself without the delimiting " For cref it is the string without leading & and trailing ; For b64 it is the b64 string without the leading = but with any padding = For uri it is the uri (without the leading ~) For pdef it is "PARNAME=STRING" For char, cref it is a string "#NUMBER=CHAR"

doc

public java.lang.String doc()

Returns the document name of this token.

Specified by:

doc in interface Locator

line

public int line()

Returns the line number of this token.

Specified by:

line in interface Locator

col

public int col()

Return the column number of this token.

Specified by:

col in interface Locator

fullName

public java.lang.String fullName()

Returns the "full name" part of this token's data. For cref,label,eref,elt,pref,pi,attr,pelt this is the same as data(). For pdef it is the first part "name" of the name/value pair name=value. For uc, char it is the hex code of the character's codepoint (0xXXXX). Otherwise it is the empty string.

prefix

public java.lang.String prefix()

Returns the prefix or namespace part of this token's name. For pref or pdef, if fullName() starts with '{' it is the initial part {...} where ... contains balanced {...} or "" if there is no such initial part. Otherwise, if fullName() contains ':' it is that part before the first ':' Otherwise it is the empty string.

localName

public java.lang.String localName()

Returns the local-name part of this token's name. This is the part of fullName() after the prefix. If prefix() is "" then localName() is the same as fullName(). Otherwise PREFIX is prefix() itself if this starts with '{' or prefix()+":" otherwise, and localName is the part of fullName folowing PREFIX.

value

public java.lang.String value()

Returns the value part of this token's data. For label,eref,elt,pref,pi,attr,pelt this is "". (This class does NOT evaluate prefs.) For pdef it is the second part of the name/value pair name=value. For cref it is the UTF-16 encoded character Otherwise it is the entire data.

toString

public java.lang.String toString()

Returns a string representation of this token. This is in XML-form <type d="..." l="..." c="...">data</token> with the attributes giving type, depth, line, column, and the data "escaped" using the XML entities &amp; &apos; &quot; &lt; &gt; .

Overrides:

toString in class java.lang.Object

toString

public static java.lang.String toString(Token t)

static version of toString that checks for t==null

unescape

protected static java.lang.String unescape(java.lang.String x)

Unescapes a string where: \n->newline \r->cr, \t->tab, \c->c, for any other c and c->c otherwise.

xmlescape

protected static java.lang.String xmlescape(java.lang.String x)

escapes a string where: newline-> cr-> , tab-> , "->", '->' &->& etc and c->c otherwise.