Science Fair Project Encyclopedia
Tokenizing
(Redirected from Tokenize)
Tokenizing is the operation of splitting up a string of characters into a set of tokens.
The term is also used when during the parsing of source code of some programming languages, the symbols are converted into another format that is much smaller. Most BASIC interpreters used this to save room, a command such as print would be replaced by a single number which uses much less room in memory. In fact most lossless compression systems use a form of tokenizing, although it's typically not referred to as such.
10-26-2009 08:16:03
The contents of this article is licensed from www.wikipedia.org under the GNU Free Documentation License. Click here to see the transparent copy and copyright details
The contents of this article is licensed from www.wikipedia.org under the GNU Free Documentation License. Click here to see the transparent copy and copyright details