Using the nltk.tokenize.word_tokenize()
method, we can extract tokens from a character string using tokenize.word_tokenize ()
. This actually returns syllables from a single word. One word can contain one or two syllables.
Syntax:
tokenize.word_tokenize ()
Return: Return the list of syllables of words.
Example # 1:
In this example we see that with tokenize.word_tokenize ()
we can extract syllables from a stream of words or sentences.
|
Output:
[’An’, ’ti’, ’dis’, ’es’, ’ta’, ’blish’, ’men’, ’ta’, ’ria’, ’nism’]
Example # 2:
|
[’Ga’, ’me’, ’to’, ’phy’, ’te’]