Using the nltk.tokenize.WhitespaceTokenizer()
method we can extract tokens from a string of words or sentences without spaces , new and tabs strings with tokenize.WhitespaceTokenizer ()
.
Syntax:
tokenize.WhitespaceTokenizer ()
Return: Return the tokens from a string
Example # 1:
In this example, we see that using tokenize.WhitespaceTokenizer ()
we can extract tokens from the word stream.
|
Output:
[’GeeksforGeeks’, ’is’, ’for’, ’geeks’]
Example No. 2:
|
Output:
[’The’, ’price’, ’of’, ’burger’, ’in’, ’BurgerKing’, ’is’, ’Rs.36.’]