Python NLTK | nltk.WhitespaceTokenizer

Michael Zippo

Using the nltk.tokenize.WhitespaceTokenizer() method we can extract tokens from a string of words or sentences without spaces , new and tabs strings with tokenize.WhitespaceTokenizer () .

Syntax: tokenize.WhitespaceTokenizer ()
Return: Return the tokens from a string

Example # 1:
In this example, we see that using tokenize.WhitespaceTokenizer () we can extract tokens from the word stream.

# import WhitespaceTokenizer () method from nltk

from nltk .tokenize import WhitespaceTokenizer

# Create a reference variable for the WhitespaceTokenizer class

tk = WhitespaceTokenizer ()

# Create input line

gfg = "Python.Engineering is for geeks"

# Use the tokenization method

geek = tk.tokenize (gfg)

print (geek)

Output:

[’GeeksforGeeks’, ’is’, ’for’, ’geeks’]

Example No. 2:

# import WhitespaceTokenizer () method from nltk

from nltk.tokenize import WhitespaceTokenizer

# Create a reference variable for the WhitespaceTokenizer class

tk = WhitespaceTokenizer ()

# Create input line

gfg = "The price of burger in BurgerKing is Rs. 36. "

# Use the tokenization method

geek = tk.tokenize (gfg)

print (geek)

Output:

[’The’, ’price’, ’of’, ’burger’, ’in’, ’BurgerKing’, ’is’, ’Rs.36.’]

Shop

Learn programming in R: courses

Best Python online courses for 2022

Best laptop for Fortnite

Best laptop for Excel

Best laptop for Solidworks

Best laptop for Roblox

Best computer for crypto mining

Best laptop for Sims 4

Python NLTK | nltk.WhitespaceTokenizer

Shop

News

Wiki