Change language

NLP | Configuring Using Tagged Corpus Reader

| | |

How can we use Tagged Corpus Reader?

  • Configuring the word tokenizer
  • Configuring the tokenizer proposal
  • Configuring the reader paragraph blocks
  • Customizing the tag separator
  • Converting tags to a universal set of tags

Code # 1: Configuring Word Tokenizer

# Loading libraries

from nltk. tokenize import SpaceTokenizer

from nltk.corpus.reader import TaggedCorpusRead er

 

x = TaggedCorpusReader ( ’ .’ , r ’. *. pos’

  word_tokenizer = SpaceTokenizer ())

 
x.words ()

Output:

 [’The’,’ expense’, ’and’,’ time ’,’ involved’, ’are’, ...] 

Code # 2: for proposal

# Loading Libraries

from nltk.tokenize import LineTokenizer

from nltk .corpus.reader import TaggedCorpusReader

 

x = TaggedCorpusReader ( ’.’ , r ’. *. pos’

sent_tokenizer = LineTokenizer ())

  
x.sents ()

Output:

 [[’The’,’ expense’, ’and’,’ time’, ’involved’,’ are’, ’astronomical’,’ .’]] 

Paragraph setup

  • Suppose a paragraph separated by blank lines
  • Made with para_block_reader function, which is nltk.corpus.reader.util.read_blankline_block
  • Number of other block readers is present in nltk.corpus.reader.util whose purpose is to read blocks of text from the stream.

Setting the delimiter tag

  • If & # 39; / & # 39; is not used as a word / tag separator, you can pass an alternate string to the TaggedCorpusReader for sep.
  • By default, this is sep = & # 39; / & # 39; , but if anyone someone wants to separate words and tags with & # 39; | & # 39;, for example & # 39; word | tag & # 39; then sep = & # 39; | & # 39; passed to.

Converting tags to a generic tag set
Tagset: is a list of POS tags used by one or more corporations. 
Generic tag set: this is a simplified and concise tag set with only 12 part-of-speech tags

Code # 3: Match corpus tags to generic tag set

from nltk.corpus.reader import TaggedCorpusReader

  

x = TaggedCorpusReader ( ’.’ , r ’. *. pos’ , tagset = ’en-brown’ )

x.tagged_words (tagset = ’universal’ )

Output:

 [(’The’,’ DET’), (’expense’,’ NOUN’), (’and’,’ CONJ’), ...] 

Code # 4: Map corpus tags to generic tags

Output:

 [(’Pierre’,’ NNP’), (’Vinken’,’ NNP’), (’,’, ’,’), ...] [(’Pierre’,’ NOUN’), (’Vinken’,’ NOUN’), (’,’, ’.’), ... ] [(’Pierre’,’ UNK’), (’Vinken’,’ UNK’), (’,’, ’UNK’), ...] 

Shop

Learn programming in R: courses

$

Best Python online courses for 2022

$

Best laptop for Fortnite

$

Best laptop for Excel

$

Best laptop for Solidworks

$

Best laptop for Roblox

$

Best computer for crypto mining

$

Best laptop for Sims 4

$

Latest questions

NUMPYNUMPY

Common xlabel/ylabel for matplotlib subplots

12 answers

NUMPYNUMPY

How to specify multiple return types using type-hints

12 answers

NUMPYNUMPY

Why do I get "Pickle - EOFError: Ran out of input" reading an empty file?

12 answers

NUMPYNUMPY

Flake8: Ignore specific warning for entire file

12 answers

NUMPYNUMPY

glob exclude pattern

12 answers

NUMPYNUMPY

How to avoid HTTP error 429 (Too Many Requests) python

12 answers

NUMPYNUMPY

Python CSV error: line contains NULL byte

12 answers

NUMPYNUMPY

csv.Error: iterator should return strings, not bytes

12 answers


Wiki

Python | How to copy data from one Excel sheet to another

Common xlabel/ylabel for matplotlib subplots

Check if one list is a subset of another in Python

sin

How to specify multiple return types using type-hints

exp

Printing words vertically in Python

exp

Python Extract words from a given string

Cyclic redundancy check in Python

Finding mean, median, mode in Python without libraries

cos

Python add suffix / add prefix to strings in a list

Why do I get "Pickle - EOFError: Ran out of input" reading an empty file?

Python - Move item to the end of the list

Python - Print list vertically

from nltk.corpus.reader import TaggedCorpusReader

from nltk.corpus import treebank

 
treebank.tagged_words ()

 

treebank.tagged_words (tagset = ’universal’ )

 

treebank.tagged_words (tagset = ’brown’ )