Next, we can train the Punkt tokenizer like: custom_sent_tokenizer = PunktSentenceTokenizer(train_text) One is a State of the Union address from 2005, and the other is from 2006 from past President George W. Sample_text = state_union.raw("2006-GWBush.txt") Now, let's create our training and testing data: train_text = state_union.raw("2005-GWBush.txt") First, let's get some imports out of the way that we're going to use: import nltkįrom nltk.tokenize import PunktSentenceTokenizer This tokenizer is capable of unsupervised machine learning, so you can actually train it on any body of text that you use. How might we use this? While we're at it, we're going to cover a new sentence tokenizer, called the PunktSentenceTokenizer. VBG verb, gerund/present participle taking Here's a list of the tags, what they mean, and some examples: POS tag list:ĮX existential there (like: "there is". Even more impressive, it also labels by tense, and more.
This means labeling words in a sentence as nouns, adjectives, verbs.etc. One of the more powerful aspects of the NLTK module is the Part of Speech tagging that it can do for you.
0 kommentar(er)
