Cheat Sheet for Natural Language Processing using NTLK
|
import nltk
|
|
|
nltk.download()
|
This step will bring up a window in which you can download ‘All Corpora’
|
|
from nltk.book import *
|
|
|
text1[0:100]
|
-first 101 tokens
|
|
text2[5]
|
- fifth token
|
|
> text3.concordance(‘begat’)
|
- basic keyword-in-context
|
|
text1.concordance(‘sea’, lines=100)
|
-- show other than default 25 lines
|
|
> text1.concordance(‘sea’, lines=100)
|
- show other than default 25 lines
|
|
text1.concordance(‘sea’, lines=all)
|
- show all results
|
|
text1.concordance(‘sea’, 10, lines=all) -
|
- change left and right context width to 10 characters and show all results
|
|
text1.common_contexts([‘sea’,’ocean’])
|
|
Count a String
|
len(‘this is a string of text’) – number of characterslen(‘this is a string of text’) – number of characters
|
|
Count a list of tokens
|
len(text1) –number of tokens
|
|
Make and Count a list of unique tokens
|
len(set(text1)) – notice that set return a list of unique tokens
|
|
Count Occurences
|
text1.count(‘heaven’) – how many times does a word occur?
|
|
Frequency
|
|
|
fd = nltk.FreqDist(text1) – creates a new data object that contains information about word frequency
|
|
|
fd[‘the’] – how many occurences of the word ‘the’
|
|
|
fd.keys() – show the keys in the data object
|
|
|
fd.values() – show the values in the data object
|
|
|
fd.items() – show everything
|
|
|
fd.keys()[0:50] – just show a portion of the info
|
|
|
Frequency Plots
|
fd.plot(50,cumulative=False) – generate a chart of the 50 most frequent words
|
|
Other FreqDist functions
|
fd.hapaxes()
|
|
fd.freq(‘the’)
|
|
|
Get word lengths
|
lengths = [len(w) for w in text1]
|
|
And do FreqDist
|
fd = nltk.FreqDist(lengths)
|
|
FreqDist as Table
|
fd.tabulate()
|
|
De-punctuate
|
[w for w in text1 if w.isalpha() ] – not so much getting rid of punctuation, but keeping alphabetic characters
|
|
De-uppercaseify (?)
|
>[w.lower() for w in text] – make each word in the tokenized list lowercase
|
|
[w.lower() for w in text if w.isalpha()] – all in one go
|
|
|
Sort
|
sorted(text1) – careful with this!
|
|
Unique Words
|
set(text1) – set is oddly named, but very powerful. Leaves you with a list of only one of each word.
|
|
Exclude Stopwords
|
Make your own list of word to be excluded:
|
|
stopwords = [‘the’,’it’,’she’,’he’]
|
|
|
mynewtext = [w for w in text1 if w not in stopwords]
|
|
|
Or you can also use predefined stopword lists from NLTK:
|
|
|
from nltk.corpus import stopwords
|
|
|
stopwords = stopwords.words(‘english’)
|
|
|
mynewtext = [w for w in text1 if w not in stopwords]
|
|
Dispersion Plot
|
text4.dispersion_plot([‘American’,’Liberty’,’Government’])
|
|
Find Word that ends with...
|
[w for w in text4 if w.endswith(‘ness’)]
|
|
Find Word that start with...
|
[w for w in text4 if w.startsswith(‘ness’)]
|
|
Find Word that contain...
|
[w for w in text4 if ‘ee’ in w]
|
|
Combine them together
|
[w for w in text4 if ‘ee’ in w and w.endswith(‘ing’)] Regular expressions ‘Regular expressions’ is a syntax for describing sequences
|
|
Regular Expressions
|
‘Regular expressions’ is a syntax for describing sequences of characters usually used to construct search queries. The Python ‘re’ module must first be imported:
|
|
Import
|
|
|
>>>import re >>>[w for w in text1 if re.search('^ab',w)] – ‘Regular expressions’ is too big of a topic to cover here. Google it!
|
|
|
>>>import re >>>[w for w in text1 if re.search('^ab',w)] – ‘Regular expressions’ is too big of a topic to cover here. Google it!
|
|
|
Collocations
|
> text4.collocations() - multi-word expressions that commonly co-occur. Notice that is not necessarily related to the frequency of the words
|
|
>text4.collocations(num=100) – alter the number of phrases returned Bigrams, Trigrams, and n-grams are useful for comparing texts, particularly for plagiarism detection and collation
|
|
|
Bi-grams
|
>nltk.bigrams(text4) – returns every string of two words
|
|
Tri-grams
|
nltk.trigrams(text4) – return every string of three word
|
|
n-grams
|
nltk.ngrams(text4, 5)
|
|
part-of-speach tagging
|
mytext = nltk.word_tokenize(“This is my sentence”)
|
|
nltk.pos_tag(mytext)
|
|
Open a file for reading
|
>file = open(‘myfile.txt’) – make sure you are in the correct directory before starting Python
|
|
Read the file
|
t = file.read();
|
|
Tokenize the file
|
tokens = nltk.word_tokenize(t)
|
|
Convert to NLTK text object
|
text = nltk.Text(tokens)
|
|
Quit
|
quit()
|
Cheatography n. The art of mapping complex subjects into clear, instantly navigable reference guides.
