Histories and Cultures of Tourism. Histories of American Education. The Liberty Hyde Bailey Library. New Netherland Institute Studies. The Northern Forest Atlas Guides. Brown Democracy Medal Books. Cornell Studies in Money. Cornell Studies in Classical Philology. Myth and Poetics II. Persian Gulf Studies. Religion and American Public Life. Religion and Conflict. New Japanese Horizons. Zona Tropical Publications. Cornell University Press. Comstock Publishing.
ILR Press. Northern Illinois University Press. Three Hills. We can pull them apart by indexing and slicing them, and we can join them together by concatenating them.
However, we cannot join strings and lists:. When we open a file for reading into a Python program, we get a string corresponding to the contents of the whole file.
If we use a for loop to process the elements of this string, all we can pick out are the individual characters — we don't get to choose the granularity. By contrast, the elements of a list can be as big or small as we like: for example, they could be paragraphs, sentences, phrases, words, characters. So lists have the advantage that we can be flexible about the elements they contain, and correspondingly flexible about any downstream processing.
Consequently, one of the first things we are likely to do in a piece of NLP code is tokenize a string into a list of strings 3. Conversely, when we want to write our results to a file, or to a terminal, we will usually format them as a string 3.
Lists and strings do not have exactly the same functionality. Lists have the added power that you can change their elements:. On the other hand if we try to do that with a string — changing the 0th character in query to 'F' — we get:. This is because strings are immutable — you can't change a string once you have created it. However, lists are mutable , and their contents can be modified at any time.
As a result, lists support operations that modify the original value rather than producing a new value. Your Turn: Consolidate your knowledge of strings by trying some of the exercises on strings at the end of this chapter. Our programs will often need to deal with different languages, and different character sets. The concept of "plain text" is a fiction. In this section, we will give an overview of how to use Unicode for processing texts that use non-ASCII character sets.
Unicode supports over a million characters. Each character is assigned a number, called a code point. Within a program, we can manipulate Unicode strings just like normal strings. However, when Unicode characters are stored in files or displayed on a terminal, they must be encoded as a stream of bytes. Some encodings such as ASCII and Latin-2 use a single byte per code point, so they can only support a small subset of Unicode, enough for a single language.
Other encodings such as UTF-8 use multiple bytes and can represent the full range of Unicode characters. Text in files will be in a particular encoding, so we need some mechanism for translating it into Unicode — translation into Unicode is called decoding. Conversely, to write out Unicode to a file or a terminal, we first need to translate it into a suitable encoding — this translation out of Unicode is called encoding , and is illustrated in 3.
From a Unicode perspective, characters are abstract entities which can be realized as one or more glyphs. Only glyphs can appear on a screen or be printed on paper. A font is a mapping from characters to glyphs. Let's assume that we have a small text file, and that we know how it is encoded. For example, polish-lat2. This file is encoded as Latin-2, also known as ISO The function nltk.
The Python open function can read encoded data into Unicode strings, and write out Unicode strings in encoded form. It takes a parameter to specify the encoding of the file being read or written. So let's open our Polish file with the encoding 'latin2' and inspect the contents of the file:. We find the integer ordinal of a character using ord. For example:. The hexadecimal 4 digit notation for is type hex to discover this , and we can define a string with the appropriate escape sequence.
There are many factors determining what glyphs are rendered on your screen. If you are sure that you have the correct encoding, but your Python code is still failing to produce the glyphs you expected, you should also check that you have the necessary fonts installed on your system.
It may be necessary to configure your locale to render UTF-8 encoded characters, then use print nacute. The module unicodedata lets us inspect the properties of Unicode characters. In the following example, we select all characters in the third line of our Polish text outside the ASCII range and print their UTF-8 byte sequence, followed by their code point integer using the standard Unicode convention i. If you replace c. Alternatively, you may need to replace the encoding 'utf8' in the example by 'latin2' , again depending on the details of your system.
The next examples illustrate how Python string methods and the re module can work with Unicode characters. We will take a close look at the re module in the following section. If you are used to working with characters in a particular local encoding, you probably want to be able to use your standard methods for inputting and editing strings in a Python file.
Many linguistic processing tasks involve pattern matching. For example, we can find words ending with ed using endswith 'ed'. We saw a variety of such "word tests" in 4. Regular expressions give us a more powerful and flexible method for describing the character patterns we are interested in. There are many other published introductions to regular expressions, organized around the syntax of regular expressions and applied to searching text files.
Instead of doing this again, we focus on the use of regular expressions at different stages of linguistic processing. As usual, we'll adopt a problem-based approach and present new features only as they are needed to solve practical problems.
In our discussion we will mark regular expressions using chevrons like this: « patt ». To use regular expressions in Python we need to import the re library using: import re. We also need a list of words to search; we'll use the Words Corpus again 4. We will preprocess it to remove any proper names. We will use the re. We need to specify the characters of interest, and use the dollar sign which has a special behavior in the context of regular expressions in that it matches the end of the word:.
Suppose we have room in a crossword puzzle for an 8-letter word with j as its third letter and t as its sixth letter. In place of each blank cell we use a period:. What results do we get with the above example if we leave out both of these, and search for «.. Finally, the? We could count the total number of occurrences of this word in either spelling in a text using sum 1 for w in text if re.
The T9 system is used for entering text on mobile phones see 3. Two or more words that are entered with the same sequence of keystrokes are known as textonyms. For example, both hole and golf are entered by pressing the sequence What other words could be produced with the same sequence? The next part of the expression, « [mno] », constrains the second character to be m , n , or o. The third and fourth characters are also constrained.
Only four words satisfy all these constraints. Your Turn: Look for some "finger-twisters", by searching for words that only use part of the number-pad. Notice that it can be applied to individual letters, or to bracketed sets of letters:. Notice this includes non-alphabetic characters. You probably worked out that a backslash means that the following character is deprived of its special powers and must literally match a specific character in the word.
Thus, while. The pipe character indicates a choice between the material on its left or its right. Parentheses indicate the scope of an operator: they can be used together with the pipe or disjunction symbol like this: « w i e ai oo t », matching wit , wet , wait , and woot.
The meta-characters we have seen are summarized in 3. To the Python interpreter, a regular expression is just like any other string. If the string contains a backslash followed by particular characters, it will interpret these specially. In general, when using regular expressions containing backslash, we should instruct the interpreter not to look inside the string at all, but simply to pass it directly to the re library for processing.
We do this by prefixing the string with the letter r , to indicate that it is a raw string. If you get into the habit of using r ' The above examples all involved searching for words w that match some regular expression regexp using re. Apart from checking if a regular expression matches a word, we can use regular expressions to extract material from words, or to modify words in specific ways.
The re. Let's find all the vowels in a word, then count them:. Let's look for all sequences of two or more vowels in some text, and determine their relative frequency:. Replace the? Once we can use re. It is sometimes noted that English text is highly redundant, and it is still easy to read when word-internal vowels are left out.
For example, declaration becomes dclrtn , and inalienable becomes inlnble , retaining any initial or final vowel sequences. The regular expression in our next example matches initial vowel sequences, final vowel sequences, and all consonants; everything else is ignored. This three-way disjunction is processed left-to-right, if one of the three parts matches the word, any later parts of the regular expression are ignored. We use re. Next, let's combine regular expressions with conditional frequency distributions.
Here we will extract all consonant-vowel sequences from the words of Rotokas, such as ka and si. Since each of these is a pair, it can be used to initialize a conditional frequency distribution. We then tabulate the frequency of each pair:. Examining the rows for s and t , we see they are in partial "complementary distribution", which is evidence that they are not distinct phonemes in the language.
Thus, we could conceivably drop s from the Rotokas alphabet and simply have a pronunciation rule that the letter t is pronounced s when followed by i.
Note that the single entry having su , namely kasuari , 'cassowary' is borrowed from English. If we want to be able to inspect the words behind the numbers in the above table, it would be helpful to have an index, allowing us to quickly find the list of words that contains a given consonant-vowel pair, e. Here's how we can do this:. This program processes each word w in turn, and for each one, finds every substring that matches the regular expression « [ptksvr][aeiou] ». In the case of the word kasuari , it finds ka , su and ri.
One further step, using nltk. Index , converts this into a useful index. When we use a web search engine, we usually don't mind or even notice if the words in the document differ from our search terms in having different endings. A query for laptops finds documents containing laptop and vice versa.
Indeed, laptop and laptops are just two forms of the same dictionary word or lemma. For some language processing tasks we want to ignore word endings, and just deal with word stems.
There are various ways we can pull out the stem of a word. Here's a simple-minded approach which just strips off anything that looks like a suffix:. Although we will ultimately use NLTK's built-in stemmers, it's interesting to see how we can use regular expressions for this task. Our first step is to build up a disjunction of all the suffixes.
We need to enclose it in parentheses in order to limit the scope of the disjunction. Here, re. This is because the parentheses have a second function, to select substrings to be extracted. If we want to use the parentheses to specify the scope of the disjunction, but not to select the material to be output, we have to add? Here's the revised version. However, we'd actually like to split the word into stem and suffix. So we should just parenthesize both parts of the regular expression:.
This looks promising, but still has a problem. Let's look at a different word, processes :. The regular expression incorrectly found an -s suffix instead of an -es suffix. This demonstrates another subtlety: the star operator is "greedy" and the.
This works even when we allow an empty suffix, by making the content of the second parentheses optional:. This approach still has many problems can you spot them? Notice that our regular expression removed the s from ponds but also from is and basis. It produced some non-words like distribut and deriv , but these are acceptable stems in some applications. You can use a special kind of regular expression for searching across multiple words in a text where a text is a list of tokens.
The angle brackets are used to mark token boundaries, and any whitespace between the angle brackets is ignored behaviors that are unique to NLTK's findall method for texts. The second example finds three-word phrases ending with the word bro. The last example finds sequences of three or more words starting with the letter l.
Your Turn: Consolidate your understanding of regular expression patterns and substitutions using nltk. For more practice, try some of the exercises on regular expressions at the end of this chapter. It is easy to build search patterns when the linguistic phenomenon we're studying is tied to particular words. In some cases, a little creativity will go a long way. For instance, searching a large text corpus for expressions of the form x and other ys allows us to discover hypernyms cf 5 :.
With enough text, this approach would give us a useful store of information about the taxonomy of objects, without the need for any manual labor. However, our search results will usually contain false positives, i. For example, the result: demands and other factors suggests that demand is an instance of the type factor , but this sentence is actually about wage demands. Nevertheless, we could construct our own ontology of English concepts by manually correcting the output of such searches.
This combination of automatic and manual processing is the most common way for new corpora to be constructed. We will return to this in Searching corpora also suffers from the problem of false negatives, i.
Student truckers see opportunity in supply chain. Imperiled Afghans worry over long U. London switches on its Christmas lights. Concerns grow over Chinese tennis star Peng Shuai.
Defendant says he perceived Arbery as a threat. More News In Focus. Legal-Courts No immediate ruling is good news, says pro-lifer. Politics-Govt Biden points finger of blame, has 3 pointing back at himself. Culture Lawsuit: Transgender-friendly arrangement a danger to women. Media Media scrubbing Russia-Trump stories but real foe walking free. National Security Steiner: Putin mouthing more than just words about Ukraine.
Education University prez urged to protect Jewish students. Church Bishops labeled as 'politicians who pander to Biden'. Business WH policies bad for humans — but hey, they're robot-friendly. Missions Getting ahead of the 'shadow pandemic'. Sports TX lawmakers ignored unhinged criticism to defend all-girl sports. Sustainable aquafeeds can help fish farmers meet the growing global demand for seafood, while reducing their environmental footprint.
Credit: Getty Images. West Coast. More News. Protected Marine Life The Protected Resources Division works to conserve and recover marine mammals in close coordination with the State of Alaska and other partners. Marine Life News Feature Story. Humpback whales make Juneau, Alaska one of the premier whale watching destinations in the world. Habitat Overview NOAA Fisheries conducts and reviews environmental analyses for a large variety of activities ranging from commercial fishing, to coastal development, to large transportation and energy projects.
Habitat News Feature Story. Hollings Scholar Kaya Mondry spent her summer conducting a literature review on biomedical compounds produced by this species. Lawrence Creek in the Eel River watershed in California. Credit: Trout Unlimited. A pregnant sharpchin rockfish shelters within a sponge on the seafloor off Alaska.
Rockfish and sea urchins congregate around a large red tree coral Primnoa pacifica in the Gulf of Alaska. Habitat includes environmental conditions like water temperatures and salinity and this affects the distribution of fish, crabs, marine mammals and their prey.
Featured Species Alaska's coastal communities depend on healthy marine resources to support commercial and recreational fisheries, tourism, and the Alaskan way of life. View All.
0コメント