305 lines
165 KiB
Plaintext
305 lines
165 KiB
Plaintext
|
{
|
||
|
|
"cells": [
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"# Natural Language Processing LAB"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"attachments": {
|
||
|
|
"image.png": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAq8AAAG6CAYAAAAmkq8oAAAAAXNSR0IArs4c6QAAQABJREFUeAHsnQegFcXVxw+9d1CQIiIoWFBE7CKi2FtM7L0bezQaY2I0iZrEGk3iZ0mU2Av2gr1gJ2JBRRQFKYIU6Uhnv/M7j/Ncrvc97oNX7n3vDOzbvbuzU/4zc+Y/Z87M1krUSbhAIBAIBAKBQCAQCAQCgUCgABCoXQBpjCQGAoFAIBAIBAKBQCAQCAQChkCQ16gIgUAgEAgEAoFAIBAIBAIFg0CQ14IpqkhoIBAIBAKBQCAQCAQCgUCQ16gDgUAgEAgEAoFAIBAIBAIFg0CQ14IpqkhoIBAIBAKBQCAQCAQCgUCQ16gDgUAgEAgEAoFAIBAIBAIFg0CQ14IpqkhoIBAIBAKBQCAQCAQCgUCQ16gDgUAgEAgEAoFAIBAIBAIFg0CQ14IpqkhoIBAIBAKBQCAQCAQCgUCQ16gDgUAgEAgEAoFAIBAIBAIFg0CQ14IpqkhoIBAIBAKBQCAQCAQCgUCQ16gDgUAgEAgEAoFAIBAIBAIFg0CQ14IpqkhoIBAIBAKBQCAQCAQCgUCQ16gDgUAgEAgEAoFAIBAIBAIFg0CQ14IpqkhoIBAIBAKBQCAQCAQCgUCQ16gDgUAgEAgEAoFAIBAIrAaBJEmEA+fn1bwSjysIgSCvFQRsBBsIBAKBQCAQCAQC1QuBWrVqGXHlHK7qEAjyWnXYR8yBQCAQCAQCgUAgUAAILFiwQL7//ntZunSpBHGt+gKrW/VJiBQEAtULAaaTfvjhB1myZIk0atRIGjZsWCEZXLZsmcVD4I0bN5a6daM5VwjQEWiFIZCeei2NEFDXORYuXChz5syRmTNnyty5c2XGjBlGKLjHMW/ePJk/f74sWrTISAbvEG79+vWtfTRp0kSaNWtmR/PmzaVVq1bSunVradmypR385jntlvZUp06dn+SdNBNmrmn/SQDV+MaKFSusDMAGHLPhV2jZJ0+4u+66S+6++265/vrrZZtttpHatfNL90edh2BT15s2bVrtCXb0doXWkiK9eY8Anehtt90mQ4cOlVNOOUWOPPLICknzqFGj5Nprr7VO9MILL5TNN9/c4imNBFRIQiLQQGANEHASCDlIE4FZs2bJ7NmzZfr06TJu3Dj56quv5JtvvrHrMWPGGElt0KCB1KtXzzpqztR5yCbndFhOMtNxcc1BvGjRli9fbsTYCW/79u1l/fXXl06dOknHjh2Lr9dbbz1p0aKFkVyImbczwgonsnjxYqHszj33XPnyyy/lySefNAwdp0LFyMt38uTJ8tFHHxWTc+7nQ96cXD///PNy/vnny2677Wb9T6HinWu6g7zmilT4CwRyRAChhvZn2rRpds7xtTJ7o+Olk6fzpeMIFwgUEgLe8VOHIaUTJ06UL774QkaOHGnkB8JKve7QoYOsu+66piHddttt7Rpi2a5du2KNKTMPaFXROkFs0fgRPkSW9kgHz4EmlhkRwmV2BO0t2lrSMHXqVBk/frwRMDRYpOXdd98VSAttDQ1t586dpXv37tKjRw/ZeOONLW3dunUTCG+aNBdSOZRXWsEbjNGGg5mTqvIKv6rCoS5RhziTJ65xXn+rKl3peKl71Odvv/3W+p30s+p6HeS1upZsDcoX5I1pRDQ166yzjk0tfvbZZ4KmZLPNNrPOis5x7NixNp2I1mSjjTaSLl26WGdHB/bdd99Z50NHSSeIozPjYEqRzpOOEeEFKSU+4qIDxblAo5NDcBMemhyECR0ifpmKxB9hjh492oS8p4VOEc0RHSvvoMEgTjpFn3pDo0u4OKaFJkyYYP54TsePtqhNmzbF/s1j/AkEqggB6jodvLcN7+whkNTjDz74QF5++WV57733ZNKkSUYgaWOQw379+skvf/lLgRjSDqjX1HnaCB01YfmZaw/bz2Q5fe1pcCj47ff82okJZ9oxZABii50j6f3000/l448/NnL9zDPPGAEmTcgA5MmAAQNku+22s+tMzWw6LZ6GtT0jt0gbpg6kl7RBHDGHYFobzDLjRSNKPpAvmDNtscUWsskmmxiW4OD+ye/w4cON3DMY2HDDDc2vy6JsaXd5B2bIZOIBy65du1q4pJEyR1ZRB8CuT58+JqeJF7+cOZB1I0aMsDP49u3bt9hsJFPGER4DHsLsqnFtuummlj/qBwMOzxNpZkDEjBWDE+IHJ/zg0v7AFjyR8wyMttpqq2K5mvZnL6784/ghw8EO+YxDhm+55ZbF8az0bidmFegLqGvUc/JJujws0slz2gTlRbiUDe2AMDfYYAMLh8GXDxhIO3jQdggLR3l8+OGHln/SD068ny5P0ksfSp0AfwZupIv+cPvtt7d6RlieNvzQfukPSRtppP6TtrSjjtLGUegQFm2b/jDTkf633nrL2hV9JeXIANHdT3DXhIQLBAoaAW1EiU7TJ0pUk5NPPjnRaZNEiWuiUyiJCuvk4osvTrRhJUruEm0MiWpt7Py3v/0t0caaKNFNjjvuuGSPPfZIdOol0YaeqABO/vKXv9g7hKkCxDDSBp5ccMEFSf/+/ZNHHnmkGDcVvAkH8Z1xxhlJ27ZtE+2IE22k9lsFZqLaHntnp512SlSgJSpcEp2WTLSRJqRFSavF+49//CPp2bNn8rOf/Sx55513LFzyqLZWlkft1BM1F0i0s0xUGNihgii55ZZb7P3iRMVFIFBFCHh74IzTTjDRAWTywAMPJCeeeKK1Kx2wWRugvf7mN79JHn/8cWtn2okl2hlbe1FCYvW/irJhcSMPlHglam+bKDlMpkyZkqhWNtFp8eTyyy9PDj744KRXr16JEglr1717905+9atfJU888USixMrkieNQHvnwsMBr6623TtQ0yWSeaoMtDUpmk0GDBiVvv/12cdzgeN999yVKWkzmIA91oG5yijyAt5cZck0JjPlTUm7vEOZFF11k/siDpyGdHyWriWrGEyUeiZKiRJUDiZoQGG46OEmOPfbYRAlaQjqRb1xTB/79739bWXuYYLv77rsnSiqLZSSy8PTTT7f7qoSw+CkT6hMyXQmtyXjyd+aZZ5p8PvrooxMlTInXIfoIntM3kAbiR/bee++9iRLO4jzRH+y5556JmohY/KRxn332SQ455JBETVSSF154wcJM551r6okSueSEE06wOMCYQ8m3pYc+gDzij7qEPCcthE96yC/9ytNPP21+CFPJnPUhXs/wT5g6QLHzsGHD8JaoPa6VpxI86w8I73//+589o8+if/P80O+oQiY59NBDE565o3xJ6w033JDooMLqEmVIXLvsskuiShj3mig5TpT4F5cR6SL8s88+O1HCXZx+sCUs6hp1iTPx/9///Z+1KbDAPfjgg4YBWFCexEvf+de//tXKhjLMdLDocIFAQSMAsbv55psTHSFbY4LInXPOOcnDDz+c/POf/7TGADGlgetIMbnmmmusQSH46QAgjfhHKNx+++3WWCChxx9/vAmrnXfeOXnxxRetQerINznooIOSn//854lqBlbBDcFEWhBudGg06JNOOilBwOjo0+4jnIhHbVSTxx57zBonHQWNFdJKJ//555+bsEEgI1DozEknwoK0qC1t8vXXX5t/8opwIE8IRwR6uEAgXxCA6D377LPJeeedZ52TaoKsvh511FHJf/7zn0Q1cUaI6MzpoOjMVINnB9dOaKoyP6SBg8En6XSiQ1sjrXTWkAAGvn/84x+TAw44IFFtpnX6DFT//ve/JxCubB3wmuTLMUG+IWMYJCPfBg8ebDIFUqKaXxvcQjjw/+qrr5pfSDayEnnGYBf5gdx0uYesUe2ZETvSjbx86KGHEvJB2SGzSpIxKAJ0QZPlHSIDOXvjjTcs3+DCQBsCCDkjPVdccYXFA6F0os3g4MADDzQ5jux89NFHk+uuu87qDvlUu34bOJAn0rK+KiRQRvzhD3+wwcSvf/1rCxOSud9++1nZgPGQIUMsTMjqjTfemDz33HPJVVddZX5RJLz00kuWTtVIm9JAtc32Pu9BepG93IMcZiOvXiYoPMgnBBY5zQGZBmP6DZQiOLAB
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"## Useful Ressources\n",
|
||
|
|
"\n",
|
||
|
|
"* Main Book: http://www.nltk.org/book_1ed/\n",
|
||
|
|
"* List of POS tags: https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html\n",
|
||
|
|
"\n",
|
||
|
|
"## Basic Pipeline\n",
|
||
|
|
" "
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"## Main Imports"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 2,
|
||
|
|
"metadata": {
|
||
|
|
"collapsed": true
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"#main API\n",
|
||
|
|
"import nltk\n",
|
||
|
|
"#for a shorter access to sub packages\n",
|
||
|
|
"from nltk import word_tokenize, pos_tag"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"## First Example\n",
|
||
|
|
"corresponding to the WS 17/18 Semester NLP Lab Exercise 1 File"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 10,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"The used sentence is: \n",
|
||
|
|
"Hello, my name is Carsten. I'm a 23 year old Student at the University of Bonn. I'm studying ComputerScience in the third Semester and doiung my NLP Lab this Semester.\n",
|
||
|
|
"tokenize results: \n",
|
||
|
|
"\n",
|
||
|
|
"['Hello', ',', 'my', 'name', 'is', 'Carsten', '.', 'I', \"'m\", 'a', '23', 'year', 'old', 'Student', 'at', 'the', 'University', 'of', 'Bonn', '.', 'I', \"'m\", 'studying', 'ComputerScience', 'in', 'the', 'third', 'Semester', 'and', 'doiung', 'my', 'NLP', 'Lab', 'this', 'Semester', '.']\n",
|
||
|
|
"the pos tags are: \n",
|
||
|
|
"\n",
|
||
|
|
"[('Hello', 'NNP'), (',', ','), ('my', 'PRP$'), ('name', 'NN'), ('is', 'VBZ'), ('Carsten', 'NNP'), ('.', '.'), ('I', 'PRP'), (\"'m\", 'VBP'), ('a', 'DT'), ('23', 'CD'), ('year', 'NN'), ('old', 'JJ'), ('Student', 'NN'), ('at', 'IN'), ('the', 'DT'), ('University', 'NNP'), ('of', 'IN'), ('Bonn', 'NNP'), ('.', '.'), ('I', 'PRP'), (\"'m\", 'VBP'), ('studying', 'VBG'), ('ComputerScience', 'NN'), ('in', 'IN'), ('the', 'DT'), ('third', 'JJ'), ('Semester', 'NNP'), ('and', 'CC'), ('doiung', 'VB'), ('my', 'PRP$'), ('NLP', 'NNP'), ('Lab', 'NNP'), ('this', 'DT'), ('Semester', 'NNP'), ('.', '.')]\n"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"#An example Sentence to play with\n",
|
||
|
|
"sentence = \"Hello, my name is Carsten. I'm a 23 year old Student at the University of Bonn. I'm studying ComputerScience in the third Semester and doiung my NLP Lab this Semester.\"\n",
|
||
|
|
"print(\"The used sentence is: \\n\" +sentence)\n",
|
||
|
|
"\n",
|
||
|
|
"#tonkenize splits the sentences into its tokens (meaningful parts)\n",
|
||
|
|
"tokens = word_tokenize(sentence)\n",
|
||
|
|
"print(\"tokenize results: \\n\")\n",
|
||
|
|
"print(tokens)\n",
|
||
|
|
"\n",
|
||
|
|
"#POS Tags are an identifier which class of word the token is in it's sentence\n",
|
||
|
|
"pos_tags = pos_tag(tokens)\n",
|
||
|
|
"print(\"the pos tags are: \\n\")\n",
|
||
|
|
"print(pos_tags)\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"## Playing around with Penn TreeBank (PTB) Corpus\n",
|
||
|
|
"corresponding to the WS 17/18 Semester NLP Lab Exercise 1 File"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 11,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"[nltk_data] Downloading package treebank to\n",
|
||
|
|
"[nltk_data] /Users/Carsten/nltk_data...\n",
|
||
|
|
"[nltk_data] Unzipping corpora/treebank.zip.\n",
|
||
|
|
"[('Pierre', 'NNP'), ('Vinken', 'NNP'), (',', ','), ('61', 'CD'), ('years', 'NNS'), ('old', 'JJ'), (',', ','), ('will', 'MD'), ('join', 'VB'), ('the', 'DT'), ('board', 'NN'), ('as', 'IN'), ('a', 'DT'), ('nonexecutive', 'JJ'), ('director', 'NN'), ('Nov.', 'NNP'), ('29', 'CD'), ('.', '.')]\n",
|
||
|
|
"Tagged sentences: 3914\n",
|
||
|
|
"Tagged words: 100676\n",
|
||
|
|
"[nltk_data] Downloading package tagsets to /Users/Carsten/nltk_data...\n",
|
||
|
|
"[nltk_data] Unzipping help/tagsets.zip.\n",
|
||
|
|
"$: dollar\n",
|
||
|
|
" $ -$ --$ A$ C$ HK$ M$ NZ$ S$ U.S.$ US$\n",
|
||
|
|
"'': closing quotation mark\n",
|
||
|
|
" ' ''\n",
|
||
|
|
"(: opening parenthesis\n",
|
||
|
|
" ( [ {\n",
|
||
|
|
"): closing parenthesis\n",
|
||
|
|
" ) ] }\n",
|
||
|
|
",: comma\n",
|
||
|
|
" ,\n",
|
||
|
|
"--: dash\n",
|
||
|
|
" --\n",
|
||
|
|
".: sentence terminator\n",
|
||
|
|
" . ! ?\n",
|
||
|
|
":: colon or ellipsis\n",
|
||
|
|
" : ; ...\n",
|
||
|
|
"CC: conjunction, coordinating\n",
|
||
|
|
" & 'n and both but either et for less minus neither nor or plus so\n",
|
||
|
|
" therefore times v. versus vs. whether yet\n",
|
||
|
|
"CD: numeral, cardinal\n",
|
||
|
|
" mid-1890 nine-thirty forty-two one-tenth ten million 0.5 one forty-\n",
|
||
|
|
" seven 1987 twenty '79 zero two 78-degrees eighty-four IX '60s .025\n",
|
||
|
|
" fifteen 271,124 dozen quintillion DM2,000 ...\n",
|
||
|
|
"DT: determiner\n",
|
||
|
|
" all an another any both del each either every half la many much nary\n",
|
||
|
|
" neither no some such that the them these this those\n",
|
||
|
|
"EX: existential there\n",
|
||
|
|
" there\n",
|
||
|
|
"FW: foreign word\n",
|
||
|
|
" gemeinschaft hund ich jeux habeas Haementeria Herr K'ang-si vous\n",
|
||
|
|
" lutihaw alai je jour objets salutaris fille quibusdam pas trop Monte\n",
|
||
|
|
" terram fiche oui corporis ...\n",
|
||
|
|
"IN: preposition or conjunction, subordinating\n",
|
||
|
|
" astride among uppon whether out inside pro despite on by throughout\n",
|
||
|
|
" below within for towards near behind atop around if like until below\n",
|
||
|
|
" next into if beside ...\n",
|
||
|
|
"JJ: adjective or numeral, ordinal\n",
|
||
|
|
" third ill-mannered pre-war regrettable oiled calamitous first separable\n",
|
||
|
|
" ectoplasmic battery-powered participatory fourth still-to-be-named\n",
|
||
|
|
" multilingual multi-disciplinary ...\n",
|
||
|
|
"JJR: adjective, comparative\n",
|
||
|
|
" bleaker braver breezier briefer brighter brisker broader bumper busier\n",
|
||
|
|
" calmer cheaper choosier cleaner clearer closer colder commoner costlier\n",
|
||
|
|
" cozier creamier crunchier cuter ...\n",
|
||
|
|
"JJS: adjective, superlative\n",
|
||
|
|
" calmest cheapest choicest classiest cleanest clearest closest commonest\n",
|
||
|
|
" corniest costliest crassest creepiest crudest cutest darkest deadliest\n",
|
||
|
|
" dearest deepest densest dinkiest ...\n",
|
||
|
|
"LS: list item marker\n",
|
||
|
|
" A A. B B. C C. D E F First G H I J K One SP-44001 SP-44002 SP-44005\n",
|
||
|
|
" SP-44007 Second Third Three Two * a b c d first five four one six three\n",
|
||
|
|
" two\n",
|
||
|
|
"MD: modal auxiliary\n",
|
||
|
|
" can cannot could couldn't dare may might must need ought shall should\n",
|
||
|
|
" shouldn't will would\n",
|
||
|
|
"NN: noun, common, singular or mass\n",
|
||
|
|
" common-carrier cabbage knuckle-duster Casino afghan shed thermostat\n",
|
||
|
|
" investment slide humour falloff slick wind hyena override subhumanity\n",
|
||
|
|
" machinist ...\n",
|
||
|
|
"NNP: noun, proper, singular\n",
|
||
|
|
" Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
|
||
|
|
" Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
|
||
|
|
" Shannon A.K.C. Meltex Liverpool ...\n",
|
||
|
|
"NNPS: noun, proper, plural\n",
|
||
|
|
" Americans Americas Amharas Amityvilles Amusements Anarcho-Syndicalists\n",
|
||
|
|
" Andalusians Andes Andruses Angels Animals Anthony Antilles Antiques\n",
|
||
|
|
" Apache Apaches Apocrypha ...\n",
|
||
|
|
"NNS: noun, common, plural\n",
|
||
|
|
" undergraduates scotches bric-a-brac products bodyguards facets coasts\n",
|
||
|
|
" divestitures storehouses designs clubs fragrances averages\n",
|
||
|
|
" subjectivists apprehensions muses factory-jobs ...\n",
|
||
|
|
"PDT: pre-determiner\n",
|
||
|
|
" all both half many quite such sure this\n",
|
||
|
|
"POS: genitive marker\n",
|
||
|
|
" ' 's\n",
|
||
|
|
"PRP: pronoun, personal\n",
|
||
|
|
" hers herself him himself hisself it itself me myself one oneself ours\n",
|
||
|
|
" ourselves ownself self she thee theirs them themselves they thou thy us\n",
|
||
|
|
"PRP$: pronoun, possessive\n",
|
||
|
|
" her his mine my our ours their thy your\n",
|
||
|
|
"RB: adverb\n",
|
||
|
|
" occasionally unabatingly maddeningly adventurously professedly\n",
|
||
|
|
" stirringly prominently technologically magisterially predominately\n",
|
||
|
|
" swiftly fiscally pitilessly ...\n",
|
||
|
|
"RBR: adverb, comparative\n",
|
||
|
|
" further gloomier grander graver greater grimmer harder harsher\n",
|
||
|
|
" healthier heavier higher however larger later leaner lengthier less-\n",
|
||
|
|
" perfectly lesser lonelier longer louder lower more ...\n",
|
||
|
|
"RBS: adverb, superlative\n",
|
||
|
|
" best biggest bluntest earliest farthest first furthest hardest\n",
|
||
|
|
" heartiest highest largest least less most nearest second tightest worst\n",
|
||
|
|
"RP: particle\n",
|
||
|
|
" aboard about across along apart around aside at away back before behind\n",
|
||
|
|
" by crop down ever fast for forth from go high i.e. in into just later\n",
|
||
|
|
" low more off on open out over per pie raising start teeth that through\n",
|
||
|
|
" under unto up up-pp upon whole with you\n",
|
||
|
|
"SYM: symbol\n",
|
||
|
|
" % & ' '' ''. ) ). * + ,. < = > @ A[fj] U.S U.S.S.R * ** ***\n",
|
||
|
|
"TO: \"to\" as preposition or infinitive marker\n",
|
||
|
|
" to\n",
|
||
|
|
"UH: interjection\n",
|
||
|
|
" Goodbye Goody Gosh Wow Jeepers Jee-sus Hubba Hey Kee-reist Oops amen\n",
|
||
|
|
" huh howdy uh dammit whammo shucks heck anyways whodunnit honey golly\n",
|
||
|
|
" man baby diddle hush sonuvabitch ...\n",
|
||
|
|
"VB: verb, base form\n",
|
||
|
|
" ask assemble assess assign assume atone attention avoid bake balkanize\n",
|
||
|
|
" bank begin behold believe bend benefit bevel beware bless boil bomb\n",
|
||
|
|
" boost brace break bring broil brush build ...\n",
|
||
|
|
"VBD: verb, past tense\n",
|
||
|
|
" dipped pleaded swiped regummed soaked tidied convened halted registered\n",
|
||
|
|
" cushioned exacted snubbed strode aimed adopted belied figgered\n",
|
||
|
|
" speculated wore appreciated contemplated ...\n",
|
||
|
|
"VBG: verb, present participle or gerund\n",
|
||
|
|
" telegraphing stirring focusing angering judging stalling lactating\n",
|
||
|
|
" hankerin' alleging veering capping approaching traveling besieging\n",
|
||
|
|
" encrypting interrupting erasing wincing ...\n",
|
||
|
|
"VBN: verb, past participle\n",
|
||
|
|
" multihulled dilapidated aerosolized chaired languished panelized used\n",
|
||
|
|
" experimented flourished imitated reunifed factored condensed sheared\n",
|
||
|
|
" unsettled primed dubbed desired ...\n",
|
||
|
|
"VBP: verb, present tense, not 3rd person singular\n",
|
||
|
|
" predominate wrap resort sue twist spill cure lengthen brush terminate\n",
|
||
|
|
" appear tend stray glisten obtain comprise detest tease attract\n",
|
||
|
|
" emphasize mold postpone sever return wag ...\n",
|
||
|
|
"VBZ: verb, present tense, 3rd person singular\n",
|
||
|
|
" bases reconstructs marks mixes displeases seals carps weaves snatches\n",
|
||
|
|
" slumps stretches authorizes smolders pictures emerges stockpiles\n",
|
||
|
|
" seduces fizzes uses bolsters slaps speaks pleads ...\n",
|
||
|
|
"WDT: WH-determiner\n",
|
||
|
|
" that what whatever which whichever\n",
|
||
|
|
"WP: WH-pronoun\n",
|
||
|
|
" that what whatever whatsoever which who whom whosoever\n",
|
||
|
|
"WP$: WH-pronoun, possessive\n",
|
||
|
|
" whose\n",
|
||
|
|
"WRB: Wh-adverb\n",
|
||
|
|
" how however whence whenever where whereby whereever wherein whereof why\n",
|
||
|
|
"``: opening quotation mark\n",
|
||
|
|
" ` ``\n",
|
||
|
|
"None\n"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"nltk.download('treebank')\n",
|
||
|
|
"annotated_sent = nltk.corpus.treebank.tagged_sents()\n",
|
||
|
|
" \n",
|
||
|
|
"print(annotated_sent[0])\n",
|
||
|
|
"print(\"Tagged sentences: \", len(annotated_sent))\n",
|
||
|
|
"print(\"Tagged words:\", len(nltk.corpus.treebank.tagged_words()))\n",
|
||
|
|
"\n",
|
||
|
|
"# tagsets\n",
|
||
|
|
"nltk.download('tagsets')\n",
|
||
|
|
"print(nltk.help.upenn_tagset())"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {
|
||
|
|
"collapsed": true
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": []
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"metadata": {
|
||
|
|
"kernelspec": {
|
||
|
|
"display_name": "Python 3",
|
||
|
|
"language": "python",
|
||
|
|
"name": "python3"
|
||
|
|
},
|
||
|
|
"language_info": {
|
||
|
|
"codemirror_mode": {
|
||
|
|
"name": "ipython",
|
||
|
|
"version": 3
|
||
|
|
},
|
||
|
|
"file_extension": ".py",
|
||
|
|
"mimetype": "text/x-python",
|
||
|
|
"name": "python",
|
||
|
|
"nbconvert_exporter": "python",
|
||
|
|
"pygments_lexer": "ipython3",
|
||
|
|
"version": "3.6.3"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"nbformat": 4,
|
||
|
|
"nbformat_minor": 2
|
||
|
|
}
|