nlp-lab/Carsten_Solutions/Exercise 1.ipynb
2018-04-29 20:22:39 +02:00

658 lines
16 KiB
Plaintext

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 1"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import nltk\n",
"from nltk import word_tokenize, pos_tag"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Classifiers\n",
"note: for model1 and model3 you can try different classifiers: Hidden Markov Model, Logistic Regression, Maximum Entropy Markov Models, Decision Trees, Naive Bayes, etc.. __choose one!__"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from sklearn.tree import DecisionTreeClassifier\n",
"from sklearn.feature_extraction import DictVectorizer\n",
"from sklearn.pipeline import Pipeline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1. model1 = your POS tagger model (english)"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'word': 'bims', 'length': 4, 'is_capitalized': False, 'prefix-1': 'b', 'suffix-1': 's', 'prev_word': 'i', 'next_word': 'der'}\n"
]
}
],
"source": [
"def features(sentence, index):\n",
" return {\n",
" 'word': sentence[index],\n",
" 'length': len(sentence[index]),\n",
" 'is_capitalized': sentence[index][0].upper() == sentence[index][0],\n",
" 'prefix-1': sentence[index][0],\n",
" 'suffix-1': sentence[index][-1],\n",
" 'prev_word': '' if index == 0 else sentence[index - 1],\n",
" 'next_word': '' if index == len(sentence) - 1 else sentence[index + 1]\n",
" }\n",
"\n",
"print(features(\"halli hallo i bims der Programmierer\".strip().split(\" \"), 3))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2. model2 = pre-trained POS tagger model using NLTK (maxentropy english)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3. model3.x = rule-based classifiers (x = 1 to 5)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4. model4 = your POS tagger model (not english)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5. model5 = pre-trained POS tagger model using RDRPOSTagger 1 or TreeTagger 2 (not english)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Corpora\n",
"note: data split for training/test = 0.8/0.2 (sequencial)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 1. X1 = nltk.corpus.treebank (english)"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[nltk_data] Downloading package treebank to\n",
"[nltk_data] /Users/Carsten/nltk_data...\n",
"[nltk_data] Package treebank is already up-to-date!\n"
]
}
],
"source": [
"nltk.download('treebank')\n",
"x1 = nltk.corpus.treebank"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 2. X2 = nltk.corpus.brown (english)"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[nltk_data] Downloading package brown to /Users/Carsten/nltk_data...\n",
"[nltk_data] Package brown is already up-to-date!\n"
]
}
],
"source": [
"nltk.download('brown')\n",
"x2 = nltk.corpus.brown"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 3. X3 = other language (not english)"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#nltk.download('brown')\n",
"#x3 = other language"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Task 1\n",
"* get results for english (plot a graph with all classifiers x results)\n",
" * performance 1.1 = model1 in X1\n",
" * performance 1.2 = model2 in X1\n",
" * performance 1.3.x = model3.x in X1\n",
" * performance 1.4 = model1 in X2\n",
" * performance 1.5 = model2 in X2\n",
" * performance 1.6.x = model3.x in X2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### Generate Training and Testdata for X1\n",
"1. split annotaed sentences into training and testdata\n",
"2. split trainingdata into input data and teacherdata\n",
" *input is the feature vector of each word\n",
" *output is a list of POS tags for each word and sentences"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#to generate trainingsdata, delete the assigned tags as a function\n",
"def untag(tagged_sentence):\n",
" return [w for w, t in tagged_sentence]"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"got 3131 training sentences and 783 test sentences\n"
]
}
],
"source": [
"#object including the annotated sentences\n",
"annotated_sent = nltk.corpus.treebank.tagged_sents()\n",
"\n",
"#to split the data, calculate the borders for ratio\n",
"cutoff = int(.8 * len(annotated_sent))\n",
"training_sentences_X1 = annotated_sent[:cutoff]\n",
"test_sentences_X1 = annotated_sent[cutoff:]\n",
"\n",
"#show the amount of sentences\n",
"print(\"got \",len(training_sentences_X1),\" training sentences and \", len(test_sentences_X1), \" test sentences\")"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#for training split sentences with its tags into y (for a sentences its resulting tags for each word) and transform sentences and x as a list of the features extracet for echt word in the sentences\n",
"def transform_to_dataset(tagged_sentences):\n",
" X, y = [], []\n",
" for tagged_sentence in tagged_sentences:\n",
" for index in range(len(tagged_sentence)):\n",
" X.append(features(untag(tagged_sentence), index))\n",
" y.append(tagged_sentence[index][1]) \n",
" return X, y"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [],
"source": [
"#trainings inputset X and training teacher set y\n",
"X1, y1 = transform_to_dataset(training_sentences_X1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### Generate Training and Testdata for X2\n",
"1. split annotaed sentences into training and testdata\n",
"2. split trainingdata into input data and teacherdata\n",
" *input is the feature vector of each word\n",
" *output is a list of POS tags for each word and sentences"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"got 45872 training sentences and 11468 test sentences\n"
]
}
],
"source": [
"#object including the annotated sentences\n",
"annotated_sent = nltk.corpus.brown.tagged_sents()\n",
"\n",
"#to split the data, calculate the borders for ratio\n",
"cutoff = int(.8 * len(annotated_sent))\n",
"training_sentences_X2 = annotated_sent[:cutoff]\n",
"test_sentences_X2 = annotated_sent[cutoff:]\n",
"\n",
"#show the amount of sentences\n",
"print(\"got \",len(training_sentences_X2),\" training sentences and \", len(test_sentences_X2), \" test sentences\")"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#trainings inputset X and training teacher set y\n",
"X2, y2 = transform_to_dataset(training_sentences_X2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"#### Implementing a classifier\n",
"relevant imports\n",
"* decision tree as the AI for classfing\n",
"* dict vercorizer transforms the feature dictionary into a vector as the input for the tree"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from sklearn.tree import DecisionTreeClassifier\n",
"from sklearn.feature_extraction import DictVectorizer\n",
"from sklearn.pipeline import Pipeline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Pipeline manages vectorizer and classifier"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"clf = Pipeline([\n",
" ('vectorizer', DictVectorizer(sparse=False)),\n",
" ('classifier', DecisionTreeClassifier(criterion='entropy'))\n",
"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### Calculate performance 1.1 \n",
"* fit the decision tree for a limited amount (size) of training \n",
"* test data and compare with score function on testdata"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"training OK\n",
"Accuracy: 0.87983432307\n"
]
}
],
"source": [
"size=10000\n",
"clf.fit(X[:size], y[:size])\n",
" \n",
"print('training OK')\n",
" \n",
"X1_test, y1_test = transform_to_dataset(test_sentences_X1)\n",
"\n",
"performance1_1 = clf.score(X1_test, y1_test)\n",
"\n",
"print(\"Accuracy:\", performance1_1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### Calculate other performances"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"calculated perfomance 1.4= 0.756485959481\n"
]
}
],
"source": [
"performance1_2 = 0\n",
"\n",
"performance1_3 = 0\n",
"\n",
"# performance1_4\n",
"size=10000\n",
"clf.fit(X2[:size], y2[:size])\n",
"X2_test, y2_test = transform_to_dataset(test_sentences_X2)\n",
"performance1_4 = clf.score(X2_test, y2_test)\n",
"print(\"calculated perfomance 1.4= \",performance1_4)\n",
"\n",
"performance1_5 = 0\n",
"\n",
"performance1_6 = 0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Using the classifier\n",
"for results the link of pos_tags:\n",
"https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3.6.3\n",
"checking...\n",
"[('Hello', 'CS'), ('world', 'NN'), (',', ','), ('lets', 'NNS'), ('do', 'DO'), ('something', 'PN'), ('awesome', 'NN'), ('today', 'NR'), ('!', 'CD')]\n"
]
}
],
"source": [
"def pos_tag(sentence):\n",
" print('checking...')\n",
" tagged_sentence = []\n",
" tags = clf.predict([features(sentence, index) for index in range(len(sentence))])\n",
" return zip(sentence, tags)\n",
"\n",
"import platform\n",
"print(platform.python_version())\n",
"\n",
"print(list(pos_tag(word_tokenize('Hello world, lets do something awesome today!'))))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Results for Task 1\n",
"* get results for english (plot a graph with all classifiers x results)\n",
" * performance 1.1 = model1 in X1\n",
" * performance 1.2 = model2 in X1\n",
" * performance 1.3.x = model3.x in X1\n",
" * performance 1.4 = model1 in X2\n",
" * performance 1.5 = model2 in X2\n",
" * performance 1.6.x = model3.x in X2"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"High five! You successfully sent some data to your account on plotly. View your plot in your browser at https://plot.ly/~carsten95/0 or inside your plot.ly account where it is named 'basic-bar'\n"
]
},
{
"data": {
"text/html": [
"<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~carsten95/0.embed\" height=\"525px\" width=\"100%\"></iframe>"
],
"text/plain": [
"<plotly.tools.PlotlyDisplay object>"
]
},
"execution_count": 57,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import plotly\n",
"plotly.tools.set_credentials_file(username='carsten95', api_key='vElf5IOxiFheQdjTxjXW')\n",
"plotly.__version__\n",
"import plotly.plotly as py\n",
"import plotly.graph_objs as go\n",
"\n",
"data = [go.Bar(\n",
" x=['performance 1.1', 'performance 1.2', 'performance 1.3', 'performance 1.4', 'performance 1.5' , 'performance 1.6'],\n",
" y=[performance1_1, performance1_2, performance1_3, performance1_4, performance1_5, performance1_6]\n",
" )]\n",
"\n",
"py.iplot(data, filename='basic-bar')"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Results for Task 2\n",
"* train your model with standard features (plot a graph with all classifiers x results)\n",
" * performance 2.1 = model4 in X3\n",
" * performance 2.2 = model5 in X3"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"High five! You successfully sent some data to your account on plotly. View your plot in your browser at https://plot.ly/~carsten95/0 or inside your plot.ly account where it is named 'basic-bar'\n"
]
},
{
"data": {
"text/html": [
"<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~carsten95/0.embed\" height=\"525px\" width=\"100%\"></iframe>"
],
"text/plain": [
"<plotly.tools.PlotlyDisplay object>"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"performance2_1 = 0\n",
"performance2_2 = 0\n",
"\n",
"data = [go.Bar(\n",
" x=['performance 2.1', 'performance 2.2'],\n",
" y=[performance2_1, performance2_2]\n",
" )]\n",
"\n",
"py.iplot(data, filename='basic-bar')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}