2018-05-13 19:40:03 +02:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"import all usefull tool"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Populating the interactive namespace from numpy and matplotlib\n"
]
}
],
"source": [
"%pylab inline\n",
"\n",
"import pandas as pd\n",
"import numpy as np\n",
"import itertools\n",
"from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, HashingVectorizer\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.linear_model import PassiveAggressiveClassifier\n",
"from sklearn.naive_bayes import MultinomialNB\n",
2018-05-15 10:28:16 +02:00
"from sklearn.neural_network import MLPClassifier\n",
2018-05-13 19:40:03 +02:00
"from sklearn import metrics\n",
2018-05-31 11:49:28 +02:00
"import matplotlib.pyplot as plt\n",
"\n",
"#https://github.com/kjam/random_hackery/blob/master/Attempting%20to%20detect%20fake%20news.ipynb"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Read in Datasets"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load Dataset 1 - fake_or_real_news.csv\n",
"Read in File fake_or_real_news.csv"
]
},
{
"cell_type": "code",
"execution_count": 2,
2018-05-31 11:49:28 +02:00
"metadata": {
"collapsed": true
},
2018-05-13 19:40:03 +02:00
"outputs": [],
"source": [
"df = pd.read_csv('/Users/Carsten/GitRepos/NLP-LAB/Carsten_Solutions/sets/fact checking/fake_or_real_news.csv')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"ignores first column"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"df = df.set_index('Unnamed: 0')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"quick view at the data"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style>\n",
" .dataframe thead tr:only-child th {\n",
" text-align: right;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>label</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Unnamed: 0</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>8476</th>\n",
" <td>You Can Smell Hillary’
" <td>Daniel Greenfield, a Shillman Journalism Fello...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10294</th>\n",
" <td>Watch The Exact Moment Paul Ryan Committed Pol...</td>\n",
" <td>Google Pinterest Digg Linkedin Reddit Stumbleu...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3608</th>\n",
" <td>Kerry to go to Paris in gesture of sympathy</td>\n",
" <td>U.S. Secretary of State John F. Kerry said Mon...</td>\n",
" <td>REAL</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10142</th>\n",
" <td>Bernie supporters on Twitter erupt in anger ag...</td>\n",
" <td>— Kaydee King (@KaydeeKing) November 9, 2016 T...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>875</th>\n",
" <td>The Battle of New York: Why This Primary Matters</td>\n",
" <td>It's primary day in New York and front-runners...</td>\n",
" <td>REAL</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" title \\\n",
"Unnamed: 0 \n",
"8476 You Can Smell Hillary’
"10294 Watch The Exact Moment Paul Ryan Committed Pol... \n",
"3608 Kerry to go to Paris in gesture of sympathy \n",
"10142 Bernie supporters on Twitter erupt in anger ag... \n",
"875 The Battle of New York: Why This Primary Matters \n",
"\n",
" text label \n",
"Unnamed: 0 \n",
"8476 Daniel Greenfield, a Shillman Journalism Fello... FAKE \n",
"10294 Google Pinterest Digg Linkedin Reddit Stumbleu... FAKE \n",
"3608 U.S. Secretary of State John F. Kerry said Mon... REAL \n",
"10142 — Kaydee King (@KaydeeKing) November 9, 2016 T... FAKE \n",
"875 It's primary day in New York and front-runners... REAL "
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2018-05-15 10:28:16 +02:00
"store label column with the classification of each text"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
"execution_count": 5,
2018-05-31 11:49:28 +02:00
"metadata": {
"collapsed": true
},
2018-05-13 19:40:03 +02:00
"outputs": [],
"source": [
"#store label before dropping it\n",
"bin_y = df.label\n",
"#y.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"cut of label column to get an unlabled array"
]
},
{
"cell_type": "code",
"execution_count": 6,
2018-05-31 11:49:28 +02:00
"metadata": {
"collapsed": true
},
2018-05-13 19:40:03 +02:00
"outputs": [],
"source": [
"df = df.drop('label', axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"from skikit learn the function: train_test_split\n",
"* in the dataframe get text column by df['text']\n",
"* use stored y label df\n",
2018-05-15 10:28:16 +02:00
"* use seed 4222\n",
"* determine split size: in this case 0.25\n",
"* **take care that these config is the same to get compareable results**"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
"execution_count": 7,
2018-05-31 11:49:28 +02:00
"metadata": {
"collapsed": true
},
2018-05-13 19:40:03 +02:00
"outputs": [],
"source": [
"bin_X_train, bin_X_test, bin_y_train, bin_y_test = train_test_split(df['text'], bin_y, test_size=0.25, random_state=4222)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load Dataset 2 - liar_dataset.zip\n",
2018-05-15 10:28:16 +02:00
"* Read in File liar_dataset.zip\n",
"* cause of three file read in each file on its own and assign it to train test vaidation \n",
"\n",
"#### So first train set\n",
"\n",
"* only use the text part and ignore the others\n",
"* it would also be possible to for example to concatiate the other information ito the text part but youll lose the robustness againstother datasets without these coulums and you'll mix up information"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
"execution_count": 8,
2018-05-31 11:49:28 +02:00
"metadata": {
"collapsed": true
},
2018-05-13 19:40:03 +02:00
"outputs": [],
"source": [
"#training data file\n",
"df = pd.read_csv('/Users/Carsten/GitRepos/NLP-LAB/Carsten_Solutions/sets/fact checking/train.tsv', delimiter=\"\\t\", header=None, usecols=[1,2], names=['y', 'claim'])"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style>\n",
" .dataframe thead tr:only-child th {\n",
" text-align: right;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>y</th>\n",
" <th>claim</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>false</td>\n",
" <td>Says the Annies List political group supports ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>half-true</td>\n",
" <td>When did the decline of coal start? It started...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>mostly-true</td>\n",
" <td>Hillary Clinton agrees with John McCain \"by vo...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>false</td>\n",
" <td>Health care reform legislation is likely to ma...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>half-true</td>\n",
" <td>The economic turnaround started at the end of ...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" y claim\n",
"0 false Says the Annies List political group supports ...\n",
"1 half-true When did the decline of coal start? It started...\n",
"2 mostly-true Hillary Clinton agrees with John McCain \"by vo...\n",
"3 false Health care reform legislation is likely to ma...\n",
"4 half-true The economic turnaround started at the end of ..."
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"mul_X_train = df.claim\n",
"mul_y_train = df.y"
]
},
2018-05-15 10:28:16 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### second dataset: test set"
]
},
2018-05-13 19:40:03 +02:00
{
"cell_type": "code",
"execution_count": 11,
2018-05-31 11:49:28 +02:00
"metadata": {
"collapsed": true
},
2018-05-13 19:40:03 +02:00
"outputs": [],
"source": [
"#test data file\n",
"df = pd.read_csv('/Users/Carsten/GitRepos/NLP-LAB/Carsten_Solutions/sets/fact checking/test.tsv', delimiter=\"\\t\", header=None, usecols=[1,2], names=['y', 'claim'])"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style>\n",
" .dataframe thead tr:only-child th {\n",
" text-align: right;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>y</th>\n",
" <th>claim</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>true</td>\n",
" <td>Building a wall on the U.S.-Mexico border will...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>false</td>\n",
" <td>Wisconsin is on pace to double the number of l...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>false</td>\n",
" <td>Says John McCain has done nothing to help the ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>half-true</td>\n",
" <td>Suzanne Bonamici supports a plan that will cut...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>pants-fire</td>\n",
" <td>When asked by a reporter whether hes at the ce...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" y claim\n",
"0 true Building a wall on the U.S.-Mexico border will...\n",
"1 false Wisconsin is on pace to double the number of l...\n",
"2 false Says John McCain has done nothing to help the ...\n",
"3 half-true Suzanne Bonamici supports a plan that will cut...\n",
"4 pants-fire When asked by a reporter whether hes at the ce..."
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"mul_X_test = df.claim\n",
"mul_y_test = df.y"
]
},
2018-05-15 10:28:16 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### as third dataset: validation set"
]
},
2018-05-13 19:40:03 +02:00
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#test data file\n",
"df = pd.read_csv('/Users/Carsten/GitRepos/NLP-LAB/Carsten_Solutions/sets/fact checking/valid.tsv', delimiter=\"\\t\", header=None, usecols=[1,2], names=['y', 'claim'])"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"mul_X_valid = df.claim\n",
"mul_y_valid = df.y"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.series.Series'>\n"
]
}
],
"source": [
"print(type(mul_X_valid))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Generate Dataset 3\n",
"* using code from Diego by copy paste with some small modifications\n",
"* thanks for distributing this code @diego ;)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"-- fake news\n",
"Index(['y', 'claim'], dtype='object')\n",
"3171\n",
"3164\n",
"6335\n",
"-- liar liar\n",
"Index(['y', 'claim'], dtype='object')\n",
2018-05-31 11:49:28 +02:00
"{'true', 'pants-fire', 'false', 'barely-true', 'mostly-true', 'half-true'} 10240\n",
2018-05-13 19:40:03 +02:00
"1676\n",
"1995\n",
"{'true', 'false'} 3671\n",
"false 5159\n",
"true 4847\n",
"Name: y, dtype: int64\n",
"done\n"
]
}
],
"source": [
"'''import random\n",
"import sys\n",
"import pandas as pd\n",
"import numpy as np\n",
"from sklearn.cross_validation import train_test_split\n",
"\n",
"ds1 = sys.argv[1]\n",
"ds2 = sys.argv[2]'''\n",
"\n",
"try:\n",
" print('-- fake news')\n",
" df1 = pd.read_csv('/Users/Carsten/GitRepos/NLP-LAB/Carsten_Solutions/sets/fact checking/fake_or_real_news.csv', sep=',', usecols=['title','text','label'])\n",
" df1['claim'] = df1[['title', 'text']].apply(lambda x: '. '.join(x), axis=1)\n",
" del df1['title']\n",
" del df1['text']\n",
" df1.rename(index=str, columns={'label': 'y'}, inplace=True)\n",
" print(df1.keys())\n",
" print(len(df1[df1['y']=='REAL']))\n",
" print(len(df1[df1['y']=='FAKE']))\n",
" df1['y'] = np.where(df1['y'] == 'FAKE', 'false', 'true')\n",
" print(len(df1))\n",
"\n",
" print('-- liar liar')\n",
" df2 = pd.read_csv('/Users/Carsten/GitRepos/NLP-LAB/Carsten_Solutions/sets/fact checking/train.tsv', sep='\\t', header=None, usecols=[1,2], names=['y', 'claim'])\n",
" print(df2.keys())\n",
" print(set(df2.y), len(df2))\n",
" print(len(df2[df2['y'] == 'true']))\n",
" print(len(df2[df2['y'] == 'false']))\n",
" df2=df2[(df2['y'] == 'true') | (df2['y'] == 'false')]\n",
" print(set(df2.y), len(df2))\n",
"\n",
" df3=pd.concat([df1, df2], ignore_index=True)\n",
"\n",
" print(df3['y'].value_counts())\n",
" print('done')\n",
" concat_X_train, concat_X_test, concat_y_train, concat_y_test = train_test_split(df3['claim'], df3['y'], test_size=0.25, random_state=4222)\n",
" \n",
" \n",
"except Exception as e:\n",
" print(e)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Vectorizer Classifiers\n",
2018-05-15 10:28:16 +02:00
"* tfids removes words in pregenerating the vectors by evaluating if this word appears more than 70% often in all articles (tfidf)\n",
2018-05-13 19:40:03 +02:00
"* an immense naive approach would be to store a set of the over all occuring word in all the texts and for each text determining how often this word occurs.\n",
2018-05-15 10:28:16 +02:00
"* also testing some min df thresholds as lower bound for regarded occurences of words over all texts ... not easy to predict if this is a good idea"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Generate Vectorizer on Binary Classes"
]
},
2018-05-15 10:28:16 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"generate two different vectorizers"
]
},
2018-05-13 19:40:03 +02:00
{
"cell_type": "code",
"execution_count": 18,
2018-05-31 11:49:28 +02:00
"metadata": {
"collapsed": true
},
2018-05-13 19:40:03 +02:00
"outputs": [],
"source": [
2018-05-15 10:28:16 +02:00
"#building the vectorspace\n",
2018-05-13 19:40:03 +02:00
"bin_count_vectorizer = CountVectorizer(stop_words='english')\n",
"bin_count_train = bin_count_vectorizer.fit_transform(bin_X_train)\n",
2018-05-15 10:28:16 +02:00
"#transform the other sets into same feature vector as trained on train data \n",
2018-05-13 19:40:03 +02:00
"bin_count_test = bin_count_vectorizer.transform(bin_X_test)"
]
},
{
"cell_type": "code",
"execution_count": 19,
2018-05-31 11:49:28 +02:00
"metadata": {
"collapsed": true
},
2018-05-13 19:40:03 +02:00
"outputs": [],
"source": [
2018-05-15 10:28:16 +02:00
"#building the vectorspace\n",
"bin_tfidf_vectorizer = TfidfVectorizer(stop_words='english', max_df=0.7)#, min_df=0.0005)\n",
2018-05-13 19:40:03 +02:00
"bin_tfidf_train = bin_tfidf_vectorizer.fit_transform(bin_X_train)\n",
2018-05-15 10:28:16 +02:00
"#transform the other sets into same feature vector as trained on train data \n",
2018-05-13 19:40:03 +02:00
"bin_tfidf_test = bin_tfidf_vectorizer.transform(bin_X_test)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"a short look on the last 10 tokens for the vectors"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
2018-05-31 11:49:28 +02:00
"['تنجح', 'حلب', 'عن', 'لم', 'ما', 'محاولات', 'من', 'هذا', 'والمرضى', 'ยงade']"
2018-05-13 19:40:03 +02:00
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"bin_tfidf_vectorizer.get_feature_names()[-10:]"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['تنجح', 'حلب', 'عن', 'لم', 'ما', 'محاولات', 'من', 'هذا', 'والمرضى', 'ยงade']"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"bin_count_vectorizer.get_feature_names()[-10:]#[:10]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2018-05-15 10:28:16 +02:00
"### Generate Vectorizer on Multilabel Classes\n",
"* same again\n",
"* only additional transform for validation set"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
"execution_count": 22,
2018-05-31 11:49:28 +02:00
"metadata": {
"collapsed": true
},
2018-05-13 19:40:03 +02:00
"outputs": [],
"source": [
"mul_count_vectorizer = CountVectorizer(stop_words='english')\n",
"mul_count_train = mul_count_vectorizer.fit_transform(mul_X_train)\n",
"mul_count_test = mul_count_vectorizer.transform(mul_X_test)\n",
"mul_count_valid = mul_count_vectorizer.transform(mul_X_valid)"
]
},
{
"cell_type": "code",
2018-05-15 10:28:16 +02:00
"execution_count": 23,
2018-05-13 19:40:03 +02:00
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"mul_tfidf_vectorizer = TfidfVectorizer(stop_words='english', max_df=0.7)#, min_df=0.0005)\n",
"mul_tfidf_train = mul_tfidf_vectorizer.fit_transform(mul_X_train)\n",
"mul_tfidf_test = mul_tfidf_vectorizer.transform(mul_X_test)\n",
"mul_tfidf_valid = mul_tfidf_vectorizer.transform(mul_X_valid)"
]
},
{
"cell_type": "code",
2018-05-15 10:28:16 +02:00
"execution_count": 24,
2018-05-13 19:40:03 +02:00
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['zip',\n",
" 'zippo',\n",
" 'zombie',\n",
" 'zombies',\n",
" 'zone',\n",
" 'zones',\n",
" 'zoning',\n",
" 'zoo',\n",
" 'zuckerberg',\n",
" 'zuckerbergs']"
]
},
2018-05-15 10:28:16 +02:00
"execution_count": 24,
2018-05-13 19:40:03 +02:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mul_tfidf_vectorizer.get_feature_names()[-10:]"
]
},
{
"cell_type": "code",
2018-05-15 10:28:16 +02:00
"execution_count": 25,
2018-05-13 19:40:03 +02:00
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"['zip',\n",
" 'zippo',\n",
" 'zombie',\n",
" 'zombies',\n",
" 'zone',\n",
" 'zones',\n",
" 'zoning',\n",
" 'zoo',\n",
" 'zuckerberg',\n",
" 'zuckerbergs']"
]
},
2018-05-15 10:28:16 +02:00
"execution_count": 25,
2018-05-13 19:40:03 +02:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mul_count_vectorizer.get_feature_names()[-10:]#[:10]"
]
},
2018-05-15 10:28:16 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Generate Vectors on merged Sets\n",
"* again the same ..."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"concat_count_vectorizer = CountVectorizer(stop_words='english')\n",
"concat_count_train = concat_count_vectorizer.fit_transform(concat_X_train)\n",
"concat_count_test = concat_count_vectorizer.transform(concat_X_test)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"concat_tfidf_vectorizer = TfidfVectorizer(stop_words='english', max_df=0.7)#, min_df=0.0005)\n",
"concat_tfidf_train = concat_tfidf_vectorizer.fit_transform(concat_X_train)\n",
"concat_tfidf_test = concat_tfidf_vectorizer.transform(concat_X_test)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['القادمون',\n",
" 'ایران',\n",
" 'جنگ',\n",
" 'سال',\n",
" 'عربي',\n",
" 'علیه',\n",
" 'مطالعاتی',\n",
" 'مورد',\n",
" 'کدآمایی',\n",
" 'ยงade']"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"concat_tfidf_vectorizer.get_feature_names()[-10:]"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['القادمون',\n",
" 'ایران',\n",
" 'جنگ',\n",
" 'سال',\n",
" 'عربي',\n",
" 'علیه',\n",
" 'مطالعاتی',\n",
" 'مورد',\n",
" 'کدآمایی',\n",
" 'ยงade']"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"concat_count_vectorizer.get_feature_names()[-10:]#[:10]"
]
},
2018-05-13 19:40:03 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Confusion Matrix Code\n",
2018-05-15 10:28:16 +02:00
"* copy paste by distributed notebook\n",
"* thx for providing"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
2018-05-15 10:28:16 +02:00
"execution_count": 30,
2018-05-13 19:40:03 +02:00
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def plot_confusion_matrix(cm, classes,\n",
" normalize=False,\n",
" title='Confusion matrix',\n",
" cmap=plt.cm.Blues):\n",
" \"\"\"\n",
" See full source and example: \n",
" http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html\n",
" \n",
" This function prints and plots the confusion matrix.\n",
" Normalization can be applied by setting `normalize=True`.\n",
" \"\"\"\n",
" #added after jonas hint\n",
" fig_1,ax_1 = plt.subplots()\n",
" \n",
" plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
" plt.title(title)\n",
" plt.colorbar()\n",
" tick_marks = np.arange(len(classes))\n",
" plt.xticks(tick_marks, classes, rotation=45)\n",
" plt.yticks(tick_marks, classes)\n",
"\n",
" if normalize:\n",
" cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
" print(\"Normalized confusion matrix\")\n",
" else:\n",
" print('Confusion matrix, without normalization')\n",
"\n",
" thresh = cm.max() / 2.\n",
" for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
" plt.text(j, i, cm[i, j],\n",
" horizontalalignment=\"center\",\n",
" color=\"white\" if cm[i, j] > thresh else \"black\")\n",
"\n",
" plt.tight_layout()\n",
" plt.ylabel('True label')\n",
" plt.xlabel('Predicted label')"
]
},
2018-05-31 11:49:28 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dictionary definition for export to rdf in the end"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"example_configuration = {\n",
" \"name\":\"experiment1\",\n",
" \n",
" \"model\":\"generic_model_1_name\",\n",
" \"model-name\":\"deepNN\",\n",
"\n",
"\n",
" \"dataset\":\"X1\",\n",
" \"dataset-name\":\"fake_or_real_news\",\n",
" \"dataset-link\":\"'/Users/Carsten/GitRepos/NLP-LAB/Carsten_Solutions/sets/fact checking/fake_or_real_news.csv'\",\n",
" \n",
" \"measurement-name\":\"measurement_configuration1\",\n",
" \n",
" \"execution-name\":\"execution_configuration1\",\n",
" \n",
" \"precision\":0.33,\n",
" \"recall\":0.73,\n",
" \"accuracy\":0.55\n",
"}"
]
},
2018-05-13 19:40:03 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Configurations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configuration 1\n",
"* model a - train - [performance measures][0:4]\n",
"* model a - test - [performance measures][0:4]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* model a = MultinomialNB\n",
" * with tfidf vectorizer* dataset 1\n",
" * in contrast to the notebook with count vect.\n",
" * with dataset 1: fake_or_real_news.csv\n",
" * ** Take care with seeds for example split train test data function**\n",
" "
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 32,
2018-05-13 19:40:03 +02:00
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
2018-05-31 11:49:28 +02:00
"clf_model_a = MultinomialNB()"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 33,
2018-05-13 19:40:03 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2018-05-31 11:49:28 +02:00
"accuracy: 0.887\n",
2018-05-13 19:40:03 +02:00
"Confusion matrix, without normalization\n"
]
},
{
"data": {
2018-05-31 11:49:28 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAVYAAAEmCAYAAAA5jbhCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xm8VXW5x/HP9xxwBAVkEAVBCacc\ncEJTU0pTnMmcUHHIQkvrds26aA6YWZZeU9Msvc6WaJmKc2o55qyAmIqomCiCiCGzcHjuH2sd2hw5\nZ29g7bPXPvv77rVe7P1ba6/1LLY9/Pazfuu3FBGYmVl26iodgJlZW+PEamaWMSdWM7OMObGamWXM\nidXMLGNOrGZmGXNitWZJWl3S3ZJmSvrTSuznKEl/zTK2SpH0ZUlvVDoOyzd5HGv1k3QkcCqwKTAL\nGAOcHxFPruR+hwHfA3aOiEUrHWjOSQqgf0RMrHQsVt3cY61ykk4FLgF+DvQANgB+CxyUwe77ABNq\nIamWQlK7SsdgVSIivFTpAqwNzAYObWGbVUkS7wfpcgmwarpuEDAZ+CEwDZgCHJ+uOxf4DFiYHuME\nYCRwc8G++wIBtEvfHwe8TdJrfgc4qqD9yYLP7Qw8D8xM/9y5YN2jwHnAU+l+/gp0bebcGuP/cUH8\nQ4B9gQnADOCMgu0HAk8D/063vRxYJV33eHouc9LzPbxg//8DfAjc1NiWfqZfeoxt0/frAdOBQZX+\nb8NLZRf3WKvbl4DVgDta2OYnwE7AAGBrkuRyZsH6dUkS9PokyfMKSZ0j4hySXvCtEdEhIq5pKRBJ\nawKXAftEREeS5DlmGdt1Ae5Nt10HuBi4V9I6BZsdCRwPdAdWAU5r4dDrkvwdrA+cDVwNHA1sB3wZ\nOFvSRum2DcB/A11J/u72AL4LEBG7pdtsnZ7vrQX770LSex9eeOCIeIsk6f5B0hrAdcD1EfFoC/Fa\nDXBirW7rANOj5Z/qRwE/jYhpEfERSU90WMH6hen6hRFxH0lvbZMVjGcxsIWk1SNiSkS8uoxt9gPe\njIibImJRRNwCvA4cULDNdRExISLmAbeR/KPQnIUk9eSFwCiSpHlpRMxKj/8qsBVARLwYEc+kx50E\n/B7YvYRzOiciFqTxLCUirgbeBJ4FepL8Q2Y1zom1un0MdC1S+1sPeLfg/btp25J9NEnMc4EOyxtI\nRMwh+fl8EjBF0r2SNi0hnsaY1i94/+FyxPNxRDSkrxsT39SC9fMaPy9pY0n3SPpQ0qckPfKuLewb\n4KOImF9km6uBLYDfRMSCIttaDXBirW5PA/NJ6orN+YDkZ2yjDdK2FTEHWKPg/bqFKyPiwYj4GknP\n7XWShFMsnsaY3l/BmJbHlSRx9Y+ItYAzABX5TIvDZiR1IKlbXwOMTEsdVuOcWKtYRMwkqSteIWmI\npDUktZe0j6RfpZvdApwpqZukrun2N6/gIccAu0naQNLawOmNKyT1kHRgWmtdQFJSaFjGPu4DNpZ0\npKR2kg4HNgfuWcGYlkdH4FNgdtqb/k6T9VOBjT73qZZdCrwYEd8iqR3/bqWjtKrnxFrlIuJikjGs\nZwIfAe8BpwB3ppv8DHgBGAe8AryUtq3IsR4Cbk339SJLJ8M6ktEFH5BcKd+d9MJQk318DOyfbvsx\nyRX9/SNi+orEtJxOI7kwNoukN31rk/UjgRsk/VvSYcV2JukgYDBJ+QOS72FbSUdlFrFVJd8gYGaW\nMfdYzcwy5sRqZpYxJ1Yzs4w5sZqZZazNTipRv/ra0W6t7pUOw5ZDvx7LfV+CVdgHk//Fv2d8XGws\n8HKpX6tPxKLP3eT2OTHvowcjYnCWx85Km02s7dbqTs+hv650GLYcbvrBbsU3slwZdmCxO4KXXyya\nx6qbFB3txvwxVxS7a65i2mxiNbNqJVB1VymdWM0sXwTU1Vc6ipXixGpm+aNMy7atzonVzHLGpQAz\ns+y5x2pmliHhHquZWbbkHquZWeY8KsDMLEu+eGVmli3hUoCZWebcYzUzy5JLAWZm2atzKcDMLDue\nK8DMLGsuBZiZZc+jAszMMuYeq5lZhuRbWs3Msuceq5lZluRRAWZmmXMpwMwsQ56P1cwsax7HamaW\nPZcCzMwy5otXZmYZkksBZmbZcynAzCxbcmI1M8tO8mQWJ1Yzs+woXaqYE6uZ5Yyoq6vui1fVHb2Z\ntUmSii4l7KO3pL9Lek3Sq5L+K23vIukhSW+mf3ZO2yXpMkkTJY2TtG3Bvo5Nt39T0rHFju3Eama5\nk0ViBRYBP4yIzYCdgJMlbQ6MAB6JiP7AI+l7gH2A/ukyHLgyjaULcA6wIzAQOKcxGTfHidXM8kUl\nLkVExJSIeCl9PQt4DVgfOAi4Id3sBmBI+vog4MZIPAN0ktQT2Bt4KCJmRMQnwEPA4JaO7RqrmeWK\nKLlHWvo+pb7ANsCzQI+ImAJJ8pXUPd1sfeC9go9NTtuaa2+WE6uZ5U6JibWrpBcK3l8VEVctY18d\ngNuBH0TEpy3se1krooX2ZjmxmlnulDgqYHpEbN/SBpLakyTVP0TEX9LmqZJ6pr3VnsC0tH0y0Lvg\n472AD9L2QU3aH20x/lKiNzNrNRnVWJV0Ta8BXouIiwtWjQYar+wfC9xV0H5MOjpgJ2BmWjJ4ENhL\nUuf0otVeaVuz3GM1s9zJqMa6CzAMeEXSmLTtDOAC4DZJJwD/Ag5N190H7AtMBOYCxwNExAxJ5wHP\np9v9NCJmtHRgJ1Yzy5WsLl5FxJM037fdYxnbB3ByM/u6Fri21GM7sZpZ7niuADOzrFV3XnViNbOc\nUcmjAnLLidXMcselADOzDJXjzqvW5sRqZvlT3XnViTUPfjV0K766eQ8+nr2AvX/5OACbr78W5x+6\nJau2r2NRQ3DWn8cz9l//pl/3NbnwyAF8sddaXHTvG1z997cB2Kj7mlx+7JJZzui9zhr8+v4JXPvY\nOxU5p1pzwK5bskaHDtTX1VPfrp6bRj/GpT8/k8cfeYD27VehV58NOefCK+i4VicWLVzIeSO+x+uv\njqVh0SL2O/gIjv/uDyt9CvkhlwIsA39+djI3PDGJi48asKRtxAGbcemDE3j0tY8YtFl3Tj9wM464\n/Gn+PXchI28fz15brrvUPt6eNod9L3wCgDrBs+fuyYPjPmzV86h1v//jPXTqss6S9zvu+hVO/vFI\n2rVrx2UXnM11v72Y74/4KQ/fdyeffbaAWx94mvnz5nLo13Zk7wMPYb1efSoYfb5Ue2Kt7ktvbcRz\nb89g5tyFTVqDDqsl/+6ttXo7ps6cD8DHsz9j3HszWbS4+Tkgdtm4K+9On8v7n8wrV8hWgp1224N2\n7ZLvcMttdmDahx8kKyTmz53LokWLmD9/Pu3bt2fNDh0rGGn+qE5FlzxzjzWnzr3jn9x40o6cceDm\n1El849KnSv7sAduux+iXPihjdNaUBCcfMwRJHDz0eA4+8vil1o++7Wa+tv/BAOy5z0E89tC9DN5x\nY+bPm8epZ/6ctTt1qUTYuVXtPdayJVZJDcArBU1DImJSuu5S4BCgd0QsTtuOA7aPiFMk1QHXAQ3A\nCcA7wKz0PcDjEfH9csWeB0fv0ofz7niVB8Z9yH4DevLLI7bi6CufLfq59vVizy+uy6/ufr0VorRG\n1/z5r3Tr0ZMZ0z/i5GFD6NtvY7bdcZdk3eUXUt+uHfsMOQyA8WNfpL6+ngeeeYNPZ/6bbx02mIG7\nDqLXBhtW8hRyYzmeEJBb5SwFzIuIAQXLJIA0aX6dZOLY3Zp+KJ2R5ndAe+Bb6f27AF8p2FebTqoA\n39ihFw+kNdJ7x0xh6z6dSvrcoM26M37yTKbP/qyc4VkT3Xr0BKBL124M2nt/Xh37IgD33P5Hnvzb\ng/zskquXJIsH7/oTX9ptT9q1b0+Xrt3YevudeG3cyxWLPY8yejRLxVSixvoVYDzJ82SGLmP9pcA6\nwDGNvdlaNO3T+ez0heR
2018-05-13 19:40:03 +02:00
"text/plain": [
2018-05-31 11:49:28 +02:00
"<matplotlib.figure.Figure at 0x1a1640c550>"
2018-05-13 19:40:03 +02:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
2018-05-31 11:49:28 +02:00
"clf_model_a.fit(bin_tfidf_train, bin_y_train)\n",
"pred = clf_model_a.predict(bin_tfidf_train)\n",
2018-05-13 19:40:03 +02:00
"score = metrics.accuracy_score(bin_y_train, pred)\n",
"print(\"accuracy: %0.3f\" % score)\n",
"cm = metrics.confusion_matrix(bin_y_train, pred, labels=['FAKE', 'REAL'])\n",
"plot_confusion_matrix(cm, classes=['FAKE', 'REAL'])"
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 34,
2018-05-13 19:40:03 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2018-05-31 11:49:28 +02:00
"accuracy: 0.826\n",
2018-05-13 19:40:03 +02:00
"Confusion matrix, without normalization\n"
]
},
{
"data": {
2018-05-31 11:49:28 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAVAAAAEmCAYAAAA0k8gFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xe8FNX9//HX+4IgCIKIIFLEgprY\nEGs0VoxdIcauiAaDGk23xfhNNImJiYmxxkTjV9EU9GuiMWo0xBLLzxILoEYF7CiCWLAA0j6/P+Zc\nsl7v3d277GVnue+nj3nszpmzM2dZ+XDKzDmKCMzMrPUaal0AM7N65QBqZlYhB1Azswo5gJqZVcgB\n1MysQg6gZmYVcgC1FknqIulvkuZI+r9lOM+Rkv5RzbLViqQdJT1f63JYPsj3gdY/SUcA3wY2Aj4A\nJgLnRsQDy3jeUcDXgO0jYtEyFzTnJAUwJCKm1bosVh9cA61zkr4NXAj8BOgLDAJ+DYyowunXBqa0\nh+BZDkkda10Gy5mI8FanG9AD+BA4uEiezmQB9o20XQh0Tsd2AaYD3wFmATOAY9Oxc4AFwMJ0jTHA\n2cDvC849GAigY9o/BniRrBb8EnBkQfoDBZ/bHvg3MCe9bl9w7F7gR8CD6Tz/AHq38N0ay39aQflH\nAvsAU4B3gDML8m8DPAS8l/JeCnRKx+5L3+Wj9H0PLTj/6cCbwHWNaekz66VrDEv7awGzgV1q/f+G\nt+WzuQZa3z4HrAzcVCTP94DtgKHA5mRB5KyC42uSBeL+ZEHyMkmrRcQPyGq110dEt4i4qlhBJK0C\nXAzsHRHdyYLkxGby9QJuS3lXBy4AbpO0ekG2I4BjgT5AJ+CUIpdek+zPoD/wfeBK4ChgS2BH4PuS\n1k15FwPfAnqT/dkNB74KEBE7pTybp+97fcH5e5HVxscWXjgiXiALrn+Q1BW4GrgmIu4tUl5bgTiA\n1rfVgdlRvIl9JPDDiJgVEW+R1SxHFRxfmI4vjIjbyWpfG1ZYniXAJpK6RMSMiHimmTz7AlMj4rqI\nWBQRfwKeA/YvyHN1REyJiHnADWTBvyULyfp7FwLjyYLjRRHxQbr+M8BmABHxeEQ8nK77MvBbYOcy\nvtMPIuLjVJ5PiIgrganAI0A/sn+wrJ1wAK1vbwO9S/TNrQW8UrD/Skpbeo4mAXgu0K21BYmIj8ia\nvScAMyTdJmmjMsrTWKb+BftvtqI8b0fE4vS+McDNLDg+r/HzkjaQdKukNyW9T1bD7l3k3ABvRcT8\nEnmuBDYBLomIj0vktRWIA2h9ewiYT9bv15I3yJqfjQaltEp8BHQt2F+z8GBE3BkRXyCriT1HFlhK\nlaexTK9XWKbWuJysXEMiYlXgTEAlPlP0NhVJ3cj6la8Czk5dFNZOOIDWsYiYQ9bvd5mkkZK6SlpJ\n0t6Sfp6y/Qk4S9Iaknqn/L+v8JITgZ0kDZLUA/hu4wFJfSUdkPpCPybrCljczDluBzaQdISkjpIO\nBT4L3FphmVqjO/A+8GGqHZ/Y5PhMYN1Pfaq4i4DHI+I4sr7d3yxzKa1uOIDWuYi4gOwe0LOAt4DX\ngJOBm1OWHwOPAZOBp4AnUlol15oAXJ/O9TifDHoNZKP5b5CNTO9MGqBpco63gf1S3rfJRtD3i4jZ\nlZSplU4hG6D6gKx2fH2T42cD4yS9J+mQUieTNALYi6zbArLfYZikI6tWYss130hvZlYh10DNzCrk\nAGpmViEHUDOzCjmAmplVaIWdHKFDlx6xUo8+tS6GtUL/1buWzmS58tYbr/H+u++Uupe2VTqsunbE\nok899PUpMe+tOyNir2peu7VW2AC6Uo8+DDjq4loXw1rhZ6OKPbFpeXTaEdWPX7FoHp03LHkXGfMn\nXlbqKbI2t8IGUDOrVwLVR++iA6iZ5YuAhg61LkVZHEDNLH9U1W7VNuMAamY54ya8mVnlXAM1M6uA\ncA3UzKwycg3UzKxiHoU3M6uEB5HMzCoj6qYJXx9h3szaFzWU3kqdQtpQ0sSC7X1J35TUS9IESVPT\n62opvyRdLGmapMmShpW6hgOomeWMqhJAI+L5iBgaEUOBLclWeL0JOAO4KyKGAHelfYC9gSFpG0u2\nCGFRDqBmlj8NKr21znDghYh4BRgBjEvp4/jvqrYjgGsj8zDQU1K/Yid1H6iZ5Uv5z8L3lvRYwf4V\nEXFFC3kPI1uhFqBvRMwAiIgZkhrnvexPtihjo+kpbUZLBXAANbOcKXsUfnZEbFXybFIn4AAKluFu\n+cKfUnTVTTfhzSx/pNJb+fYGnoiImWl/ZmPTPL3OSunTgYEFnxtAtkx3ixxAzSx/qjCIVOBw/tt8\nB7gFGJ3ejwb+WpB+dBqN3w6Y09jUb4mb8GaWL62vYRY5lboCXwCOL0g+D7hB0hjgVeDglH47sA8w\njWzE/thS53cANbP8qdKTSBExF1i9SdrbZKPyTfMGcFJrzu8AamY5Iz8Lb2ZWsTp5lNMB1MzyxfOB\nmplVyrMxmZlVzk14M7MKeRDJzKwCchPezKxybsKbmVVGDqBmZq2XrejhAGpm1nqi+YnlcsgB1Mxy\nRjQ0eBDJzKwibsKbmVXIAdTMrBLuAzUzq4yQa6BmZpVyADUzq5BH4c3MKuE+UDOzyrkJb2ZWgXoa\nRKqPjgYza1ckldzKPE9PSTdKek7Ss5I+J6mXpAmSpqbX1VJeSbpY0jRJkyUNK3V+B1Azyx+VsZXn\nIuCOiNgI2Bx4FjgDuCsihgB3pX2AvYEhaRsLXF7q5A6gZpYvykbhS20lTyOtCuwEXAUQEQsi4j1g\nBDAuZRsHjEzvRwDXRuZhoKekfsWu4QBqZrlTZhO+t6THCraxTU6zLvAWcLWkJyX9TtIqQN+ImAGQ\nXvuk/P2B1wo+Pz2ltciDSGaWK60YRJodEVsVOd4RGAZ8LSIekXQR/22uN3/pT4tiBXAN1Mzypzp9\noNOB6RHxSNq/kSygzmxsmqfXWQX5BxZ8fgDwRrELuAaaQ/eeuTMffbyYxUuCxUuCL170/wAYtcPa\njNphEIuXBPc8+xY/v+15AE7YbV0O3mYAi5cEP7r5We6fMruWxW93Zr/5Opec9Q3ee3sWUgNf+NJR\n7HvkcVx/+S+46y9/ZNXVegFwxNe+y7Adh7Nw4QKu+NFpvPCfyaihgWNP/SGbbL19jb9Fjqg694FG\nxJuSXpO0YUQ8DwwH/pO20cB56fWv6SO3ACdLGg9sC8xpbOq3xAE0p466/BHenbtw6f526/Vi9437\nsN8vH2TB4iX06tYJgPX7dmPfof3Y+/wH6NOjM9eO3Ybdf/YvlhRteFg1dejQkdHf+T7rfmYz5n30\nIacdvhebbbcTAPse9RVGjD7xE/n/+ec/AHDBjXcz553ZnHvSkZz3h7/XzeOLy0MV7wP9GvAHSZ2A\nF4FjyVreN0gaA7wKHJzy3g7sA0wD5qa8RTmA1okjth/Eb+95kQWLlwDwzocLANh94z7cNnEGCxYv\nYfo783jl7Y/YfFBPnnzlvVoWt11ZbY2+rLZGXwC6rNKN/uuuzzuzWq64TH9xCptuuyMAPXr1pmv3\nHrzwzCSGbLrFcilvPVBDdQJoREwEmusnHd5M3gBOas35/U9eDgVwzditufmb23PotlmXzODeq7D1\nOqtx49c/xx9P3JZNB/YAoG+PlZnx3vyln33zvfn07bFyLYptwKzXX+Pl555myKbZPdh3jL+abx88\nnMt+8C0+fD/7R23wBhvz6D13snjRIma+/iov/mcyb88s2tXW7lTrRvq21mY1UEmLgacKkkZGxMvp\n2EXAQcDAiFiS0o4BtoqIkyU1AFcDi4ExwEvAB2kf4L6I+Hpblb3WDr30YWa9/zG9unVi3NitefGt\nD+nYQazaZSUOuvghNhvYg4tHDWXXn/yr+WHDcPu9FubN/YhfnHIcx5z6Q7p2686eh4zmoLHfQhLj\nL/s54355Died8yt2G3k
2018-05-13 19:40:03 +02:00
"text/plain": [
2018-05-31 11:49:28 +02:00
"<matplotlib.figure.Figure at 0x1a162f7b70>"
2018-05-13 19:40:03 +02:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
2018-05-31 11:49:28 +02:00
"#clf_model_a.fit(bin_tfidf_train, bin_y_train)\n",
"pred = clf_model_a.predict(bin_tfidf_test)\n",
2018-05-13 19:40:03 +02:00
"score = metrics.accuracy_score(bin_y_test, pred)\n",
"print(\"accuracy: %0.3f\" % score)\n",
"cm = metrics.confusion_matrix(bin_y_test, pred, labels=['FAKE', 'REAL'])\n",
2018-05-31 11:49:28 +02:00
"plot_confusion_matrix(cm, classes=['FAKE', 'REAL'])\n",
"\n",
"precision = cm[1,1]/(cm[1,1]+cm[0,1])\n",
"recall = cm[1,1]/(cm[1,1]+cm[1,0])"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
"configuration_1 = {\n",
" \"name\":\"configuration1\",\n",
" \n",
" \"model\":\"model_a\",\n",
" \"model-name\":\"multinomialNB\",\n",
"\n",
"\n",
" \"dataset\":\"X1\",\n",
" \"dataset-name\":\"fake_or_real_news\",\n",
" \"dataset-link\":\"unknown\",\n",
" \n",
" \"measurement-name\":\"measurement_config1\",\n",
" \n",
" \"execution-name\":\"execution_config1\",\n",
" \n",
" \"precision\":precision,\n",
" \"recall\":recall,\n",
" \"accuracy\":score\n",
"}"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configuration 2\n",
"* model b - train - [performance measures]\n",
"* model b - validation - [performance measures]\n",
"* model b - test - [performance measures]"
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 36,
2018-05-13 19:40:03 +02:00
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
2018-05-31 11:49:28 +02:00
"clf_model_b = MultinomialNB()"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 37,
2018-05-13 19:40:03 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"accuracy: 0.602\n",
"Confusion matrix, without normalization\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAV4AAAEmCAYAAAAqWvi2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzsnXd8VMXXh5+TBKL0DqG3UKTX0KU3\n4QcISFMsKIJd6UVA6UhRaSqKCiLiKyq99y4JXToCEnqHUJNw3j/uTdyEJBtCdpPVefzMJ/eeO3fm\nu7t4dvbM3DmiqhgMBoPBfXgltQCDwWD4r2Ecr8FgMLgZ43gNBoPBzRjHazAYDG7GOF6DwWBwM8bx\nGgwGg5sxjtfgVkTkSRFZICLXReT/HqOdTiKyPDG1JRUiUlNEDiW1DoP7ELOO1xATItIR+AAoBtwE\ndgHDVXXjY7b7AvA2UE1Vwx5baDJHRBTwV9WjSa3FkHwwI17DQ4jIB8CnwAggO5AXmAK0SITm8wGH\n/wtONz6IiE9SazAkAapqiimRBUgPhABt46jji+WYz9jlU8DXvlYbCAZ6ABeAs8DL9rWPgPtAqN1H\nF2AI8IND2/kBBXzs85eAv7BG3ceBTg72jQ73VQO2A9ftv9Ucrq0FhgKb7HaWA1lieW0R+ns76G8J\nNAUOA1eA/g71KwNbgGt23UlASvvaevu13LJfbzuH9vsA54CZETb7nkJ2H+Xt85zAJaB2Uv/bMCXx\nihnxGqJTFXgC+C2OOgOAKkBZoAyW8xnocD0HlgPPheVcJ4tIRlUdjDWKnqOqaVT1m7iEiEhq4HOg\niaqmxXKuu2KolwlYZNfNDIwHFolIZodqHYGXgWxASqBnHF3nwHoPcgGDgGnA80AFoCYwSEQK2nXD\ngfeBLFjvXT3gDQBVrWXXKWO/3jkO7WfCGv13dexYVY9hOeVZIpIK+Bb4TlXXxqHX4GEYx2uITmbg\nksYdCugEfKyqF1T1ItZI9gWH66H29VBVXYw12iuaQD0PgJIi8qSqnlXVP2Oo8wxwRFVnqmqYqs4G\nDgLNHep8q6qHVfUO8DPWl0ZshGLFs0OBn7Cc6meqetPu/0+gNICqBqnqVrvfE8CXwNPxeE2DVfWe\nrScKqjoNOAJsA/ywvugM/yKM4zVE5zKQxUnsMSdw0uH8pG2LbCOa474NpHlUIap6C+vneTfgrIgs\nEpFi8dAToSmXw/m5R9BzWVXD7eMIx3je4fqdiPtFpIiILBSRcyJyA2tEnyWOtgEuqupdJ3WmASWB\niap6z0ldg4dhHK8hOluAu1hxzdg4g/UzOYK8ti0h3AJSOZzncLyoqstUtQHWyO8glkNypidC0+kE\nanoUpmLp8lfVdEB/QJzcE+dSIhFJgxU3/wYYYodSDP8ijOM1REFVr2PFNSeLSEsRSSUiKUSkiYiM\nsavNBgaKSFYRyWLX/yGBXe4CaolIXhFJD/SLuCAi2UXkf3as9x5WyCI8hjYWA0VEpKOI+IhIO+Ap\nYGECNT0KaYEbQIg9Gu8e7fp5oOBDd8XNZ0CQqr6KFbv+4rFVGpIVxvEaHkJVx2Ot4R0IXAROAW8B\nv9tVhgGBwB5gL7DDtiWkrxXAHLutIKI6Sy+s1RFnsGb6n8aeuIrWxmWgmV33MtaKhGaqeikhmh6R\nnlgTdzexRuNzol0fAnwvItdE5DlnjYlIC6AxVngFrM+hvIh0SjTFhiTHPEBhMBgMbsaMeA0Gg8HN\nGMdrMBgMbsY4XoPBYHAzxvEaDAaDmzEbdLgI7yfTq0/6bEktI0ZK5Eqf1BJi5V7og6SWECc+3s6W\n6CYd3l7JU9vfJ09w6dKlRBPnnS6fathDD/w9hN65uExVGydWv4mJcbwuwid9NnJ3+iypZcTIxpFN\nklpCrBw9H5LUEuIkc5qUSS0hVjIlU23Vq1RK1PY07A6+RZ2uzOPursnOniBMMozjNRgMnoUIeHkn\ntYrHwjheg8HgeYhnT08Zx2swGDwPSZ7x7PhiHK/BYPAwTKjBYDAY3ItgQg0Gg8HgXsTjQw2e/bVh\nMBj+m4iX8+KsCZHpInJBRPY52OaIyC67nBCRXbY9v4jccbj2hcM9FURkr4gcFZHPRZx/K5gRr8Fg\n8DASLcb7HVZy0hkRBlVtF9mLyDis5KkRHFPVmFJGTcXKnbcVa2/oxsCSuDo2I16DweBZCFaowVlx\ngqqux9rn+eEurFHrc1ib/scuRcQPSKeqW9TaY3cGcWdvAYzjNRgMnkj8Qg1ZRCTQoXR11qwDNYHz\nqnrEwVZARHaKyDoRqWnbcgHBDnWCiZrrL0ZMqMFgMHgYAt7xCjVcUtWKCeykA1FHu2eBvKp6WUQq\nAL+LSAlizq/nNLuEGfEmEaPaleKPIfVY0rNmpO39xv4s6lGDBR/U4LuulciWzheANE/48NUrFVjY\nowZLetWkdaXcABTPmZb/e7sqS3rVZFGPGjxT1s+lmu/evUut6gEEVCxLxbIlGfbx4CjXe7z3Ntky\npXWpBkfu3b1Lx+a1aduoGq3qVWbKuOEAqCoTx3xM86fL0bJuRWZNnxp5z/YtG3iucXVa1avMK21d\nt2dFj7e6UqZIHupVKx9p+3Pvbpo3qEXDWpVpWrcaO4O2A3Dt2lW6vPAc9WtU5Jn6NTi4P6YM9q4h\n+NQpGjeoS7lST1GhTEkmT4y6v8in48eSKqUXly65I4tSPIlYTvaYk2uxNm9l2H4WhzROqnrPTjGF\nqgYBx4AiWCPc3A635yYeiV/NiDeJmLs9mJkbTzK2Q5lI27Q1x5mw1Ppl82KNfLzdwJ8P5+7jher5\nOHo+hK7Tg8iUOiUr+tZi/o7T3Ln/gF6zd3Pi0m2ypfNl3vvVWX/wIjfvhsXW7WPh6+vL4mWrSJMm\nDaGhodSvU5OGjZpQOaAKO4ICuXb9uvNGEpGUvr58/dNCUqW29LzUuiE16jTgr6OHOXcmmHlrgvDy\n8uLypYsA3Lh+jREDPmDKzF/xy5Un0u4K2nZ8gZde68573btE2oYP7s/7vQdQt0EjVq1YyvAh/fll\nwQomjh9DiZKl+Wbmzxw9fIgBvd9lzu9LXabNEW8fH0aOGUu5cuW5efMm1QMqUrdeA4o/9RTBp06x\netVK8uTN6xYtj4Rrl5PVBw6qamQIQUSyAldUNVxECgL+wF+qekVEbopIFWAb0BmY6KwDM+JNIrb/\ndZVrt0Oj2ELu/eMwn0zpg9q/WFQhta/1HZnK15vrt0MJe6CcuHSLE5duA3Dhxj0uh9x36e5ZIkKa\nNGkACA0NJTQ0FBEhPDycAf16M2zEaJf1HZueVKktPWFhoYSFhYEIP8/8mtff64OXl/XPO3OWrAAs\nmfd/1GvSHL9ceaLYXUGVajXJkDHjQ3pDbt4A4OaN62TPYf1COXLoADWergNA4SJFCf77JBcvnHeZ\nNkf8/PwoV84aladNm5aixYpz5sxpAHr3/IBhI0YTj9VRbsZe1eCsOGtFZDawBSgqIsEiEvEt2Z6H\nJ9VqAXtEZDfwC9BNVSMm5roDXwNHsUbCca5oADPiTXb0aFKEVhVzcfNOGJ2mbgNg5qYTfPVKRbYM\nrktqXx/embmT6DlKS+dJTwpvL05evu1SfeHh4VSvUpG/jh2la7c3qFQ5gMkTP6PpM83x83NtqCM2\nPR2eqcXfJ/6iXefXKF2uEsEnj7Nswa+sXrqQjJkz0+ejMeQrUJiTfx0lLCyULs815VZICJ1e6Ubz\nNh3dpnXIiLF0atOMoYP68kCVeUvXAPBUyVIsWTCPylWqszNoO8Gn/ubsmdNkzZbdbdoATp44we7d\nO6lUOYCFC+aTM1dOSpcp4/zGpCARnlxT1Q6x2F+KwTYXmBtL/UCg5KP0/a8f8YrIOyJyQERmxXK9\ntogsjOlaUjBuyWFqDF3DvB1neKFGPgBqFs3K/tM3qPrRapqP28iQViVI4/vPd2bWtL6M61iGPj/t\necghJzbe3t5s3b6Tw3+
"text/plain": [
2018-05-31 11:49:28 +02:00
"<matplotlib.figure.Figure at 0x1a14d84a20>"
2018-05-13 19:40:03 +02:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
2018-05-31 11:49:28 +02:00
"clf_model_b.fit(mul_tfidf_train, mul_y_train)\n",
"pred = clf_model_b.predict(mul_tfidf_train)\n",
2018-05-13 19:40:03 +02:00
"\n",
"#print(pred[:20])\n",
"\n",
"score = metrics.accuracy_score(mul_y_train, pred)\n",
"print(\"accuracy: %0.3f\" % score)\n",
"cm = metrics.confusion_matrix(mul_y_train, pred, labels=['false', 'barely-true' , 'half-true' , 'mostly-true' , 'true'])\n",
"plot_confusion_matrix(cm, classes=['false', 'barely-true' , 'half-true' , 'mostly-true' , 'true'])"
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 38,
2018-05-13 19:40:03 +02:00
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
2018-05-31 11:49:28 +02:00
"#clf = MultinomialNB()"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 39,
2018-05-13 19:40:03 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"accuracy: 0.238\n",
"Confusion matrix, without normalization\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAVgAAAEmCAYAAAAnRIjxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xd8FFXXwPHfSUJvAULvTUCQIkhX\nioC9dxABUWyIXbG3R0WfVx57b4io2EVEihQFpBcpioAURXovCZBy3j/uBJeQZEPY3dmF8/WzH3Zn\nZmdONvHsnTN37hVVxRhjTOjF+R2AMcYcqyzBGmNMmFiCNcaYMLEEa4wxYWIJ1hhjwsQSrDHGhIkl\nWBNRIlJERL4TkZ0i8vlR7KeniIwLZWx+EZFTReQPv+MwoSfWD9ZkR0R6AHcCDYDdwALgKVWdepT7\n7QXcCrRT1bSjDjTKiYgC9VR1hd+xmMizFqw5jIjcCbwAPA1UAKoDrwEXhGD3NYBlx0NyzQsRSfA7\nBhNGqmoPexx8AKWAPcBluWxTCJeA13mPF4BC3rpOwFrgLmATsB7o6617HDgApHrH6Ac8BnwUsO+a\ngAIJ3us+wEpcK3oV0DNg+dSA97UDZgM7vX/bBaybDDwJTPP2Mw5IyuFny4z/3oD4LwTOBpYB24AH\nArZvBUwHdnjbvgIU9Nb97P0se72f94qA/d8HbACGZS7z3lPHO8bJ3uvKwBagk99/G/Y48oe1YE1W\nbYHCwNe5bPMg0AZoBjTFJZmHAtZXxCXqKrgk+qqIlFbVR3Gt4hGqWlxV380tEBEpBrwEnKWqJXBJ\ndEE225UBvve2LQsMAb4XkbIBm/UA+gLlgYLA3bkcuiLuM6gCPAK8DVwNtABOBR4RkdretunAHUAS\n7rM7HbgZQFVP87Zp6v28IwL2XwbXmu8feGBV/ROXfIeLSFHgfeADVZ2cS7wmSlmCNVmVBbZo7qfw\nPYEnVHWTqm7GtUx7BaxP9danqupoXOutfj7jyQAai0gRVV2vqkuy2eYcYLmqDlPVNFX9BFgKnBew\nzfuqukxVU4DPcF8OOUnF1ZtTgU9xyfNFVd3tHX8J0ARAVeeq6gzvuKuBN4GOefiZHlXV/V48h1DV\nt4HlwEygEu4LzcQgS7Amq61AUpDaYGVgTcDrNd6yg/vIkqCTgeJHGoiq7sWdVt8IrBeR70WkQR7i\nyYypSsDrDUcQz1ZVTfeeZybAjQHrUzLfLyIniMgoEdkgIrtwLfSkXPYNsFlV9wXZ5m2gMfCyqu4P\nsq2JUpZgTVbTgX24umNO1uFObzNV95blx16gaMDrioErVXWsqnbDteSW4hJPsHgyY/onnzEdiddx\ncdVT1ZLAA4AEeU+uXXdEpDiurv0u8JhXAjExyBKsOYSq7sTVHV8VkQtFpKiIFBCRs0TkOW+zT4CH\nRKSciCR523+Uz0MuAE4TkeoiUgq4P3OFiFQQkfO9Wux+XKkhPZt9jAZOEJEeIpIgIlcAJwKj8hnT\nkSgB7AL2eK3rm7Ks3wjUPuxduXsRmKuq1+Fqy28cdZTGF5ZgzWFUdQiuD+xDwGbgb2AA8I23yX+A\nOcBCYBEwz1uWn2ONB0Z4+5rLoUkxDtcbYR3uynpHvAtIWfaxFTjX23YrrgfAuaq6JT8xHaG7cRfQ\nduNa1yOyrH8MGCoiO0Tk8mA7E5ELgDNxZRFwv4eTRaRnyCI2EWM3GhhjTJhYC9YYY8LEEqwxxoSJ\nJVhjjAkTS7DGGBMmNtBEmBQsnqhFylbyO4xs1SpbNPhGPvlt7U6/Q8hVpSj+7BILF/A7hGyt/WsN\nW7duCdY3OM/iS9ZQTTvsBrjDaMrmsap6ZqiOmx+WYMOkSNlKtBv0gd9hZOujXi38DiFHTe79zu8Q\ncvXANc39DiFHF55YJfhGPujesU1I96dpKRSqH7THG/sWvBrsjrqwswRrjIktIhAX73cUeWIJ1hgT\neyQ2Lh9ZgjXGxB4JWUk3rCzBGmNijJUIjDEmPISYKRHERpTGGHOQuBJBsEewvYi8JyKbRGRxwLL/\nishSEVkoIl+LSGLAuvtFZIWI/CEiZ+QlUkuwxpjYI3HBH8F9gBu5LNB4oLGqNsHNwXY/gIicCFwJ\nNPLe85qIBK1TWII1xsQYrwYb7BGEqv6MGwYzcNm4gNk4ZgBVvecXAJ960/ysAlbg5qLLlSVYY0xs\nEfJaIkgSkTkBj/5B9pzVtcAP3vMquHGRM63l0CmJsmUXuYwxsSdvJYAtqtoyX7sXeRBIA4ZnLspm\ns6CDaVuCNcbEGIH48HXTEpHeuBkyTtd/ZyRYC1QL2KwqeZiHzhJslKmSWJj7u9U9+LpSycIMm72W\nBhWKUzWxMADFCyaw50AaAz5fnNNuwmLgTdcxbsxoksqVZ+qsBQBs37aN6/r04K+/1lC9eg3eHfoJ\niaVLRzSuTP0616FH+5oI8PG01bwz6U9OrFKSwVc1p2iheNZuS2bA+3PYsy+3GcnD44GLOlC4aHHi\n4uOIi0/ggfdHMnfC94x690U2rF7BoHe/oUbDJhGPC+D2W65n/JjRJJUrx08z3O/18YcGMX7MKAoU\nLEjNWrV54dV3KJWYGGRPERLGbloiciZwH9BRVZMDVo0EPhaRIbhZjOsBs4Ltz2qwUeafHfsY8Pli\nBny+mIFfLGZfWjq/rNzG4PErDi6funIbv6zcHvHYruzZmxFfHzqP4ItDnuO0jl2YveB3TuvYhReH\nPJfDu8OrfqUS9Ghfk3OenUy3pyfS9aSK1CpXjP9efTJPf7uYrk9N5IcF67mpaz1f4gO489WPeejD\n0Tzw/kgAKtepzw3PvE7dZkGvlYTVFT2u4ZMvD/29dux8OpNnLGDSL/OoXaceLw151qfochCablqf\n4GZRri8ia0WkH/AKbiLL8SKyQETeAFDVJcBnwG/AGOCWgKndc2QJNoo1q1KK9Tv3s2nPgUOWn1a3\nDJNXRGI+v0O163AqpUsfOoP0D99/xxU9ewFwRc9ejB41MuJxAdSrWIJ5q7axLzWd9AxlxvItnNms\nMnXKF2fG8q0ATFm6ibObV/YlvuxUqlmXijXq+B0GbdufethZR6fTu5GQ4E5wW5zSmvXrIjEDel6F\nrBfBVapaSVULqGpVVX1XVeuqajVVbeY9bgzY/ilVraOq9VX1h9z2nckSbBTrWLcMP63YesiyxpVK\nsD05lXU79/sU1aE2b95IxYpu3NuKFSuxZcsmX+JYun43beomUbpYQQoXiKdLo4pULl2EP9bvonsT\nF9+5zatQuXQRX+ITEV687Rqe7nMeU7752JcY8uuTjz6gS7c89auPnND0gw27Y74GKyIDcXPVz1PV\nw6Y+FpFOwN2qem6kY8tNQpzQumZp3p/59yHLO9Ure1jSNbBiw25eHb+MT25tz979afz2z07S05U7\nh83jycubcMfZ9Rm3cAOpaf7MonzPm1+QWK4Cu7Zt4cXbelGxRh3qNW/tSyxH4oX/PkNCQgKXXN7D\n71D+lccSQDQ45hMscDNwltc5OGa0rJ7In1uS2ZHy7wWZOIF2tcow8IvIXtzKTblyFdiwYT0VK1Zi\nw4b1JCWV9y2WT39Zw6e/rAFg0Pknsn5HCn9u3EOPl38BoHb54pzeuIIvsSWWc8ctWSaJZh3PYNVv\nv0Z9gh3x8YeMHzuaz0eORaItocXIYC/R0Y4OE69AXRsYKSL3icgvIjLf+7d+Ntt39ArbC7ztSnjL\n7xGR2d79yY9HIvZOdcsyefmhddbmVUuxdkcKW/YeyOFdkXfm2ecyYvgwAEYMH8ZZ55znWyxlixcE\noHLpIpzVrDLfzF57cJkI3HZWfYZNWR3xuPanJLNv756Dz3+fOYUqtQ/784sqE38cyysv/B9DP/2K\nokWjbZocsRJBNFDVG71uF52BA8DzqpomIl2Bp4FLsrzlbtzVwWkiUhzYJyLdcV0yWuE6iIwUkdO8\n2+wO4d0p0h+gcJmK+Y6
"text/plain": [
2018-05-31 11:49:28 +02:00
"<matplotlib.figure.Figure at 0x1a162ee240>"
2018-05-13 19:40:03 +02:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
2018-05-31 11:49:28 +02:00
"#clf.fit(mul_tfidf_train, mul_y_train)\n",
"pred = clf_model_b.predict(mul_tfidf_test)\n",
2018-05-13 19:40:03 +02:00
"score = metrics.accuracy_score(mul_y_test, pred)\n",
"print(\"accuracy: %0.3f\" % score)\n",
"cm = metrics.confusion_matrix(mul_y_test, pred, labels=['false', 'barely-true' , 'half-true' , 'mostly-true' , 'true'])\n",
"plot_confusion_matrix(cm, classes=['false', 'barely-true' , 'half-true' , 'mostly-true' , 'true'])"
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 40,
2018-05-13 19:40:03 +02:00
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
2018-05-31 11:49:28 +02:00
"#clf_model_b = MultinomialNB()"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 41,
2018-05-13 19:40:03 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"accuracy: 0.231\n",
"Confusion matrix, without normalization\n"
]
},
{
"data": {
2018-05-15 10:28:16 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAVgAAAEmCAYAAAAnRIjxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xd8FNUWwPHfSUIVhEAIHeldQUCU\nakdQivhAlCJiQRTbU1RUxIIFnr2giA0EVCyoiIgigggiHUR6FVB6kRBqyHl/3AkuIY2Q3dmF8/Wz\nn+zOzM6cbOTsnXPv3BFVxRhjTM6L8jsAY4w5VVmCNcaYILEEa4wxQWIJ1hhjgsQSrDHGBIklWGOM\nCRJLsCakRCSfiHwjIv+IyGcnsZ8uIvJDTsbmFxFpJiLL/Y7D5DyxcbAmLSLSGbgPqA4kAAuAZ1R1\n2knutxtwF9BYVZNOOtAwJyIKVFHVVX7HYkLPWrDmOCJyH/AK8CxQHCgHvAm0y4HdnwWsOB2Sa1aI\nSIzfMZggUlV72OPoAygE7AU6ZrBNHlwC/tt7vALk8dZdBGwE7ge2ApuAHt66J4FDwGHvGDcDTwAj\nA/ZdHlAgxnt9I7AG14peC3QJWD4t4H2NgdnAP97PxgHrpgADgOnefn4A4tL53VLifzAg/quBK4EV\nwE7gkYDtGwIzgN3etm8Aub11U73fJdH7fTsF7P8hYDMwImWZ955K3jHqea9LAduBi/z+f8MeJ/6w\nFqxJrRGQF/gyg20eBS4A6gJ1cEmmX8D6ErhEXRqXRAeLSKyqPo5rFY9W1QKq+l5GgYjIGcBrQCtV\nLYhLogvS2K4I8K23bVHgJeBbESkasFlnoAcQD+QG+mRw6BK4z6A00B94B+gK1AeaAf1FpKK37RHg\nv0Ac7rO7FLgDQFWbe9vU8X7f0QH7L4JrzfcMPLCqrsYl31Eikh/4ABimqlMyiNeEKUuwJrWiwHbN\n+BS+C/CUqm5V1W24lmm3gPWHvfWHVXU8rvVWLZvxJAO1RSSfqm5S1cVpbHMVsFJVR6hqkqp+DCwD\n2gRs84GqrlDV/cCnuC+H9BzG1ZsPA5/gkuerqprgHX8xcA6Aqs5V1d+8464D3gYuzMLv9LiqHvTi\nOYaqvgOsBGYCJXFfaCYCWYI1qe0A4jKpDZYC/gx4/ae37Og+UiXofUCBEw1EVRNxp9W9gE0i8q2I\nVM9CPCkxlQ54vfkE4tmhqke85ykJcEvA+v0p7xeRqiIyTkQ2i8geXAs9LoN9A2xT1QOZbPMOUBt4\nXVUPZrKtCVOWYE1qM4ADuLpjev7Gnd6mKOcty45EIH/A6xKBK1X1e1W9HNeSW4ZLPJnFkxLTX9mM\n6US8hYuriqqeCTwCSCbvyXDojogUwNW13wOe8EogJgJZgjXHUNV/cHXHwSJytYjkF5FcItJKRP7n\nbfYx0E9EiolInLf9yGwecgHQXETKiUgh4OGUFSJSXETaerXYg7hSw5E09jEeqCoinUUkRkQ6ATWB\ncdmM6UQUBPYAe73W9e2p1m8BKh73roy9CsxV1VtwteUhJx2l8YUlWHMcVX0JNwa2H7AN2ADcCXzl\nbfI0MAf4HVgEzPOWZedYE4HR3r7mcmxSjMKNRvgb17N+IV4HUqp97ABae9vuwI0AaK2q27MT0wnq\ng+tAS8C1rkenWv8EMFxEdovItZntTETaAS1xZRFwf4d6ItIlxyI2IWMXGhhjTJBYC9YYY4LEEqwx\nxgSJJVhjjAkSS7DGGBMkNtFEkMTkL6S5ChX3O4w0VYo/4TH/IbNk7Ta/Q8hQmVKF/Q4hXUXz5/E7\nhDStX7+OHdu3ZzY2OMuizzxLNem4C+COo/u3fa+qLXPquNlhCTZIchUqTqWbB/sdRpo+793Y7xDS\ndW73t/wOIUMPPv4fv0NIV9d65fwOIU0XNzk/R/enSfvJUy3TEW8cWDA4syvqgs4SrDEmsohAVLTf\nUWSJJVhjTOSRyOg+sgRrjIk8kmMl3aCyBGuMiTBWIjDGmOAQrERgjDHBIVYiMMaYoLEWrDHGBIPV\nYI0xJjgEKxEYY0zQWInAGGOCQSDaSgQmm7o1Lsd/6pdGgZVb9tJvzGL+U7803RqXo1zR/DR9dgq7\n9x0OeVyb/tpI33tuZfvWLUhUFNd27cENt/Tm+aceZfLE8eTKnZuyZ1Xg2ZeHcGah4E+KMuS+K2h1\nQSW27d5Hg57DAIgtmJcRj7bmrOKF+HPLP3R9+ht27z1I60aV6N+9KcmqJB1J5sG3JvPr4lDcE/Ff\nyUeO8Pwt7ShUrDi9/vcey+dM56s3B6LJyeTJl5+ujz5PsTLlQxpTakMGv8bwD94DVW7ocTO333mP\nr/GkKYKGaUVGlKeR+IJ56NKoHJ3emkn712cQJdDq7OLMX7+bWz6Yy1+7Mp9FKFiiY2J4sP9zfDt1\nHqPHTeajYe+wasVSGje/hLGTZ/P1pJmUr1iFoa+/GJJ4RkxcTLtHPj9mWZ9ODZkyfz1n93iPKfPX\n06eTm2hk8vz1NOw1nAtu/5BeL07gzftahCTGQFM++4DiZ1U6+nr0C4/Rvf/L9B32LfUvb8uE4f5O\nDrRk8R8M/+A9Jk2dwS8z5/H9d9+yetVKX2NKl0jmjzBgCTYMxUQJeXJFER0l5MsVzbaEgyzblMDf\nuw/4Gld88RLUOqcuAGcUKEilytXYsmkTTS66lJgYdzJUp/55bNkUmpbh9EUb2Zlw7GfSulFlRk5c\nDMDIiYtp07gyAIkH/m3xn5E3F6G+Fd2urZtYPGMyjdp0OrpMRDiQuBeAA4kJFIqLD21QqaxYvozz\nzjuf/PnzExMTQ5OmzRk39qvM3xhy3iiCzB5hwEoEYWZrwkGGTVvHj32acSApmV9X7eDXVTv9Dus4\nf234k6V/LKROvQbHLB/z8QhatfNvSr/42Pxs3pkIwOadiRQrnP/ourZNKvPUTc0oVig/1zw2JqRx\njXltAO1u78vBfYlHl13f9zneeuAmcufJS94zCnDf21+ENKbUatSsxdNPPMbOHTvImy8fE7//jrqp\n/r5hw0oE4UFE7haRpSIyKp31F4nIuLTW+eHMvDFcXCOeK16cxiWDppIvVzSt65TwO6xjJCbu5e5b\nutD3qUEUKHjm0eVDXv0f0THRtLmmUwbv9s/Y6auoe/MHXPvk1/Tv3jRkx/1j+iQKFC5KuepnH7N8\n8uj3uf359xnw5a+cf2UHvnz9mZDFlJZq1Wtwz30P0L51Szq0u5JaZ9chJiY8WoLHyEp5IExKBKdD\nC/YOoJWqrvU7kKy4oFIR/tq1n11eJ9akJVupW64w4xZu9jky5/Dhw9xzSxfaXNOJFle2O7r8q09H\nMeXHCXwwehzi4//cW3fto0SRM9i8M5ESRc5g2+59x20zfdFGKpYqTNEz87FjT/Br2msWzeWP6ZNY\n8tsUDh86yIHEvbz1wE1s/XMN5Wu5kku9S67irT49gh5LZrrdeBPdbrwJgKf6P0qp0mV8jigdYVIC\nyMwp3YIVkSFARWCsiDwkIr+KyHzvZ7U0tr9QRBZ4j/kiUtBb/oCIzBaR30XkyWDGvOmfA5xTphB5\nc7k/zfmVirBmW2Im7woNVaXf/XdQsUo1brztrqPLf5k8kXcHv8Sbw0aTL3/+DPYQfN/+tpqul9cC\noOvltRg3YxUAFQNu9VK3cjy5Y6JCklwB2vZ6kAFf/sqTn/9Cjydeo2r9RvR8bij7ExPYun4NAMvn\nTDumA8wv27ZuBWDDhvWMG/sVHa69zueI0iKuRJDZI7O9iLwvIltF5I+AZUVEZKKIrPR+xnrLRURe\nE5FVXh6ol5VIT+kWrKr2EpGWwMXAIeBFVU0SkcuAZ4HUxcI+QG9VnS4iBYADItICqAI0xA0QGSsi\nzVV1aurjiUhPoCdArjOz12GxaOMeJi7ewqd3XMCRZGXZpj18NnsjXS4oS49m5YkrkJsxdzbilxXb\nefyrJdk6RnbNmzWDsZ9
2018-05-13 19:40:03 +02:00
"text/plain": [
2018-05-31 11:49:28 +02:00
"<matplotlib.figure.Figure at 0x1a16437780>"
2018-05-13 19:40:03 +02:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
2018-05-31 11:49:28 +02:00
"#clf_model_b.fit(mul_tfidf_train, mul_y_train)\n",
"pred = clf_model_b.predict(mul_tfidf_valid)\n",
2018-05-13 19:40:03 +02:00
"score = metrics.accuracy_score(mul_y_valid, pred)\n",
"print(\"accuracy: %0.3f\" % score)\n",
"cm = metrics.confusion_matrix(mul_y_valid, pred, labels=['false', 'barely-true' , 'half-true' , 'mostly-true' , 'true'])\n",
"plot_confusion_matrix(cm, classes=['false', 'barely-true' , 'half-true' , 'mostly-true' , 'true'])"
]
},
2018-05-31 11:49:28 +02:00
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"precision = cm[1,1]/(cm[1,1]+cm[0,1])\n",
"recall = cm[1,1]/(cm[1,1]+cm[1,0])"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"configuration_2 = {\n",
" \"name\":\"configuration2\",\n",
" \n",
" \"model\":\"model_a\",\n",
" \"model-name\":\"multinomialNB\",\n",
"\n",
"\n",
" \"dataset\":\"X2\",\n",
" \"dataset-name\":\"liar\",\n",
" \"dataset-link\":\"unknown\",\n",
" \n",
" \"measurement-name\":\"measurement_config2\",\n",
" \n",
" \"execution-name\":\"execution_config2\",\n",
" \n",
" \"precision\":precision,\n",
" \"recall\":recall,\n",
" \"accuracy\":score\n",
"}"
]
},
2018-05-13 19:40:03 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
2018-05-31 11:49:28 +02:00
"## Configuration 3\n",
"**In contrast to the discussed true false FAKE REAL bin class approch, I just want to see the expected horible results :)**\n",
2018-05-13 19:40:03 +02:00
"* model a - test - dataset2 - [performance measures]\n",
"* model b - test - dataset1 - [performance measures]"
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 44,
2018-05-13 19:40:03 +02:00
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
2018-05-31 11:49:28 +02:00
"clf_tmp = MultinomialNB()"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 45,
2018-05-13 19:40:03 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2018-05-15 10:28:16 +02:00
"0 true\n",
"1 false\n",
"2 false\n",
"3 half-true\n",
"4 pants-fire\n",
2018-05-13 19:40:03 +02:00
"Name: y, dtype: object\n"
]
}
],
"source": [
"#print(type(bin_tfidf_train),type(mul_tfidf_test))\n",
"\n",
"tmp_mul_tfidf_test = bin_tfidf_vectorizer.transform(mul_X_test)\n",
"\n",
"print(mul_y_test[:5])"
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 46,
2018-05-13 19:40:03 +02:00
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
2018-05-31 11:49:28 +02:00
"clf_tmp.fit(bin_tfidf_train, bin_y_train)\n",
"pred = clf_tmp.predict(tmp_mul_tfidf_test)"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 47,
2018-05-13 19:40:03 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1267\n",
"['true' 'true' 'true' 'true' 'true' 'true' 'true' 'true' 'true' 'true'\n",
" 'true' 'true' 'true' 'true' 'true' 'true' 'false' 'true' 'true' 'true'\n",
2018-05-31 11:49:28 +02:00
" 'true' 'true' 'true' 'false' 'false' 'true' 'true' 'true' 'true' 'false']\n"
2018-05-13 19:40:03 +02:00
]
}
],
"source": [
"print(len(pred))\n",
"pred = np.array(pred, dtype=object)\n",
"\n",
"pred[pred == \"FAKE\"] = \"false\"\n",
"pred[pred == \"REAL\"] = \"true\"\n",
"\n",
"print(pred[:30])\n",
"#pred[pred['FAKE']==false]"
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 48,
2018-05-13 19:40:03 +02:00
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2018-05-31 11:49:28 +02:00
"accuracy: 0.170\n",
2018-05-13 19:40:03 +02:00
"Confusion matrix, without normalization\n"
]
},
{
"data": {
2018-05-31 11:49:28 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAVgAAAEmCAYAAAAnRIjxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XeYFFXWx/HvjyGIgqISlBkRCUpS\nkiBGUAwgSVQWEAMmzGFdVFRcMGd9ZU3rqmtWzKgo4Lq6ZkmiiAkUFAZEQFFA4nDeP24NNMOEnqF7\nups5H59+6K6qrjrTwpnbp26QmeGccy7xKqU6AOec21p5gnXOuSTxBOucc0niCdY555LEE6xzziWJ\nJ1jnnEsST7CuXEmqLuk1Sb9Len4LzjNI0oRExpYqkg6W9G2q43CJJ+8H6woj6QTgEqAZsAyYBtxg\nZh9s4XlPAi4ADjCzdVscaJqTZEBTM5uV6lhc+fMWrNuMpEuA/wNuBOoBDYD7gD4JOP3uwHcVIbnG\nQ1LlVMfgksjM/OGPDQ9gB2A50K+YY6oREvD86PF/QLVoXxdgHvA34BdgAXBqtO8aYA2wNrrG6cBI\n4MmYczcEDKgcvR4M/EBoRc8GBsVs/yDmfQcAk4Dfoz8PiNn3LnAd8GF0nglA7SJ+tvz4L4uJ/xjg\naOA74FfgypjjOwIfA0ujY+8Bqkb73ot+lhXRz9s/5vyXAz8DT+Rvi97TOLpGu+h1fWAx0CXVfzf8\nUfqHt2BdQfsD2wAvF3PMVUAnoA3QmpBkhsfs34WQqLMJSfReSTua2QhCq3i0mdUws4eLC0TSdsAo\noLuZ1SQk0WmFHLcTMDY6dmfgTmCspJ1jDjsBOBWoC1QFhhZz6V0In0E28HfgX8CJQHvgYODvkhpF\nx+YBfwVqEz67rsC5AGZ2SHRM6+jnHR1z/p0IrfkhsRc2s+8JyfcpSdsC/wYeNbN3i4nXpSlPsK6g\nnYHFVvxX+EHAtWb2i5ktIrRMT4rZvzbav9bM3iC03vYqYzzrgVaSqpvZAjObUcgxPYCZZvaEma0z\ns2eAb4BeMcf828y+M7OVwHOEXw5FWUuoN68FniUkz7vNbFl0/RnAPgBmNsXMPomuOwf4J9A5jp9p\nhJmtjuLZhJn9C5gJfArsSviF5jKQJ1hX0BKgdgm1wfrAjzGvf4y2bThHgQT9J1CjtIGY2QrC1+qz\ngQWSxkpqFkc8+TFlx7z+uRTxLDGzvOh5fgJcGLN/Zf77Je0p6XVJP0v6g9BCr13MuQEWmdmqEo75\nF9AK+IeZrS7hWJemPMG6gj4GVhHqjkWZT/h6m69BtK0sVgDbxrzeJXanmY03syMILblvCImnpHjy\nY8otY0ylcT8hrqZmtj1wJaAS3lNs1x1JNQh17YeBkVEJxGUgT7BuE2b2O6HueK+kYyRtK6mKpO6S\nbo0OewYYLqmOpNrR8U+W8ZLTgEMkNZC0A3BF/g5J9ST1jmqxqwmlhrxCzvEGsKekEyRVltQfaAG8\nXsaYSqMm8AewPGpdn1Ng/0Kg0WbvKt7dwBQzO4NQW35gi6N0KeEJ1m3GzO4k9IEdDiwC5gLnA69E\nh1wPTAa+AKYDU6NtZbnWW8Do6FxT2DQpViL0RphPuLPemegGUoFzLAF6RscuIfQA6Glmi8sSUykN\nJdxAW0ZoXY8usH8k8JikpZL+UtLJJPUBuhHKIhD+P7STNChhEbty4wMNnHMuSbwF65xzSeIJ1jnn\nksQTrHPOJYknWOecSxKfaCJJdtq5tmXv1iDVYRSqapb/Xt0afTbrl1SHUChb+Su2ZnlJfYPjlrX9\n7mbrNhsAV8h1F403s26Jum5ZeIJNkuzdGvDyhA9THUahcnaqnuoQXBLs2HtUqkMo1OoPb0/o+Wzd\nSqrtVWKPN1ZNu7ekEXVJ5wnWOZdZJKiUleoo4uIJ1jmXeZQZZS5PsM65zKOElXSTyhOscy7DeInA\nOeeSQ3iJwDnnkkNeInDOuaTxFqxzziWD12Cdcy45hJcInHMuabxE4JxzySDIyowSQWb8GqhAVq9a\nxXFHHUyvQ/ej+yHtufvW6wC44uKz6XXofvTs0pHzTz+BFSuWpzhSmDB+HPu03IuWzZpw2603pzqc\nTXhsRcupXYNxNx3LZw+cyJT7B3Fen9YAHHtQE6bcP4gVr19Au6Z1N3lPq4Y78+4d/Zhy/yAm3XcC\n1aqkMMHld9Mq6ZEGvAWbZqpWq8bjL73JdtvVYO3atQzo1ZVDDjuKK6+7lZo1twfgxr9fzpMPP8BZ\nFw5NWZx5eXlcfOF5jH3zLbJzcjioUwd69uxN8xYtUhaTxxafdXnrGfbQ+0z7fhE1qlfho1EDeHvq\nXGb8uIQB14/lngsO2+T4rErikUuP4vTbJzB99mJ2qrkNa/PWl1u8hcqQGmx6pHm3gSS2264GAOvW\nrmXdurVIbEiuZsaqVStT/hds0sSJNG7chD0aNaJq1ar06z+A118bk9KY8nlsxfv5tz+Z9v0iAJav\nXMs3P/1G/drb8e3c35iZu3Sz4w9v14AvZy9m+uywhuSvy1axfn0q1/KLehGU9EgDnmDTUF5eHr0O\n249OLXfnwM5dadO+IwCXXzSE/VvtwQ8zv+Pk0wuuDl2+5s/PJSdntw2vs7NzyM3NTWFEG3ls8WtQ\ntyZtGtdh0jcLizymafaOGPDqdX34aNQALjm+XfkFWJQMKRGkRxRJJOlCSV9LeqqI/V0kvV7YvlTJ\nysritf9+yvvTZvLF1Ml89/UMAG65+0E+/OJ7Gu+5F2PHvJDSGAtbjVhp8rXNY4vPdttU4ZmrenDp\ng++xbOWaIo+rnCUOaLErp942nq6XvkDv/RvTpXVOOUZagBTfIw1s9QkWOBc42swybl357XeoxX4H\nHsx777y1YVtWVhZH9zme8a+/ksLIQstr3ry5G17n5s6jfv36KYxoI4+tZJWzKvHMVUcz+t1vGfPR\n98Uem7t4Oe9Pz2XJH6tYuXod4ybPoW2TusW+J+m8RJB6kh4AGgGvSrpc0keSPov+3KuQ4ztLmhY9\nPpNUM9p+qaRJkr6QdE0yY16yeBF//B7qYKtWruSj995hj8ZN+XF2+EdgZrwz4Q0aN90s/HK1b4cO\nzJo1kzmzZ7NmzRqeH/0sPXr2TmlM+Ty2kj1wcVe+nfsro17+rMRj35r6E632qE31apXJqiQObpXN\n1z/9Wg5RFkUZUyLYqnsRmNnZkroBhwJrgDvMbJ2kw4EbgeMKvGUocJ6ZfSipBrBK0pFAU6AjoYPI\nq5IOMbP3Cl5P0hBgCED9mDpbaSxa+DOXXXgm6/PWs379err3OZZDj+jOwN6Hs3zZMsyMZi335ppb\n7y7T+ROlcuXK3HX3PfTqcRR5eXmcMvg0WrRsmdKY8nlsxTugxa4M6tqc6bMX88k/BgIw4rGPqFYl\nizvP6ULtHarz0sjefPHDInpfPYaly1cz6uXP+OD/+mMG4yfPYdykOeUa82bSpARQEhVWE9qaSJoD\n7AtUB0YRkqUBVcysmaQuwFAz6ylpGNAXeAp4yczmSbodOB7Iv71aA7jJzB4u7rp7t2lnviaXK0/p\nvCbX+t9/SlhGrFSrgVU76LISj1s19oIpZrZvoq5bFlt1C7aA64B3zKyvpIbAuwUPMLObJY0FjgY+\niVq6IiTUf5ZjrM65ImXOZC/pUagoHzsA+f1hBhd2gKTGZjbdzG4BJgPNgPHAaVHJAEnZklJc4Xeu\ngvMabNq5FXhM0iXAf4s45mJJhwJ5wFfAm2a2WlJz4OOoO81y4EQgPRehd64iyJAa7FafYM2sYfR0\nMbBnzK6ro/3vEpULzOyCIs5xN5Dau0rOucCX7XbOueRJl4EjJfEE65zLKGG+bU+wzjmXeBKq5AnW\nOeeSIlNasOnRl8E550pBUomPOM6xm6R3osmgZki6KNq+k6S3JM2M/twx2i5JoyTNiobNlzitmCdY\n51xmEaiSSnzEYR3wNzN
2018-05-13 19:40:03 +02:00
"text/plain": [
2018-05-31 11:49:28 +02:00
"<matplotlib.figure.Figure at 0x1a164043c8>"
2018-05-13 19:40:03 +02:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"score = metrics.accuracy_score(mul_y_test, pred)\n",
"print(\"accuracy: %0.3f\" % score)\n",
"cm = metrics.confusion_matrix(mul_y_test, pred, labels=['false', 'barely-true' , 'half-true' , 'mostly-true' , 'true' ])\n",
"plot_confusion_matrix(cm, classes=['false', 'barely-true' , 'half-true' , 'mostly-true' , 'true'])"
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 49,
"metadata": {
"collapsed": true
},
2018-05-13 19:40:03 +02:00
"outputs": [],
"source": [
2018-05-31 11:49:28 +02:00
"clf_tmp = MultinomialNB()\n",
2018-05-15 10:28:16 +02:00
"#from sklearn.neural_network import MLPClassifier\n",
2018-05-13 19:40:03 +02:00
"\n",
2018-05-15 10:28:16 +02:00
"#clf = MLPClassifier(hidden_layer_sizes=(40,20))"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 50,
2018-05-13 19:40:03 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0 true\n",
"1 false\n",
"2 false\n",
"3 half-true\n",
"4 pants-fire\n",
"5 true\n",
"6 true\n",
"7 barely-true\n",
"8 true\n",
"9 barely-true\n",
"Name: y, dtype: object\n"
]
}
],
"source": [
"tmp_bin_tfidf_test = mul_tfidf_vectorizer.transform(bin_X_test)\n",
"\n",
"print(mul_y_test[:10])"
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 51,
"metadata": {
"collapsed": true
},
2018-05-13 19:40:03 +02:00
"outputs": [],
"source": [
2018-05-31 11:49:28 +02:00
"clf_tmp.fit(mul_tfidf_train, mul_y_train)\n",
"pred = clf_tmp.predict(tmp_bin_tfidf_test)"
2018-05-13 19:40:03 +02:00
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 52,
2018-05-13 19:40:03 +02:00
"metadata": {},
2018-05-15 10:28:16 +02:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['half-true' 'false' 'false' 'half-true' 'half-true' 'barely-true' 'false'\n",
" 'mostly-true' 'half-true' 'false' 'half-true' 'half-true' 'false' 'false'\n",
" 'false' 'half-true' 'half-true' 'half-true' 'false' 'true']\n"
]
}
],
2018-05-13 19:40:03 +02:00
"source": [
"print(pred[:20])\n",
"\n",
"bin_y_test = np.array(bin_y_test, dtype=object)\n",
"\n",
"bin_y_test[bin_y_test == \"FAKE\"] = \"false\"\n",
"bin_y_test[bin_y_test == \"REAL\"] = \"true\""
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 53,
2018-05-13 19:40:03 +02:00
"metadata": {},
2018-05-15 10:28:16 +02:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"accuracy: 0.220\n",
"Confusion matrix, without normalization\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAVgAAAEmCAYAAAAnRIjxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzsnXeYVFXSh9/fzACCIEFEJShJRWUF\nBXNCREVFMWFCBXNEXcMa1uy64rrqZ17jmhNGVDDrohgAERExgIIKIgoKAhKH+v44Z6AZZ7p7hu7p\nbqjX5z7ce8+551S3UF23Tp0qmRmO4zhO5inKtQCO4zirKq5gHcdxsoQrWMdxnCzhCtZxHCdLuIJ1\nHMfJEq5gHcdxsoQrWKdGkVRX0ouSZksatBLj9JX0WiZlyxWSdpb0Va7lcDKPPA7WqQhJRwLnAB2A\nOcAY4Boze28lxz0aGADsYGZLVlrQPEeSARuZ2cRcy+LUPG7BOn9C0jnA/wH/BNYFNgDuAHpnYPgN\nga9XB+WaDpJKci2Dk0XMzA8/lh1AQ2Au0CdJnzoEBfxjPP4PqBPbugFTgHOBn4FpwLGx7UpgEbA4\nznE8cAXwSMLYrQEDSuJ1f+BbghU9CeibcP+9hOd2AEYCs+OfOyS0vQNcDQyP47wGNK3ks5XJ/7cE\n+Q8A9gG+Bn4FLk7ovw3wATAr9r0NqB3bhsXPMi9+3sMSxr8A+Al4uOxefKZdnGOreN0cmAF0y/Xf\nDT+qfrgF65Rne2AN4Lkkff4ObAd0BjoRlMwlCe3rERR1C4ISvV1SYzO7nGAVP2lm9c3svmSCSFoT\nuAXY28waEJTomAr6NQFejn3XBm4EXpa0dkK3I4FjgWZAbeC8JFOvR/gOWgCXAfcARwFdgJ2ByyS1\njX1Lgb8CTQnf3e7AaQBmtkvs0yl+3icTxm9CsOZPSpzYzL4hKN9HJdUD/gs8YGbvJJHXyVNcwTrl\nWRuYYclf4fsCV5nZz2b2C8EyPTqhfXFsX2xmQwjW2ybVlGcp0FFSXTObZmafV9BnX2CCmT1sZkvM\n7HHgS2C/hD7/NbOvzWw+8BThx6EyFhP8zYuBJwjK82YzmxPn/xzYAsDMPjazD+O8k4G7gF3T+EyX\nm9nCKM8KmNk9wATgI2B9wg+aU4C4gnXKMxNomsI32Bz4LuH6u3hv2RjlFPQfQP2qCmJm8wiv1acA\n0yS9LKlDGvKUydQi4fqnKsgz08xK43mZApye0D6/7HlJG0t6SdJPkn4nWOhNk4wN8IuZLUjR5x6g\nI3CrmS1M0dfJU1zBOuX5AFhA8DtWxo+E19syNoj3qsM8oF7C9XqJjWb2qpntQbDkviQonlTylMk0\ntZoyVYU7CXJtZGZrARcDSvFM0tAdSfUJfu37gCuiC8QpQFzBOitgZrMJfsfbJR0gqZ6kWpL2lvSv\n2O1x4BJJ60hqGvs/Us0pxwC7SNpAUkPgorIGSetK2j/6YhcSXA2lFYwxBNhY0pGSSiQdBmwGvFRN\nmapCA+B3YG60rk8t1z4daPunp5JzM/CxmZ1A8C3/Z6WldHKCK1jnT5jZjYQY2EuAX4AfgDOA52OX\nfwCjgLHAZ8DoeK86c70OPBnH+pgVlWIRIRrhR8LK+q7EBaRyY8wEesW+MwkRAL3MbEZ1ZKoi5xEW\n0OYQrOsny7VfATwoaZakQ1MNJqk30JPgFoHw/2ErSX0zJrFTY/hGA8dxnCzhFqzjOE6WcAXrOI6T\nJVzBOo7jZAlXsI7jrJZIWkPSCEmfSvpc0pXx/gOSJkkaE4/O8b4k3SJpoqSxkrZKNYcnmsgSqlXP\nVKdhrsWokI7tm6fulCPGTfol1yIkZaNW+RuSWrd2ca5FqJAfvv+OmTNmpIoNTpvitTY0W/KnDXB/\nwub/8qqZ9UzSZSHQ3czmSqoFvCdpaGw738yeLtd/b2CjeGxLiIHeNpkMrmCzhOo0pM5f+uVajAp5\n6aUrci1CpWxy9N25FiEpd9x4ZK5FqJROLRvlWoQK6bFLUh1UZWzJfOpskjLijQVjbk+6o85CCNXc\neFkrHsnCqnoDD8XnPpTUSNL6ZjatsgfcReA4TmEhQVFx6iNs+R6VcJz056FULGkMIXPa62b2UWy6\nJroBbpJUJ95rQYgJL2MKK27H/hNuwTqOU3goLdtwhpl1TdYh5pzoLKkR8JykjoTdhD8Rsq7dTchu\ndhUVb4FOupHALVjHcQoPKfVRBcxsFiFvcM+Ytc1ikp3/EtJxQrBYWyU81pIUOThcwTqOU2Ck7SJI\nPkrIpdEontcFegBfSlo/3hMh6dG4+Mhg4JgYTbAdMDuZ/xXcReA4TqEh0nURpGJ9Qp6IYoKx+ZSZ\nvSTpLUnrxJnGsDwvxBBCZYuJhJSXx6aawBWs4zgFRtVdABVhZmOBLSu4372S/gacXpU5XME6jlN4\nZMaCzTquYB3HKTCUlo81H3AF6zhOYSEy4iKoCVzBOo5TeLiLwHEcJxsIigvDRVAYPwOrOHVql/Du\nfafz0UNn8fGjf+WSE3qs0H7jOfvzy5tXLrvesXMb3n9gAHPevYYDd+tYY3IuWLCA/ffYiZ67bkOP\nHbfixoFXAzB82Dvss9v27LFTF845/QSWLElW8Ttz1KlVzLs3HcpHtx3Bx3f25ZK+Yc/7Kb22YNy9\nxzB/yJmsvdYay/qvVa82T1++37L+R++xaY3ICfDsQ3dxwn47c3yvnXjmwVBi65svxzHg8L05Yf9d\nuOTUvsybO6fG5Clj6pQfOGCfHuzQ5S/stHUn7rrjFgBeeO5pdtq6E83Wqs2Y0aNqXK6klIVppTry\nALdg84CFi5bQ84x7mDd/ESXFRbx11ym89sFXjPj8B7bq0IKG9ddYof8PP83ipKsHcXbfXWpUzjp1\n6vD4c6+wZv36LF68mEP27c4u3Xtw7hkn8NizQ2nbfiNuuPYqnn7iEQ4/qn/W5Vm4uJSeFz3HvAWL\nw/f270N4bdR3fDB+GkNGTOK16w5eof/Jvbbgy+9ncsiVL9J0rbp8es/RPPH2VyxesjSrck76+guG\nDHqE2556lVq1anPhiYex7a57cMOlf+Xk86+g0zY7MvSZR3nqvts49qyLUg+YQYpLSrjyn/+iU+et\nmDtnDrvvvC3duvdg000354FHn+Lcs/5UAi0/KBAfbH6oeYd58xcBUKukmJKSYsygqEj884x9+Pvt\nQ1fo+/1PvzHum59YurRm66lJYs369QFYsngxixcvobi4mNp16tC2/UYA7NytO0Nfej7ZMBll3oLF\nANQqKaKkuAjD+PTbX/j+5z9bgwbUr1sbgDXr1uK3OQtYUppd5Qrw/bdfs2mnLqxRtx7FJSV02noH\nhr8xhCmTJrLF1jsA0GWHbrz7ek0UwV2R9dZbn06dQ1rT+g0asPEmHZj2449s3GFT2m+8SY3Lkx6Z\n2clVE7iCzROKisSHD57J90Mu4a0RExg5/gdOPWQHXn7vC36aWfOvjpVRWlrK3t22ZatNN2Dnbt3p\nvNXWLFm8mLGffAzAkBefY9rUKTUmT1GR+PDWI/j+sRN465PvGfnV9Er7/ufFT+nQqgnfPnI8o+44\nkvPuGkZN1PxsvdGmjB31AbN/+5UF8//go2Fv8PNPU2m90aa8/9YrAAx7dTC/TJuafWGS8P13k/ls\n7Bi6dN0mdedcUyAugvyQIotIOlPSF5IeraS9m6SaNx3KsXSpsV2/W2jf+1q6btaKHTu34aDuf+GO\nQe/nWrQVKC4uZug7H/Hh2ImMGT2Kr78cz633PMRVl/6N/ffYiTXrN6C4pOY8T0uXGtsNeJz2x9xP\n143XY7MNK0+IvcdWGzL2219oe9R9bHvG49x06q40iBZtNtmw3cYcfsIALjj+EC468TDaddic4uIS\nzrvmZgY/dj+nHrw7f8ybS0mt7MtSGXPnzuXYow7lHwNvoMFaa+VMjrRIJ9FLnrgQVgcf7GnA3mY2\nKdeCpMPsuQsYNvpbdt2
"text/plain": [
2018-05-31 11:49:28 +02:00
"<matplotlib.figure.Figure at 0x1a15546a90>"
2018-05-15 10:28:16 +02:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
2018-05-13 19:40:03 +02:00
"source": [
"score = metrics.accuracy_score(bin_y_test, pred)\n",
"print(\"accuracy: %0.3f\" % score)\n",
"cm = metrics.confusion_matrix(bin_y_test, pred, labels=['false', 'barely-true' , 'half-true' , 'mostly-true' , 'true'])\n",
"plot_confusion_matrix(cm, classes=['false', 'barely-true' , 'half-true' , 'mostly-true' , 'true'])"
]
},
{
2018-05-31 11:49:28 +02:00
"cell_type": "markdown",
"metadata": {},
"source": [
"As expected not these fancy results -.- "
]
2018-05-13 19:40:03 +02:00
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configuration 4\n",
"* model c - train - [performance measures][0:4]\n",
"* model c - test - [performance measures][0:4]"
]
},
2018-05-15 10:28:16 +02:00
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 54,
2018-05-15 10:28:16 +02:00
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
2018-05-31 11:49:28 +02:00
"clf_model_c = MultinomialNB()"
2018-05-15 10:28:16 +02:00
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 55,
2018-05-15 10:28:16 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"accuracy: 0.860\n",
"Confusion matrix, without normalization\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAVYAAAEmCAYAAAA5jbhCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xm8FXX9x/HX+wICiooIKosIIqJG\ngkq4lqaluKWW+nPLJfelNJcktdwyzcyKXEpzwSWVyoXcyX1DWUQRCQERQVBABFERAT+/P2YuHvAu\n58Lce+ae+372mMc95zvfmfkOxNvv/c7MdxQRmJlZdipK3QAzs3LjYDUzy5iD1cwsYw5WM7OMOVjN\nzDLmYDUzy5iD1aokqbWk/0iaL+mfq7CfwyU9nmXbSkXStyVNKHU7LP/k+1gbN0mHAWcCmwELgDHA\nZRHx/Cru98fAT4EdImLJKjc05yQF0DMiJpW6Ldb4ucfaiEk6E/gT8FtgfaArcB2wXwa73wh4qymE\najEkNS91G6wRiQgvjXAB1gY+AQ6qoU5LkuCdkS5/Alqm63YBpgNnAbOAmcAx6bqLgS+AxekxjgUu\nAu4o2Hc3IIDm6fejgbdJes1TgMMLyp8v2G4HYAQwP/25Q8G6p4FLgRfS/TwOtK/m3Crb/4uC9u8P\n7AW8BcwFziuo3x94CZiX1r0GWC1d92x6Lp+m5/t/Bfs/F3gfuL2yLN2mR3qMrdPvnYA5wC6l/v+G\nl9Iv7rE2XtsDrYD7aqhzPrAd0BfoQxIuFxSs34AkoDuThOe1ktaJiAtJesH3RESbiLippoZIWgMY\nBOwZEWuShOeYKuq1Ax5K664LXA08JGndgmqHAccA6wGrAWfXcOgNSP4MOgO/Bm4EjgC2Ab4N/FrS\nxmndpcDPgfYkf3a7AacARMR30jp90vO9p2D/7Uh67ycUHjgiJpOE7p2SVgduAW6NiKdraK81EQ7W\nxmtdYE7U/Kv64cAlETErImaT9ER/XLB+cbp+cUQ8TNJb67WS7fkS6C2pdUTMjIhxVdTZG5gYEbdH\nxJKIuAv4H7BvQZ1bIuKtiFgIDCH5j0J1FpOMJy8G7iYJzT9HxIL0+OOALQEiYlREDE+P+w7wN2Dn\nIs7pwohYlLZnORFxIzAReBnoSPIfMjMHayP2IdC+lrG/TsDUgu9T07Jl+1ghmD8D2tS1IRHxKcmv\nzycBMyU9JGmzItpT2abOBd/fr0N7PoyIpennyuD7oGD9wsrtJW0q6UFJ70v6mKRH3r6GfQPMjojP\na6lzI9Ab+EtELKqlrjURDtbG6yXgc5JxxerMIPk1tlLXtGxlfAqsXvB9g8KVEfFYRHyfpOf2P5LA\nqa09lW16byXbVBfXk7SrZ0SsBZwHqJZtarxlRlIbknHrm4CL0qEOMwdrYxUR80nGFa+VtL+k1SW1\nkLSnpCvTancBF0jqIKl9Wv+OlTzkGOA7krpKWhv4ZeUKSetL+kE61rqIZEhhaRX7eBjYVNJhkppL\n+j9gC+DBlWxTXawJfAx8kvamT15h/QfAxl/bqmZ/BkZFxHEkY8d/XeVWWllwsDZiEXE1yT2sFwCz\ngWnAacD9aZXfACOB14GxwOi0bGWONQy4J93XKJYPwwqSuwtmkFwp35n0wtAK+/gQ2Cet+yHJFf19\nImLOyrSpjs4muTC2gKQ3fc8K6y8CBkuaJ+ng2nYmaT9gAMnwByR/D1tLOjyzFluj5QcEzMwy5h6r\nmVnGHKxmZhlzsJqZZczBamaWsbKbWELNW4dWW7PUzbA62mrzrqVugq2E0aNHzYmIDlntr9laG0Us\n+dpDblWKhbMfi4gBWR07S+UXrKutSctetd4tYznzwsvXlLoJthJat9CKT9KtkliysOh/v5+Puba2\nJ+dKpuyC1cwaM4Ea/wilg9XM8kNARbNSt2KVOVjNLF9U2xQO+edgNbMc8VCAmVn23GM1M8uQcI/V\nzCxbco/VzCxzvivAzCxLvnhlZpYt4aEAM7PMucdqZpYlDwWYmWWvwkMBZmbZ8VwBZmZZ81CAmVn2\nfFeAmVnG3GM1M8uQ/EirmVn23GM1M8uSfFeAmVnmPBRgZpYhz8dqZpY138dqZpY9DwWYmWXMPVYz\nswypPO4KaPz/aTCz8lL5kEBtS627UStJr0h6TdI4SRen5d0lvSxpoqR7JK2WlrdMv09K13cr2Ncv\n0/IJkvao7dgOVjPLFUlFLUVYBOwaEX2AvsAASdsBvwP+GBE9gY+AY9P6xwIfRcQmwB/TekjaAjgE\n+AYwALhOUo3dagermeVG8maWbII1Ep+kX1ukSwC7Av9KywcD+6ef90u/k67fTcmB9gPujohFETEF\nmAT0r+nYDlYzyw/VYSlmd1IzSWOAWcAwYDIwLyKWpFWmA53Tz52BaQDp+vnAuoXlVWxTJV+8MrMc\nKfrXfID2kkYWfL8hIm4orBARS4G+ktoC9wGbV7GfWHbwqtdVV14tB6uZ5UpFRdG/SM+JiH7FVIyI\neZKeBrYD2kpqnvZKuwAz0mrTgQ2B6ZKaA2sDcwvKKxVuU/U5FHsGZmYNIasxVkkd0p4qkloD3wPG\nA08BB6bVjgIeSD8PTb+Trn8yIiItPyS9a6A70BN4paZju8dqZvlRh/HTInQEBqdX8CuAIRHxoKQ3\ngbsl/QZ4FbgprX8TcLukSSQ91UMAImKcpCHAm8AS4NR0iKFaDlYzyw3VbYy1RhHxOrBVFeVvU8VV\n/Yj4HDiomn1dBlxW7LEdrGaWK1kFayk5WM0sVxysZmZZEqjCwWpmlin3WM3MMpTlxatScrCaWa44\nWM3Mstb4c9XBamY5IvdYzcwyV4e5AnLLwWpmueGLV2Zm9aHx56pntyqllqs157nbz+blewYy6l/n\nc8FJey23/upzD2L2C39Y9v3Ks37I8LsHMvzugbx+/6+Z+eyVAHTtuA4v3PkLht+d7Oe4A3dq0PNo\nyk487id07bQe2/Tt/bV1f7z6Klq3EHPmzAEgIjjzjJ/xjc024Vtbbcmro0c3dHPzT5m+mqVk3GMt\noUVfLGHACYP4dOEXNG9ewZM3n8njL7zJK2PfYesturJ2m9bL1f/FH+5d9vnkQ3amT68uAMyc/THf\nPfpqvli8hDVar8aof53PQ8+MZebs+Q16Pk3Rj486mpNOOY3jfnLkcuXTpk3jyf8OY8OuXZeVPfbo\nI0yeNJE3xk/klZdf5menncxzL77c0E3OvbyHZjHcYy2xTxd+AUCL5s1o3rwZEUFFhfjtGftz/p/v\nr3a7gwdsw5BHRwGweMlSvlicvGmi5WotqCiD/2M2Fjt9+zu0a9fua+W/OPvnXHb5lcuFxINDH+Cw\nI45EEttutx3z589j5syZDdncRsE9VltlFRXixX+cS48NO/C3e55lxBtTOfXQXXjombG8P+fjKrfp\n2nEdNuq0Lk+PmLCsrMv6bbl30Mn02LAD5/3pfvdWS+jB/wylU6fObNmnz3LlM2a8R5cuX01E37lz\nF2a89x4dO3Zs6CbmmucKKFI6i/dhEXFdQxyvMfnyy2C7Q65g7Tatuefq49lx6x788Ptbsfvxf652\nm4P22Ib7nxjDl19+9dqd6R/Mo///XU7HDmsz5Orjue+/rzJr7oKGOAUr8Nlnn/G7yy/jwUce/9q6\nZDL65eW959XQGkNvtBgNNRTQFjhlxcLa3s3dlMz/ZCHPjpzIzv02ZeMNOzBu6IX876GLWb1VC954\n4MLl6h64xzYMeXRklfuZOXs+b05+nx237tEQzbYVvD15MlPfmUL/bfrQa5NuvDd9Otv335r333+f\nzp27MH36Vy/7fO+96XTs1KmErc2nchgKaKhgvQLoIWmMpBGSnpL0D2CspG6S3qisKOlsSReln3tI\nelTSKEnPSdqsgdrbINqv02bZBapWLVuw67a9eHX8NLp//zw22/tCNtv7Qj77fDG997t42TY9N1qP\nddZaneGvTVlW1nm9trR
"text/plain": [
2018-05-31 11:49:28 +02:00
"<matplotlib.figure.Figure at 0x1a155caf60>"
2018-05-15 10:28:16 +02:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
2018-05-31 11:49:28 +02:00
"clf_model_c.fit(concat_tfidf_train, concat_y_train)\n",
"pred = clf_model_c.predict(concat_tfidf_train)\n",
2018-05-15 10:28:16 +02:00
"score = metrics.accuracy_score(concat_y_train, pred)\n",
"print(\"accuracy: %0.3f\" % score)\n",
"cm = metrics.confusion_matrix(concat_y_train, pred, labels=['true', 'false'])\n",
"plot_confusion_matrix(cm, classes=['true', 'false'])"
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 56,
2018-05-15 10:28:16 +02:00
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
2018-05-31 11:49:28 +02:00
"#clf = MultinomialNB()"
2018-05-15 10:28:16 +02:00
]
},
{
"cell_type": "code",
2018-05-31 11:49:28 +02:00
"execution_count": 57,
2018-05-15 10:28:16 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"accuracy: 0.749\n",
"Confusion matrix, without normalization\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAVYAAAEmCAYAAAA5jbhCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XecFdXdx/HPdwEpAiIgoKBiQSw8\nit1YMfYKJtbYRbHFEks0amKPGn1ssUVDFMtjiSkSe9doFClib6CiKEovAirg7/ljZsllXXYvy+ze\n2bvft6957b0z586cYd3vnj1z5owiAjMzy05FqStgZlZuHKxmZhlzsJqZZczBamaWMQermVnGHKxm\nZhlzsFq1JLWW9C9JMyT9dSn2c7CkJ7OsW6lI2kbSB6Wuh+WfPI61cZP0C+A0YG1gFjAauDQiXlrK\n/R4KnARsGRHzl7qiOScpgF4RMabUdbHGzy3WRkzSacC1wO+BrsAqwE1A/wx2vyrwYVMI1WJIal7q\nOlgjEhFeGuECLAd8A+xXQ5mWJMH7ZbpcC7RMt/UDxgOnAxOBCcCR6bYLge+BeekxBgIXAHcX7Lsn\nEEDz9P0RwMckreZPgIML1r9U8LktgeHAjPTrlgXbngcuBl5O9/Mk0Hkx51ZZ/18X1H8AsDvwITAV\nOKeg/GbAK8D0tOwNwDLpthfTc5mdnu8BBfs/C/gKuKtyXfqZNdJjbJS+XwmYDPQr9f8bXkq/uMXa\neP0EaAX8o4Yy5wJbAH2BDUjC5byC7d1IAro7SXjeKGn5iDifpBV8f0S0jYjBNVVE0rLA9cBuEdGO\nJDxHV1OuI/BIWrYTcDXwiKROBcV+ARwJdAGWAc6o4dDdSP4NugO/A24DDgE2BrYBfidp9bTsAuBX\nQGeSf7sdgBMAImLbtMwG6fneX7D/jiSt90GFB46IsSShe4+kNsDtwB0R8XwN9bUmwsHaeHUCJkfN\nf6ofDFwUERMjYhJJS/TQgu3z0u3zIuJRktZa7zrW5wegj6TWETEhIt6ppswewEcRcVdEzI+Ie4H3\ngb0KytweER9GxFzgAZJfCoszj6Q/eR5wH0loXhcRs9LjvwOsDxARIyPi1fS4nwJ/ArYr4pzOj4jv\n0vosIiJuAz4ChgErkvwiM3OwNmJTgM619P2tBIwreD8uXbdwH1WCeQ7QdkkrEhGzSf58Pg6YIOkR\nSWsXUZ/KOnUveP/VEtRnSkQsSF9XBt/XBdvnVn5e0lqSHpb0laSZJC3yzjXsG2BSRHxbS5nbgD7A\nHyPiu1rKWhPhYG28XgG+JelXXJwvSf6MrbRKuq4uZgNtCt53K9wYEU9ExE4kLbf3SQKntvpU1umL\nOtZpSdxMUq9eEdEeOAdQLZ+pcciMpLYk/daDgQvSrg4zB2tjFREzSPoVb5Q0QFIbSS0k7SbpD2mx\ne4HzJK0gqXNa/u46HnI0sK2kVSQtB/ymcoOkrpL2TvtavyPpUlhQzT4eBdaS9AtJzSUdAKwLPFzH\nOi2JdsBM4Ju0NX18le1fA6v/6FM1uw4YGRFHk/Qd37LUtbSy4GBtxCLiapIxrOcBk4DPgV8C/0yL\nXAKMAN4E3gJGpevqcqyngPvTfY1k0TCsIBld8CXJlfLtSC8MVdnHFGDPtOwUkiv6e0bE5LrUaQmd\nQXJhbBZJa/r+KtsvAIZImi5p/9p2Jqk/sCtJ9wck34eNJB2cWY2t0fINAmZmGXOL1cwsYw5WM7OM\nOVjNzDLmYDUzy1jZTSyh5q1Dy7QrdTVsCW24ziqlroLVwahRIydHxApZ7a9Z+1Uj5v/oJrdqxdxJ\nT0TErlkdO0vlF6zLtKNl71pHy1jOvDzshlJXweqgdQtVvZNuqcT8uUX//H47+sba7pwrmbILVjNr\nzARq/D2UDlYzyw8BFc1KXYul5mA1s3xRbVM45J+D1cxyxF0BZmbZc4vVzCxDwi1WM7NsyS1WM7PM\neVSAmVmWfPHKzCxbwl0BZmaZc4vVzCxL7gowM8tehbsCzMyy47kCzMyyVh5dAY3/DMysvEjFLbXu\nRn+RNFHS2wXrOkp6StJH6dfl0/WSdL2kMZLelLRRwWcOT8t/JOnwYk7BwWpm+aKK4pba3QFUfcLA\n2cAzEdELeCZ9D7Ab0CtdBgE3QxLEwPnA5sBmwPmVYVwTB6uZ5UexrdUiWqwR8SIwtcrq/sCQ9PUQ\nYEDB+jsj8SrQQdKKwC7AUxExNSKmAU/x47D+Efexmlm+FN/H2lnSiIL3t0bErbV8pmtETACIiAmS\nuqTruwOfF5Qbn65b3PoaOVjNLEe0JKMCJkfEJtkd+EeihvU1cleAmeVLRl0Bi/F1+ic+6deJ6frx\nwMoF5XoAX9awvkYOVjPLj8r5WLO5eFWdoUDllf3DgYcK1h+Wjg7YApiRdhk8Aewsafn0otXO6boa\nuSvAzHIku3Gsku4F+pH0xY4nubp/OfCApIHAZ8B+afFHgd2BMcAc4EiAiJgq6WJgeFruooioekHs\nRxysZpYvGc1uFREHLWbTDtWUDeDExeznL8BfluTYDlYzy5cyuPPKwWpm+aElGhWQWw5WM8sXT3Rt\nZpYtOVjNzLKTPJnFwWpmlh1R/b1OjYyD1cxyRG6xmpllraLCw63MzDLlFquZWZbcx2pmli25j9XM\nLHsOVjOzjDlYzcyyJFCFg9XMLFNusZqZZcgXr8zM6oGD1cwsa40/Vx2sZpYjcovVzCxznivAzCxD\nvnhlZlYfGn+u0vjb3I3cLecfzLhnLmPEX89ZuG759m14+OZf8tZDv+Phm39Jh3atAWjfthUPXnss\nw+4/m5EPnsuhe28BwCorLs/L9/yaV+9L1h+979YlOZem6Nijj2KVlbqwcd8+C9ddctEFrL5qdzbf\nuC+bb9yXxx97FIApU6awy47b07lDW049+ZelqnK+pX2sxSx55mAtsbv+9Sr9T7xxkXVnHLkTz7/2\nAf/T/yKef+0DzjhyZwCO3X9b3v/4KzY/4HJ2OeY6Lj9tH1o0b8aESTPZ/oir2eLAy9n20Cs548id\nWHGF5UpxOk3OoYcfwUMPP/6j9Sed8iuGjRzNsJGj2XW33QFo1aoVv7vgYi674qqGrmaj4mC1pfby\nqLFMnTFnkXV79lufu/81DIC7/zWMvbZfH4AA2i7bEoBlW7dk2ow5zF/wA/PmL+D7efMBaLlMCypy\n/j9dOdl6m23p2LFjUWWXXXZZttp6a1q1alXPtWrcHKxWL7p0asdXk2cC8NXkmazQsR0At9z3Amuv\n1o2Pn7yUEX89hzOufJCIAKBH1w68dv9v+Oixi/nfO55mwqQZJau/wS033cCmG67PsUcfxbRp00pd\nnUZFFSpqybMGCVZJHSSd0BDHKmc7bbkOb34wntV3PpfND7yMa87ej3bLJq2f8V9PZ7MDLqNP/ws5\nZK/N6JKGsTW8Y449nnc/GMuwkaPptuKKnH3m6aWuUqNRbGvVLdZEB+BHwSqpWQMdv1GZOGUW3Tq3\nB6Bb5/ZMmjoLgEP33oKHnn0DgI8/n8ynX0yhd8+ui3x2wqQZvDv2K7baaI2GrbQt1LVrV5o1a0ZF\nRQVHDTyGESNeK3WVGhUHa/EuB9aQNFrScEnPSfo/4C1JPSW9XVlQ0hmSLkhfryHpcUkjJf1b0toN\nVN+SeuSFtzhkr80BOGSvzXn4+TcB+PyrafTbrDcAXTq2Y62eXfnki8l079KBVi1bANChXWt+0nd1\nPvx0Ymkqb0yYMGHh64f++Q/WXa9PDaWtqnII1oYax3o20Cci+krqBzySvv9EUs8aPncrcFxEfCRp\nc+Am4KdVC0kaBAwCoEXbbGtez4ZcdgTbbNyLzh3aMubxi7n4lke56vanuPuKozh8wE/4fMI0Dv71\nYAAuv+1xbr3wEIY/cA4SnHvdQ0yZPpufbr42l5+2D0EgxLV3PsM7Y74s8Zk1DYcdchD/fuF5Jk+e\nzBo9e/Db313Iiy88z5t
"text/plain": [
2018-05-31 11:49:28 +02:00
"<matplotlib.figure.Figure at 0x1a16435278>"
2018-05-15 10:28:16 +02:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
2018-05-31 11:49:28 +02:00
"#clf.fit(concat_tfidf_train, concat_y_train)\n",
"pred = clf_model_c.predict(concat_tfidf_test)\n",
2018-05-15 10:28:16 +02:00
"score = metrics.accuracy_score(concat_y_test, pred)\n",
"print(\"accuracy: %0.3f\" % score)\n",
"cm = metrics.confusion_matrix(concat_y_test, pred, labels=['true', 'false'])\n",
"plot_confusion_matrix(cm, classes=['true', 'false'])"
]
},
2018-05-31 11:49:28 +02:00
{
"cell_type": "code",
"execution_count": 58,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"precision = cm[1,1]/(cm[1,1]+cm[0,1])\n",
"recall = cm[1,1]/(cm[1,1]+cm[1,0])"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"configuration_4 = {\n",
" \"name\":\"configuration4\",\n",
" \n",
" \"model\":\"model_c\",\n",
" \"model-name\":\"multinomialNB\",\n",
"\n",
"\n",
" \"dataset\":\"X3\",\n",
" \"dataset-name\":\"concat\",\n",
" \"dataset-link\":\"unknown\",\n",
" \n",
" \"measurement-name\":\"measurement_config4\",\n",
" \n",
" \"execution-name\":\"execution_config4\",\n",
" \n",
" \"precision\":precision,\n",
" \"recall\":recall,\n",
" \"accuracy\":score\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exporting results as RDF data"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from rdflib import Namespace, Graph, Literal\n",
"from rdflib.namespace import FOAF, OWL, XSD, RDFS, DCTERMS, DOAP, DC, RDF"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"prov = Namespace('http://www.w3.org/ns/prov#')\n",
"dcat = Namespace('http://www.w3.org/ns/dcat#')\n",
"mexalgo = Namespace('http://mex.aksw.org/mex-algo#')\n",
"mexperf = Namespace('http://mex.aksw.org/mex-perf#')\n",
"mexcore = Namespace('http://mex.aksw.org/mex-core#')\n",
"this = Namespace('http://mex.aksw.org/examples/')"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [],
"source": [
"example_configuration = {\n",
" \"name\":\"experiment1\",\n",
" \n",
" \"model\":\"generic_model_1_name\",\n",
" \"model-name\":\"deepNN\",\n",
"\n",
"\n",
" \"dataset\":\"X1\",\n",
" \"dataset-name\":\"Liar\",\n",
" \"dataset-link\":\"google.com\",\n",
" \n",
" \"measurement-name\":\"generic_measurement_hub\",\n",
" \n",
" \"execution-name\":\"genereric_execution_name\",\n",
" \n",
" \"precision\":0.33,\n",
" \"recall\":0.73,\n",
" \"accuracy\":0.55\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [],
"source": [
"def experiment_root_graph(root_node_name):\n",
" g = Graph()\n",
" g.add((this[root_node_name],RDF.type, mexcore.Experiment))\n",
" g.add((this[root_node_name],RDF.type, mexcore.ApplicationContext))\n",
" g.add((this[root_node_name],RDFS.label, Literal('2719095')))\n",
" g.add((this[root_node_name],DCTERMS.date, Literal('2018-05-22',datatype=XSD.date)))\n",
" g.add((this[root_node_name],FOAF.givenName, Literal('Carsten')))\n",
" g.add((this[root_node_name],FOAF.mbox, Literal('carsten.draschner@gmail.com')))\n",
" return g ,this[root_node_name]\n"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'rdflib.term.URIRef'>\n"
]
}
],
"source": [
"g, r = experiment_root_graph(\"first_test\")\n",
"print(type(r))"
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#extend a rdf graph with a configuration\n",
"def experiment_to_rdf(graph, experiment, root_node):\n",
" \n",
" g.add((this[experiment[\"name\"]],RDF.type,mexcore.ExperimentConfiguration))\n",
" g.add((this[experiment[\"name\"]],prov.used, this[experiment[\"model\"]]))\n",
" g.add((this[experiment[\"name\"]],prov.wasStartedBy, root_node))\n",
" \n",
" #dataset information\n",
" g.add((this[experiment[\"dataset\"]],RDF.type,mexcore.Dataset))\n",
" g.add((this.dataset2,RDFS.label,Literal(experiment[\"dataset-name\"])))\n",
" g.add((this.dataset2,DCTERMS.landingPage,Literal(experiment[\"dataset-link\"])))\n",
" \n",
" #model description\n",
" g.add((this[experiment[\"model\"]],RDF.type,mexalgo.Algorithm))\n",
" g.add((this[experiment[\"model\"]],RDFS.label,Literal(experiment[\"model-name\"])))\n",
" g.add((this[experiment[\"model\"]],DCTERMS.identifier,Literal(experiment[\"model-name\"])))\n",
" ###g.add((this.model1,mexalgo.hasHyperParameter,this.hyerparameter1))\n",
" \n",
" #execution\n",
" g.add((this[experiment[\"execution-name\"]],RDF.type,mexcore.ExecutionOverall))\n",
" g.add((this[experiment[\"execution-name\"]],prov.generated,this[experiment[\"measurement-name\"]]))\n",
" g.add((this[experiment[\"execution-name\"]],prov.used,this.test))\n",
" g.add((this[experiment[\"execution-name\"]],prov.used,this[experiment[\"model\"]]))\n",
" \n",
" #test\n",
" g.add((this.test,RDF.type,mexcore.Test))\n",
" g.add((this.test,RDFS.label,Literal('Test')))\n",
" \n",
" #evaluation information\n",
" g.add((this[experiment[\"measurement-name\"]],RDF.type,mexcore.PerformanceMeasure))\n",
" g.add((this[experiment[\"measurement-name\"]],mexperf.precision,Literal(experiment[\"precision\"],datatype=XSD.float)))\n",
" g.add((this[experiment[\"measurement-name\"]],mexperf.recall,Literal(experiment[\"recall\"],datatype=XSD.float)))\n",
" g.add((this[experiment[\"measurement-name\"]],mexperf.accuracy,Literal(experiment[\"accuracy\"],datatype=XSD.float)))\n",
" g.add((this[experiment[\"measurement-name\"]],prov.wasGeneratedBy,this[experiment[\"execution-name\"]]))\n"
]
},
{
"cell_type": "code",
"execution_count": 68,
"metadata": {},
"outputs": [],
"source": [
"experiment_to_rdf(g,configuration_1,r)\n",
"experiment_to_rdf(g,configuration_2,r)\n",
"experiment_to_rdf(g,configuration_4,r)"
]
},
{
"cell_type": "code",
"execution_count": 69,
"metadata": {},
"outputs": [],
"source": [
"with open('task2_metadata.ttl','wb') as f:\n",
" f.write(g.serialize(format='turtle'))"
]
},
2018-05-13 19:40:03 +02:00
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
