nlp-lab/Jonas_Solutions/Task_02_JonasWeinz.ipynb

1002 lines
84 KiB
Plaintext
Raw Normal View History

2018-05-09 10:47:51 +02:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
2018-05-11 12:19:53 +02:00
"# NLP-LAB Exercise 01 by Jonas Weinz\n",
2018-05-09 16:50:59 +02:00
"## links:\n",
"\n",
"* Article: https://miguelmalvarez.com/2017/03/23/how-can-machine-learning-and-ai-help-solving-the-fake-news-problem/\n",
" * corresponding code: https://github.com/kjam/random_hackery/blob/master/Attempting%20to%20detect%20fake%20news.ipynb\n",
"\n",
"* Tutorial on Datacamp: https://www.datacamp.com/community/tutorials/scikit-learn-fake-news\n",
"\n",
"* liar dataset paper: https://www.cs.ucsb.edu/~william/papers/acl2017.pdf\n",
2018-05-09 19:13:08 +02:00
" * dataset: https://www.cs.ucsb.edu/~william/data/liar_dataset.zip"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2018-05-09 16:50:59 +02:00
"## Dependencies for this Notebook:\n",
2018-05-09 10:47:51 +02:00
"* library [rdflib](https://github.com/RDFLib/rdflib)\n",
2018-05-09 19:13:08 +02:00
" * install: `pip3 install rdflib`\n"
2018-05-09 10:47:51 +02:00
]
},
{
"cell_type": "code",
2018-05-09 19:13:08 +02:00
"execution_count": 1,
2018-05-09 10:47:51 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Populating the interactive namespace from numpy and matplotlib\n"
]
}
],
"source": [
2018-05-11 12:19:53 +02:00
"%pylab inline"
2018-05-09 16:50:59 +02:00
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 2,
2018-05-09 16:50:59 +02:00
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import itertools\n",
"import sklearn.utils as sku\n",
2018-05-09 19:13:08 +02:00
"from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, HashingVectorizer\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.linear_model import PassiveAggressiveClassifier\n",
"from sklearn.naive_bayes import MultinomialNB\n",
"from sklearn import metrics\n",
"import matplotlib.pyplot as plt\n",
"from pprint import pprint as pp\n",
2018-05-09 16:50:59 +02:00
"import os"
]
},
2018-05-09 19:13:08 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Tools used later"
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 3,
2018-05-09 19:13:08 +02:00
"metadata": {},
"outputs": [],
"source": [
"def plot_confusion_matrix(cm, classes,\n",
" title,\n",
" normalize=False,\n",
" cmap=plt.cm.Blues):\n",
" fig_1, ax_1 = plt.subplots()\n",
" \"\"\"\n",
" See full source and example: \n",
" http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html\n",
" \n",
" This function prints and plots the confusion matrix.\n",
" Normalization can be applied by setting `normalize=True`.\n",
" \"\"\"\n",
" plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
" plt.title('Confusion Matrix for: ' + title)\n",
" plt.colorbar()\n",
" tick_marks = np.arange(len(classes))\n",
" plt.xticks(tick_marks, classes, rotation=45)\n",
" plt.yticks(tick_marks, classes)\n",
"\n",
" if normalize:\n",
" cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
" print(\"Normalized confusion matrix\")\n",
" else:\n",
" print('Confusion matrix, without normalization')\n",
"\n",
" thresh = cm.max() / 2.\n",
" for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
" plt.text(j, i, cm[i, j],\n",
" horizontalalignment=\"center\",\n",
" color=\"white\" if cm[i, j] > thresh else \"black\")\n",
"\n",
" plt.tight_layout()\n",
" plt.ylabel('True label')\n",
" plt.xlabel('Predicted label')"
]
},
2018-05-09 16:50:59 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Generate/Download Datasets we are working on\n",
"\n",
"* running bash script to download all needed data and store it into the `data` subfolder"
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 4,
2018-05-09 16:50:59 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"================================================================================\n",
2018-05-11 12:19:53 +02:00
"checking whether unzip is installed\n",
2018-05-09 16:50:59 +02:00
"================================================================================\n",
"UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.\n",
"\n",
"Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;\n",
"see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.\n",
"\n",
"Compiled with gcc 6.3.0 20170415 for Unix (Linux ELF).\n",
"\n",
"UnZip special compilation options:\n",
" ACORN_FTYPE_NFS\n",
" COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)\n",
" SET_DIR_ATTRIB\n",
" SYMLINKS (symbolic links supported, if RTL and file system permit)\n",
" TIMESTAMP\n",
" UNIXBACKUP\n",
" USE_EF_UT_TIME\n",
" USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)\n",
" USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)\n",
" UNICODE_SUPPORT [wide-chars, char coding: UTF-8] (handle UTF-8 paths)\n",
" LARGE_FILE_SUPPORT (large files over 2 GiB supported)\n",
" ZIP64_SUPPORT (archives using Zip64 for large files supported)\n",
" USE_BZIP2 (PKZIP 4.6+, using bzip2 lib version 1.0.6, 6-Sept-2010)\n",
" VMS_TEXT_CONV\n",
" WILD_STOP_AT_DIR\n",
" [decryption, version 2.11 of 05 Jan 2007]\n",
"\n",
"UnZip and ZipInfo environment options:\n",
" UNZIP: [none]\n",
" UNZIPOPT: [none]\n",
" ZIPINFO: [none]\n",
" ZIPINFOOPT: [none]\n",
"================================================================================\n",
2018-05-11 12:19:53 +02:00
"successfully finished action: checking whether unzip is installed\n",
"================================================================================\n",
"================================================================================\n",
"downloading and unpacking https://www.cs.ucsb.edu/~william/data/liar_dataset.zip if not already existing\n",
"================================================================================\n",
"================================================================================\n",
2018-05-09 16:50:59 +02:00
"successfully finished action: downloading and unpacking https://www.cs.ucsb.edu/~william/data/liar_dataset.zip if not already existing\n",
"================================================================================\n",
"================================================================================\n",
"downloading and unpacking https://raw.githubusercontent.com/GeorgeMcIntire/fake_real_news_dataset/master/fake_or_real_news.csv.zip if not already existing\n",
"================================================================================\n",
"================================================================================\n",
"successfully finished action: downloading and unpacking https://raw.githubusercontent.com/GeorgeMcIntire/fake_real_news_dataset/master/fake_or_real_news.csv.zip if not already existing\n",
"================================================================================\n"
]
}
],
"source": [
"%%bash\n",
"./Task_2_gen_data.sh"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Read in fake news table"
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 5,
2018-05-09 16:50:59 +02:00
"metadata": {},
"outputs": [],
"source": [
"df_1 = pd.read_csv('data/fake_or_real_news.csv').set_index('Unnamed: 0')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* display first 10 entries"
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 6,
2018-05-09 16:50:59 +02:00
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(6335, 3)"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>label</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Unnamed: 0</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>8476</th>\n",
" <td>You Can Smell Hillarys Fear</td>\n",
" <td>Daniel Greenfield, a Shillman Journalism Fello...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10294</th>\n",
" <td>Watch The Exact Moment Paul Ryan Committed Pol...</td>\n",
" <td>Google Pinterest Digg Linkedin Reddit Stumbleu...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3608</th>\n",
" <td>Kerry to go to Paris in gesture of sympathy</td>\n",
" <td>U.S. Secretary of State John F. Kerry said Mon...</td>\n",
" <td>REAL</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10142</th>\n",
" <td>Bernie supporters on Twitter erupt in anger ag...</td>\n",
" <td>— Kaydee King (@KaydeeKing) November 9, 2016 T...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>875</th>\n",
" <td>The Battle of New York: Why This Primary Matters</td>\n",
" <td>It's primary day in New York and front-runners...</td>\n",
" <td>REAL</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6903</th>\n",
" <td>Tehran, USA</td>\n",
" <td>\\nIm not an immigrant, but my grandparents ...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7341</th>\n",
" <td>Girl Horrified At What She Watches Boyfriend D...</td>\n",
" <td>Share This Baylee Luciani (left), Screenshot o...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>95</th>\n",
" <td>Britains Schindler Dies at 106</td>\n",
" <td>A Czech stockbroker who saved more than 650 Je...</td>\n",
" <td>REAL</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4869</th>\n",
" <td>Fact check: Trump and Clinton at the 'commande...</td>\n",
" <td>Hillary Clinton and Donald Trump made some ina...</td>\n",
" <td>REAL</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2909</th>\n",
" <td>Iran reportedly makes new push for uranium con...</td>\n",
" <td>Iranian negotiators reportedly have made a las...</td>\n",
" <td>REAL</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" title \\\n",
"Unnamed: 0 \n",
"8476 You Can Smell Hillarys Fear \n",
"10294 Watch The Exact Moment Paul Ryan Committed Pol... \n",
"3608 Kerry to go to Paris in gesture of sympathy \n",
"10142 Bernie supporters on Twitter erupt in anger ag... \n",
"875 The Battle of New York: Why This Primary Matters \n",
"6903 Tehran, USA \n",
"7341 Girl Horrified At What She Watches Boyfriend D... \n",
"95 Britains Schindler Dies at 106 \n",
"4869 Fact check: Trump and Clinton at the 'commande... \n",
"2909 Iran reportedly makes new push for uranium con... \n",
"\n",
" text label \n",
"Unnamed: 0 \n",
"8476 Daniel Greenfield, a Shillman Journalism Fello... FAKE \n",
"10294 Google Pinterest Digg Linkedin Reddit Stumbleu... FAKE \n",
"3608 U.S. Secretary of State John F. Kerry said Mon... REAL \n",
"10142 — Kaydee King (@KaydeeKing) November 9, 2016 T... FAKE \n",
"875 It's primary day in New York and front-runners... REAL \n",
"6903 \\nIm not an immigrant, but my grandparents ... FAKE \n",
"7341 Share This Baylee Luciani (left), Screenshot o... FAKE \n",
"95 A Czech stockbroker who saved more than 650 Je... REAL \n",
"4869 Hillary Clinton and Donald Trump made some ina... REAL \n",
"2909 Iranian negotiators reportedly have made a las... REAL "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(df_1.shape)\n",
"display(df_1[:10])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* create test dataset"
2018-05-09 10:47:51 +02:00
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 7,
2018-05-09 10:47:51 +02:00
"metadata": {},
"outputs": [],
2018-05-09 16:50:59 +02:00
"source": [
2018-05-09 19:13:08 +02:00
"def create_training_and_test_set(dset, cutoff=0.7):\n",
" shuffled = sku.shuffle(dset) # shuffle dataset\n",
2018-05-09 16:50:59 +02:00
" y = shuffled.label\n",
2018-05-09 19:13:08 +02:00
" shuffled = shuffled.drop('label', axis=1)['text']\n",
" size = int(cutoff * shuffled.shape[0])\n",
" return shuffled[:size], y[:size], shuffled[size:], y[size:]\n",
2018-05-09 16:50:59 +02:00
" "
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 8,
2018-05-09 16:50:59 +02:00
"metadata": {},
"outputs": [],
"source": [
2018-05-09 19:13:08 +02:00
"X,y, Xt,yt = create_training_and_test_set(df_1)"
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 9,
2018-05-09 19:13:08 +02:00
"metadata": {},
"outputs": [],
"source": [
"#X2,y2, Xt2,yt2 = train_test_split(df_1['text'],df_1.label,test_size=0.3)"
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 10,
2018-05-09 19:13:08 +02:00
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Unnamed: 0\n",
2018-05-11 12:19:53 +02:00
"5476 Share on Twitter \\nFor Robin Roberts, losing h...\n",
"1223 Jeb Bushs resignation from the presidential r...\n",
"8179 Does The U.S. Government Really Know Who Hack...\n",
"6457 Print \\n[Ed. Now teaching the gospel of raci...\n",
"2236 Video of a confrontation between a news photog...\n",
"7451 Tony Blair helpfully describes Remain voters a...\n",
"7548 Tweet Home » Headlines » World News » Its A S...\n",
"2409 President Obama's top health official testifie...\n",
"3918 And it looks like that time is nigh. Clinton i...\n",
"7698 Arnaldo Rodgers is a trained and educated ...\n",
"1461 Ben Carson pitched a tax plan with numbers tha...\n",
"3850 President Barack Obama has officially hit the ...\n",
"3168 From Coca-Cola to Microsoft, companies that ga...\n",
"5738 Trending Articles: Trending Articles: Chairma...\n",
"3655 The horrific attack in Orlando, Florida, showe...\n",
"4656 With less than two weeks to go, the race for t...\n",
"6823 Email Print This is WHY Comey wrote the letter...\n",
"3190 Donald Trump may have eased some Republican fe...\n",
"9509 Tuesday 1 November 2016 by Formelia Alberthine...\n",
"8017 API Reports A Build, DOE A Draw by IWB · Octob...\n",
"8116 Region: Russia in the World More bang for the ...\n",
"9762 “The statesmen will invent cheap lies, putti...\n",
"3760 Freddie Gray, whose death triggered Mondays r...\n",
"9438 Support Us iMAHDi the arrivals 28 Why Satani...\n",
"7859 14th Anniversary of His Passing By Joachim Hag...\n",
"5129 Hillary Clinton will already make history with...\n",
"5599 By: The Voice of Reason | In recent weeks, a c...\n",
"7691 BREAKING Investigative Journalist Found Dead...\n",
"7912 This is now becoming the norm. Just yesterday,...\n",
"6508 Email \\nWill this be the most chaotic election...\n",
2018-05-09 19:13:08 +02:00
" ... \n",
2018-05-11 12:19:53 +02:00
"10221 Podesta Goes Crazy Live On CNN Over New FBI Hi...\n",
"9568 Comments \\nConan OBrien asked comedian Louis ...\n",
"9479 posted by Eddie A list of secret Apple iPhone ...\n",
"1574 Texas Senator Ted Cruz (R) isnt about to let ...\n",
"5442 Taming the corporate media beast BRICS Do Not ...\n",
"3647 Imagine what would happen if a retail store or...\n",
"9818 Trump To Host Facebook Live Nightly Show Until...\n",
"862 Will The Real Donald Trump Please Stand Up?\\n\\...\n",
"7731 Thu, 27 Oct 2016 15:45 UTC © Lockheed An artis...\n",
"1136 (CNN) The two presidential front-runners are b...\n",
"10229 Your News Wire WikiLeaks Bombshell: There Is ...\n",
"9799 source Add To The Conversation Using Facebook ...\n",
"9649 . MMR Vaccines Cause 340% Increased Risk of Au...\n",
"236 THE HOUSE Select Committee on Benghazi further...\n",
"9275 Sesame Seeds for Knee Osteoarthritis VN:F [1.9...\n",
"4336 WASHINGTON — Macy's said Wednesday that the Tr...\n",
"7350 Getty - Kevin Mazur The Wildfire is an opinion...\n",
"4881 The banner headline on the Drudge Report the m...\n",
"2404 Ask him, and he'll tell you himself. \"I'm very...\n",
"7947 Comments \\nI learned exclusively tonight that ...\n",
"3359 EXCLUSIVE: Highly classified Hillary Clinton e...\n",
"7927 FBI Director may be sacked for intrusion into ...\n",
"4948 Earlier today, I talked with Libertarian Party...\n",
"10183 Originally appeared at Strategic Culture Found...\n",
"2073 It was once a sleepy Capitol Hill backwater wi...\n",
"8082 The Washington Post reported : \\nDonald Trump ...\n",
"8145 . The Hulk Actor Mark Ruffalo Has Joined Stand...\n",
"3890 Barack Obama will make a long-awaited trip to ...\n",
"4885 Hillary let the cat out of a bag Friday. For o...\n",
"8025 Here's something interesting from The Unz Revi...\n",
2018-05-09 19:13:08 +02:00
"Name: text, Length: 4434, dtype: object"
]
},
2018-05-11 12:19:53 +02:00
"execution_count": 10,
2018-05-09 19:13:08 +02:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X"
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 11,
2018-05-09 19:13:08 +02:00
"metadata": {},
"outputs": [],
"source": [
"count_vectorizer = CountVectorizer(stop_words='english')\n",
"count_train = count_vectorizer.fit_transform(X)\n",
"count_test = count_vectorizer.transform(Xt)"
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 12,
2018-05-09 19:13:08 +02:00
"metadata": {},
"outputs": [],
"source": [
"tfidf_vectorizer = TfidfVectorizer(stop_words='english', max_df=0.7)\n",
"tfidf_train = tfidf_vectorizer.fit_transform(X)\n",
"tfidf_test = tfidf_vectorizer.transform(Xt)"
2018-05-09 16:50:59 +02:00
]
2018-05-09 19:13:08 +02:00
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 13,
2018-05-09 19:13:08 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['00',\n",
" '000',\n",
" '0000',\n",
" '00000031',\n",
" '000035',\n",
" '00006',\n",
" '0002',\n",
2018-05-11 12:19:53 +02:00
" '000ft',\n",
" '000x',\n",
" '001']\n",
2018-05-09 19:13:08 +02:00
"['حلب', 'عربي', 'عن', 'لم', 'ما', 'محاولات', 'من', 'هذا', 'والمرضى', 'ยงade']\n"
]
}
],
"source": [
"pp(count_vectorizer.get_feature_names()[0:10])\n",
"pp(count_vectorizer.get_feature_names()[-10:])\n"
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 14,
2018-05-09 19:13:08 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['00',\n",
" '000',\n",
" '0000',\n",
" '00000031',\n",
" '000035',\n",
" '00006',\n",
" '0002',\n",
2018-05-11 12:19:53 +02:00
" '000ft',\n",
" '000x',\n",
" '001']\n",
2018-05-09 19:13:08 +02:00
"['حلب', 'عربي', 'عن', 'لم', 'ما', 'محاولات', 'من', 'هذا', 'والمرضى', 'ยงade']\n"
]
}
],
"source": [
"pp(tfidf_vectorizer.get_feature_names()[:10])\n",
"pp(tfidf_vectorizer.get_feature_names()[-10:])"
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 15,
2018-05-09 19:13:08 +02:00
"metadata": {},
"outputs": [],
"source": [
"#count_df = pd.DataFrame(count_train.A, columns=count_vectorizer.get_feature_names())\n",
"#tfidf_df = pd.DataFrame(count_train.A, columns=tfidf_vectorizer.get_feature_names())\n",
"#diff = set(count_df.columns) - set(tfidf_df.columns)\n",
"#pp(count_df.equals(tfidf_df))"
]
},
{
"cell_type": "code",
2018-05-11 12:19:53 +02:00
"execution_count": 16,
2018-05-09 19:13:08 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2018-05-11 12:19:53 +02:00
"'score: 0.8574434508153603'\n",
"Confusion matrix, without normalization\n"
2018-05-09 19:13:08 +02:00
]
},
{
"data": {
2018-05-11 12:19:53 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEmCAYAAABf+4ZQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xe8HFXdx/HPNwmBQEISCBAIgdBbIHSQIiAdhASUXhVFkKIiFkR50AcUwYJg4ZEHqUpXRKqIgMBDh9BbAoQkhDRqSAIpv+ePc26yudwyk+y9e+/e7zuveeXuzOzM2dnZ/c45Z3ZGEYGZmVm1dat1AczMrD45YMzMrE04YMzMrE04YMzMrE04YMzMrE04YMzMrE3UNGAk9ZL0D0nvS7p+EZZzmKR/VrNstSDpdklHLeRzz5I0RdLb1S6XtY1Feb9rTdKOksa1MH0VSdMkdW/nchVeb2uvoQpluUjSj9pq+Z1CRLQ6AIcCjwPTgAnA7cB2RZ7bynKPAB4FeizqstpiAHYEAvhbo/HD8vh7Cy7nTOCqNiznKsAMYPk2Wva0iiGAjyoebw9cBnzSaL6D8vPfAHbJfx8NzKmY53XgUmDtivUNyeuoXNbTLZRv61ye3k1Mewo4sY22+b3AV2q9jy5EuRu271ONxg/I7+EbBZezIzCu4vG897mzDI1fQxPTA3gW6FYx7izgshqX+438eZ8GvAvcCgyu9fZsami1BiPpFOB84KfACvkL5/fA8NaeW8CqwCsRMbsKy2ork4HPSFq2YtxRwCvVWoGSRalNrgJMjYhJC7HuHi1Nj4g3I6J3w5BHD6sYd38ed27lfBFxbTOLfCgvpy+wC+mD8oSkoY3m61exrGEtlO9hYBzwxUavayiwPnB1S6+vFqrwfldjuUs22uaHkgLfFrQScHCtC9GEffLnaEVgInBhjcvTtFaSsi8pJQ9oYZ7FSQH0Vh7OBxavPEIAvg1MItV+vpSn/Zh0xDQrr+MYGh3pM/9oq0d+fDTwGvAh6cNwWMX4Byqetw3wGPB+/n+bimn3Av8NPJiX809gQEtHOMBFwAl5XHdgPHAGFTUY4DfAWOAD4Alg+zx+j0av8+mKcpydyzEDWJOKo2LgD8CNFcv/OXA3oEZlbPiSnpuXf1kevy/wPPBeXu56jY6Avgc8A3xMiRpkfj/WbDTuMuCsFo62KmswDzQxzy3ADU295wXL9APg343GnUtFzZNU0/m/vD2eBnasmLYMqSb1FumI8KaKacOBkfl9HZ3fz7NJNbGZeZv/tuB+19L7/TSfrinuWKDsn1puK9uqYfv+EDivYvzjwOlU1GAav9eV7zMVR//AlXn/aziq/m7j95FWPne0vr9+h7S/fgRcQjrYvT0v619A/2a+M74EvJjnew34WuPPdyv7+veAVyuWt0ANBrgeeDu/5/8BNmhme70IfL5iWg/Sweumrb3HLX2m8uO9SAfqDY/3JtXePyB9J51ZMe1W4KRGy3sG2C//vS5wF/AO8DJwYKP1vJC35Xjg1FY/m63sjHsAs2nhww78BHgYWB5YLm+k/654A2fneRbLBZxesTOcyYKB0vjxvJ0FWCpvsHXytBUb3kwqvrhIXxbvkprfegCH5MfLVuzoo4G1gV758TnNvLYdSQGzDfBIxUa+E/gKCwbM4cCyeZ3fzjvdEk29ropyvAlskJ+zGAt+4SxJqiUdTWqGmgKs3FI5Kx6vTfog7pqX+11gFNCzYgcdCQwGeuVxvwd+3+oO0zYB82VgYlNfEEWG/Dpmk5sJSH2L44AR+fEgYGp+77rl7TIVWK7iQ3ct0D9vrx3y+C1JXxy75ucNAtateP++UlGGIvtds+93o9dzLPASsHSBsn9qua1sq4btO4T05dOdVNN7iXSwUjpgmvnSW+B9pIXPHcX214dJoTKIdLD6JLAJsATwb+C/mlnv3sAagIAdSN8/mzb1GprZ19ciHTA2fC4bB8yXgT7MP9Ae2cz2OgP4c8W0vYEXi+yfrXymlgQuB65o9H2wYV7WRqQaTsNn4UDyd1l+PCyvqyfpO3YsKZR75O07BVg/zzuB+QfO/Ru2Y0tDa9XpZYEp0XIT1mHATyJiUkRMJtVMjqiYPitPnxURt5GOcNZpZb3NmQsMldQrIiZExPNNzLM38GpEXBkRsyPiatKHZ5+KeS6NiFciYgZwHbBxSyuNiP8DlpG0DnAkcEUT81wVEVPzOn9J2uFae52XRcTz+TmzGi1vOmk7/gq4inTUUbRD8iDg1oi4Ky/3F6QP9TYV81wQEWPzNiAivh4RXy+4/KacKum9PEwp+dy3SF/QlaZULO/Ulp4cEWNJX1gN+93OpO1/a358OHBbRNwWEXMj4i7SEfteklYE9gSOi4h38356X37eMcCf8nacGxHjI+KlZopRZL9r9v1uIGk70pfYvhHxQUtlL7PcJowjHaHuQtqnryz4vEXR3OeuyP56YURMjIjxwP2kL8mnImIm8DfSl+GnRMStETE6kvtINaftS5Q5gB8BP5LUs4nl/ykiPoyIj0kHksMk9W1iOX8B9pW0ZH58KPObb4u8x43dJOk95h8AnVdRpnsj4tm8rGfyenbIk28G1pa0Vn58BHBtRHwCfJ50gHFp3peeAm4EDsjzzgLWl7R0/qw82UL5gNbPIpsKDGilnX4lYEzF4zF53LxlNAqo6UBvSoqIj0g74nHABEm3Slq3QHkayjSo4nHlmVZFy3MlcCKwE2mHXoCkUyW9mM+Ie4/UvDiglWWObWliRDxCqtaL9IEsaoFtEBFz87oqt0GL614Iv4iIfnlo7XU3NohUJa80oGJ5vyiwjMuZHzBHANdUfNmuChxQEVjvAduRasGDgXci4t0mljmYdNRdRJH9rsVtLmkw6X0+KiIa+vhaKnuh5bbgClKt8hDaJ2Ca+9wV2V8nVvw9o4nHTX6GJe0p6WFJ7+Rttxetfy4XkA+MxwFfa7Ts7pLOkTRa0gekmgVNLT8iRpGayfbJIbMvKXSg2Hvc2IiI6EeqwZ0I3CdpYC7XVpLukTRZ0vuk78wBuRwzSbX1w3N/XeV7vyqwVaNyHAYMzNO/QNp+YyTdJ+kzrW271gLmIVIb/YgW5nkrF6zBKnncwviIVOVrMLByYkTcGRG7kjb8S8DFBcrTUKbxC1mmBlcCXycdaUyvnCBpe1K1/kBS818/0pGFGorezDKbG9+w3BNIR+Jv5eUXtcA2kCTSl2XlNmhx3e1sP9JR6aL4K7CypJ2A/UmB02AscGVFYPWLiKUi4pw8bRlJ/ZpY5lhS80pTGm+/Ivtds9tcUi/gJuD8iLi9YNlbXW4rbiTVvF6LiDebmD6dFj6PjSzK/lRkfy1N0uKk1/gLYIX8ubyN+Z/LMk4n9fVVbo9DSX10u5AOKIc0rLqZZVxN+kIfDryQQweKvcdNiog5EfFXUp/gdnn0X0g1lcER0ZfUh1xZpstJwbEzMD0iHqoox32NytE7Io7P63osIoaTukNuosBBb4sBExHvk9oOfydphKQlJS2WjwrOzbNdDfxQ0nKSBuT5r2ptxc0YCXw2n8veFzitYYKkFSQNl7QUKfSmkZrMGruNVAU8VFIPSQeR2phvWcgyARARr5Oqmac3MbkPqQ9gMtBD0hmk9vMGE4EhZc4ckrQ2qankcNIR+XcltdiUV+E6YG9JO0tajNQn9DGpf6xDyEd/q0m6kNRm/ONFWV6u4d5A6qwfExGPV0y+inTkuHte7xL5NxArR0TDafe/l9Q/79+fzc+7BPhS3o7dJA2qqDVPBFavWMei7nd/Al6KiHMbjW+27M0tSNKZku5tbYV5m32O1J/YlJHAoXm9ezC/maUpjbdHGW21v/YkHaBNBmZL2hPYbWEWFBH3As+RziBt0IdUzqmk4PlpK4u5Jq//eObXXmAh3uMGSoaT+kRerCjXOxExU9KWpCCsfC0Pkb47f8mCNddbSPvwEflzsJikLSStJ6mn0u8N++aWgQ9o+vt3Aa1+4UXqTziFdNbJZFLKnUhKMEhfgo+TzkR4ltT5dlZry21mXXeRqm/PkDrWKj+c3XI
2018-05-09 19:13:08 +02:00
"text/plain": [
2018-05-11 12:19:53 +02:00
"<matplotlib.figure.Figure at 0x7fb256550278>"
2018-05-09 19:13:08 +02:00
]
},
"metadata": {},
"output_type": "display_data"
2018-05-11 12:19:53 +02:00
}
],
"source": [
"clf = MultinomialNB()\n",
"clf.fit(tfidf_train, y)\n",
"pred = clf.predict(tfidf_test)\n",
"score = metrics.accuracy_score(yt, pred)\n",
"pp(\"score: \" + str(score))\n",
"cm = metrics.confusion_matrix(yt, pred, labels=[\"FAKE\", \"REAL\"])\n",
"plot_confusion_matrix(cm, classes=[\"FAKE\", \"REAL\"], title= \"TFIDF_Vecctorizer, Multinomial Naive Bayes\")"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
2018-05-09 19:13:08 +02:00
{
"name": "stdout",
"output_type": "stream",
"text": [
2018-05-11 12:19:53 +02:00
"'score: 0.8916359810625987'\n",
2018-05-09 19:13:08 +02:00
"Confusion matrix, without normalization\n"
]
2018-05-11 12:19:53 +02:00
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEmCAYAAACj7q2aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xe8HFX9//HX+6YRWhIIICSBAAmioIQQ+hcpAaS3rzSRZgRUBBW+Koo/xY6FIog0EQJKLxJ6CSBFAgQITVqAhCQEUoAIJJQkn98f51zYXO/O7r3Zm73l/byPedydM7MzZ3Zm5zOn7IwiAjMzs9ZoqHcGzMys43IQMTOzVnMQMTOzVnMQMTOzVnMQMTOzVnMQMTOzVqtrEJHUW9INkuZIumoxlnOQpNtrmbd6kHSLpENb+d5fSpol6fVa58tar6Mfm5ImSdq+YHqrj9nF0ZL1VtqGxczHVpKeb4tldxgRUXEAvgyMB94FpgO3AP9TzXsrLPdg4GGg++Iuqy0GYBsggOuapG+Q0++pcjknAX9rw3yuDswDVm7DdQg4FngaeA+YClwFfK6N98Hg/Fk3e4wABwCTADVJ7w7MAHZbjHUfBty/JI+5Gn5u9+TPbYMm6dfl9G2qXM4kYPv8uk2P4zb8LD7ehmamXZQ/j01K0oakU2Nd83wS8FE+574LPAv8b70/y+aGiiURSccBpwO/BlbJJ6w/A3tWem8V1gBeiIj5NVhWW5kJbC5pxZK0Q4EXarUCJYtTKlwdmB0RM1qx7u5VzvpH4NukQLICsA7wD2DXlq6zxv4B9AW2bpK+E+nkcOsSz1HWgs+2rZb7AnBIyftWBDYnHdP2iTeBX9Y7E824IiKWjYhlge8Af5O0Sr0z9V8qRMM+pCi4b8E8vUhB5rU8nA70ytO2IV2xHk+6KpwOHJ6n/Qz4kE+i7SiaXOnQ5CqUdGX4MvAO8ApwUEn6/SXv2wJ4BJiT/29RMu0e4BfAA3k5twP9y2xbY/7PAY7Oad2AacBPKCmJkE6yU4D/AI8CW+X0nZps5xMl+fhVzsc80tXPPcDX8vSzgWtKlv9bYCz/fcW9fX7/wrz8i3L6HsAzwNt5uZ9pcmX2A+BJ4AMqlASBocACSq7WyhwrF5NOUJOBHwMNJVdVRfu17D4BXs3zNl6Rbd7Mus8D/tok7UrgtJLx3YAJ+fP4F/D5kmmDgGtz3mcDfwI+A7yft/td4O0qtvOwvA2n5eX8kpJjE/h+yXa8m4+Ji0qWewHpOzItv7dbueVWcSV7D+kYnVqynG+Rjqup5JII6Ur8lyXv2waY2uRY2Z7i4/hrJfm8H/gD8BbpO7pzybJWA8aQTtoTgSNKpp1EKtn+LR8DT5EuVH5IOndMAXZssn2N610buCt/NrOAvwN9m25Dmc/pIuBU4HVg65y2SEkEOJxUEniHdP45qrnPi/SdurrJ8v8InFFpHzeTr5NoUurLn8MW+XU/4EbScfhWfj0wT9sXeLTJe48Dri85Z/+B9N16g3R+652n9c/Lejvvp/vIx3fZY63CgbgTMJ+Ckwzwc2AcsDKwEukL+ouSD3h+nqcHsAswF+hX5uTSdHww+WQDLEM6QX86T1sVWK/04M2vV8gf6sH5fQfm8RVLDr6XSAdo7zx+cplt24b0hdsCeCin7QLcBnyNRYPIV4AV8zqPJx2USxUcEPfknbhefk8PFv1iLE26kjwM2Ir05RhYlM+S8XVIVU475OV+n/Sl7VnypZpAOnk2Hjx/Bv5cZvlfByZXOFYuBq4Hlsv77QVgVKX9WmmfNJ23zLq3zMdG47b0IQXWYXl8Q9IXcFPSRcCh+TPolcefIJ2glwGWIlfV0kx1VoXtPIx0vB+T92nv5paR5x1EuujaOY9fB5yb87AyqZr3qHLLLdoXJZ/p10gBuXEdD5NKIi0OIhWO49Ig8hFwRP5cv5G3UXn6vaTjbClgGOkEuF3Jst8Hvpi38WJSEDqRdAwfAbxSZr1DSMd6L9I56F7g9Oa2oZnP6SLSyfxYPjmHNA0iu5IClUgl3rnA8KafF6lmZS6wXB7vRgoYm1Xax83k6+PPOq93V9KJvW9OWxH4X9J5YjlSAP5HntaLFABKLxwfJ1eHkY71MaRz5XLADcBv8rTfkIJKjzxsRZML16ZDpSqUFYFZUVzddBDw84iYEREzSSWMg0umf5SnfxQRN5OuYj5dYb3lLATWl9Q7IqZHxDPNzLMr8GJEXBIR8yPiMuA5YPeSeS6MiBciYh7pinVY0Uoj4l/ACpI+TaoeuLiZef4WEbPzOk8h7chK23lRRDyT3/NRk+XNJX2Op5Kuzo6JiKkVltdof+CmiLgjL/cPpBPaFiXznBERU/JnQER8MyK+WWZ5K5K+DM2S1I3UNvHDiHgnIiYBp7DocVBJi/ZJqYh4gHRFtXdO2o9UTTohjx8JnBsRD0XEgogYTSqBbQZsQrpC/l5EvBcR70fE/Yuxna9FxJl5n84rs5zepGq4P0bELbmKYhfgOzkPM0hf9ANastwyLgYOkbQu6QT0YAve2xqTI+L8iFgAjCZd7K0iaRAp2P8gf8YTgL9QUt0G3BcRt+XzzVWkgHByPoYvBwZL6tt0hRExMR/rH+Rz0Kn8d/VmJecCq0vauZnl3xQRL0XyT1Jg3qqZ+SYDj/HJcbgdMDcixlW5j5vaT9LbpHPmGODXEfF2XtfsiLgmIuZGxDukWo2t87QPgCtIF7ZIWo90wXOjJJG+D9+NiDfze39dko+PSPtsjXzOvi9ydCmnUhCZDfSvUAe7GqlY32hyTvt4GU2C0Fxg2Qrr/S8R8R7p5Ph1YLqkm/IXo1J+GvM0oGS8tAdTtfm5hFQdsC3pimIRkv5P0rO5p9nbpKvh/hWWOaVoYkQ8RCo+i3RirdYin0FELMzrKv0MCtfdxGzSgVVOf9JVS9PjYEDzszerNfuk1MV8ckI6mEUD/RrA8ZLebhxIJYHV8v/JFS6UGlWzndV8rhcAz0fEb0vy14N0XDfm71zS1WpLltuca0kns2+RjuG29vF+zBdCkPblakDjSatR08/ujZLX80gXsAtKxhuXtQhJq0i6XNI0Sf8hXXRV+u4tIp94f5GHpsvfWdI4SW/mfbNLwfIvJdV+QOqQdGl+Xc0+burKiOgbEcuQSkKHSDoq52lpSedKmpy3+V6gb77QgRTAv5yDxsF5WR+QAvPSwKMl+bg1pwP8nlRrcbuklyWdUJA/oHIQeZB0xbZXwTyvkT6gRqvntNZ4j7SBjT5VOjFfpexAOqE9B5xfRX4a8zStlXlqdAnwTeDmki8HkLr5kaqM9iNV1fUltceoMetlllkY4SUdTSrRvJaXX61FPoN8IA1i0c+gcN1NjAUGShpRZvos0hVM0+OgcX2F+7WCavN5CTBS0uakEsbfS6ZNAX6Vv5CNw9K5lDqFdAXa3IVS03VX2s6K+c1fynVIbYCl+fuA1A7UmL/lI2K9apdbTj5WbyFVLTUXRFqyb1qVh+w1Uml+uZK0WnwvIV1JB6mn4PKkK3AVv6VZF5I6aezTmCCpF3ANqTS/Sv5u31yw/KuAbSQNJJVIGoNINfu4rFzqvYVPalSOJ9V0bJq3+QuNWc7zjyO1YW1FCmaN+34WKSCvV5KPPpEa78kl7OMjYi1Su+pxkkYW5a0wiETEHFLj3FmS9srRr0eOzL/Ls10G/FjSSpL65/n/VsXn0pwJwBckrS6pD6lRDfj4amNPScuQdsa7pOqtpm4G1pH0ZUndJe0PfJbUWNRqEfEKqbh4YjOTlyPVWc8Eukv6CbB8yfQ3SEXxqntgSVqHVFf7FdKVxPclVVvFcyWwq6SRknqQDrgPSO1VLRYRL5Lqsi+TtI2knpKWknSApBPy1eKVwK8kLSdpDVJDXuNxUHa/VmEmaT+vVSGPk0iNupcBd0REacnmfODrkjbNPeGWkbRrPqE9TKqqOzmnLyVpy/y+N0jBs2deR6XtLJSrSo4F9i6tkoqI6aQqklMkLS+pQdLakspWyUgaLCkkDa5i1T8iNRpPambaBGAXSStI+hSpF1A5LT6OG0XEFNL
"text/plain": [
"<matplotlib.figure.Figure at 0x7fb256506240>"
]
},
"metadata": {},
"output_type": "display_data"
2018-05-09 19:13:08 +02:00
}
],
"source": [
"clf = MultinomialNB()\n",
2018-05-11 12:19:53 +02:00
"clf.fit(count_train, y)\n",
"pred = clf.predict(count_test)\n",
2018-05-09 19:13:08 +02:00
"score = metrics.accuracy_score(yt, pred)\n",
"pp(\"score: \" + str(score))\n",
"cm = metrics.confusion_matrix(yt, pred, labels=[\"FAKE\", \"REAL\"])\n",
2018-05-11 12:19:53 +02:00
"plot_confusion_matrix(cm, classes=[\"FAKE\", \"REAL\"], title= \"Count Vectorizer, Multinomial Naive Bayes\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* comparing to PassiveAggresiveClassifier"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jonas/.local/lib/python3.6/site-packages/sklearn/linear_model/stochastic_gradient.py:117: DeprecationWarning: n_iter parameter is deprecated in 0.19 and will be removed in 0.21. Use max_iter and tol instead.\n",
" DeprecationWarning)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"accuracy: 0.933\n",
"Confusion matrix, without normalization\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ8AAAEmCAYAAAC9J50pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xm8HuP9//HXO6uENEEIEmvt9GuLva29RRHUWiVIbfXTRRVtUVpKlVqqrVIlQYVSS9VStVM7sdUWBIkQscaSRJLP74/rOsnk9CTn3s5959x5P89jHmfmmrlnrlnu+cx1zXXPKCIwMzOrpy6NzoCZmc1/HHzMzKzuHHzMzKzuHHzMzKzuHHzMzKzuHHzMzKzuGhp8JPWS9A9JH0r6WxXz2UfSv2qZt0aQdLOkoRV+9mRJEyW9Vet8Wfma5ZgEkPSspM0bnY9Ga9Q+lbS5pLEdOP/zJR1fGD5M0tuSPpa0aP6/Qs0XHBHtdsC3gEeBj4HxwM3Al0v5bDvz3Rd4GOhW7bw6ogM2BwK4tlX6Wjn9rhLncyJwWQfmcxngM2DxDpr3x4UugE8Kw18BLgGmtppuz/z5McDWuX9/YHphmleBi4GVC8tbLi+jOK8n28jXXnneapXeDZgA7FDFOu8P3Ffj7RjAijU8LmfkbTMJeAE4oKOOrxpvBwGvAP9tdF7mpQ7YALgJ+AB4L58XDyjs77F1ykf3fC5Zq6OX1W7JR9KRwNnAr4AB+WT0B2BIe58twbLAixExrQbz6ijvABtLWrSQNhR4sVYLUFJNKXQZ4N2ImFDBsrvNbXxEvB4RC7V0OXmtQtq9Oe304nQRceUcZvlAnk9fYGvSgf6YpDVbTdevMK+12pjPdUA/YLNW6duSTvS3zG29OlJ727RG3szb8QvAMcCFklavw3Kr9VVgcWAFSet31ELqtA9qQtLGwB3A3cCKwKLAYcB2DcjOAGAB4NlqZ9TuPmgnCvYlXV3tPpdpepKC05u5OxvoWYzYwI9IV6PjmRXNTyJdLX+elzGMViUEZl0Fd8vD+5OumiaRrpr3KaTfV/jcJsAjwIf5/yaFcXcBvwTuz/P5F9B/DuvWkv/zgcNzWldgHHAChZIPcA7wBvAR8BjwlZy+bav1fLKQj1NyPj4jHXR3Ad/J4/8IXFOY/6+B2/nfK/2WE3jLlfAlOX0n0gH0QZ7vaoXPjCGdsJ4CplBGyZM2ruBJJZ+T5zD9GGYv+fxPiQK4Ebi6rX3eTl4uAP7SKu0q4KzC8A7AqLwd/gP8X2Hc0sDfSRcY7wLnAasBk5lVQvug8F0Ykad9DTgO6FJYr/uBs/J8Ti6uK3APs5cY92wvb+2s9+a0uhLO+dot9/8NeIt0/N8DrFGYbnvgv6RjfxxwVE7vn/dDy5X3vYX1G5OPs6XysbZIYX7rABOB7nn4QOA54H3gVmDZVvn8C3B53u7ntRq3fM7vJODfwO+Z/XywX9727wLHtzq2TgSuBi4jfQe/Q7qtcCzwcv7MVS15J51gL8vpH5DOEwNKPc+Qvp9ntMr/9cCRuX8p4Jq8X14FvjeX/Xkf8PtS93dhnSblfblLYdyKpCD2Yd4vV+Z0kY7PCXn7PA2sWfz+AiuTjtGWmoc7Wn/nSef7M4DXgbdJ58Zerc6Xx5COv0vnehy3c5BvC0xjLicC4BfAg6SrmcVIX6JfFjIzLU/TnXTgfwosXDhgigdX6+Hl8op3AxbMG22VPG5J8peq1UGxCOnA3zd/bu88vGgef1fecSsDvfLwaXPb6aRg9lDhy3sr6eAuBp9vk65YupGC7VvAAm2tVyEfrwNr5M90Z/bg05tUutqfVLU1ERhU4sHZchBtk+d7NDAa6FE4mYwinXxbDpw/AH8o4cTXEcHnQODt1vu8hLxsmo+JlnXoSzo5rl04MU4ANiRdNAzN+emZh58kfSEXJJ2MvjynfJICz/VAn5zHF4FhhemnAUfkfdmr9Txab7e55a2E9Z65v0kn2F1IFzerFLZnH2ZdGI4qfHY8sy6MFgbWzf2nkk4k3XP3FfKFTqt9eAdwUGF+vwHOz/1DSMfZank7HAf8pzBt77y/tge+STqmexTGP0A6sfUAvpynvSyPW510QvxyHn9GXudi8Pkc2Dlvk17A90nnpkF5W/wJuCJPfwjwj5ynrsB6pFJkqeeZr5IuNlXYlp+Rgk4X0gXoCTmvK5CC2dfb2Je9SRc6W5Syv/Pw7oXl7En6ri+Zx10B/CyPKx7TX8956kcKRKsVPnMJ+ftLG98/Zg8+ZwE3kM6zffI2PLXV+f7XeXv3mutx3M5Bvg/wVjvTvAxsXxj+OjCmkJnPWq3IBGCjwgFTTvD5gHTQ9mqVh+JBsS/wcKvxDwD75/67gOMK474L3FLCl/wlYBVgZN4uswWfNj77PrnetPV6FfLxizbSvlMY3pB0FfoasHcZB+fxwFWF4S6kq9zNCyeTA9s7yc1hWXMKPpPz/vkAmFgYN4b2g8+2wOet9vkHhe6oueTnJeBbuf8gCveHSFenv2w1/QukqrqNSVel/xPkWueTdHKaCqxeSDukZf/n6V9vZx6tg88c81bCPticVNJtKaWMAvaaw7T98rL75uHXc96/0Gq6X5CC6//cl2q1D7/DrCtikU7AX83DN5MDcuG4+5Rc+iFdoL1D+j4vQLo63yWPW4Z04upd+PxlzAo+J5ADRx7unfdJMfjc0yrfzwFbFYaXJAWobqQA/T+lTUo/zyhvy5Z1P6iwXTZs43j4CXBxG9t2YN4/q5b6/W5j/ChgSO4fQaoRGNRqmi1JF0wbkUu0rb6/7QafvM6fAF8sjNsYeLWQz6nki+72uvbuM7wL9G+n7m4p0smxxWs5beY8YvZ7Op8CC1GmiPiEFOUPBcZL+qekVUvIT0ueBhaGiy3CSs3PpcD/A7YArm09UtJRkp7LLfc+IF2F929nnm/MbWREPES6YhKpyqBUs22DiJiRl1XcBnNddgXOiIh+uWtvvVsbSDqJFvUvzO+MuXx2BKk6BtKFx4jCuGWBH0n6oKUjlfaWyv9fi9LuN/YnlQZaH+fVbM+55a0Ub+Zts0hErB0RIwEkdZV0mqSXJX1EChwt6wDppLo98Jqku/P9BkglmNHAvyS9IunYOSz3GtI90CVJV/8zSFV0Let0TmF93iMduy3baSjpomhaREzO8xqaxy0FvBcRnxaWVdymSxWH83Tvtspb632wLHBtIT/PkUoZA0jf51uBkZLelHS6pO6lnmcinW1HkmpWIDXKuryw3KVa7duf5uW29j5pGy7Zxrg2SdpP0qjCvNdk1v49mrTNH86tFA/M+b2DVK38e2CCpAskfaHUZWaLkYL+Y4Vl35LTW7yT92272gs+D5DuCew8l2neJG3sFsvktEp8Qlq5FksUR0bErRGxDWlHPQ9cWEJ+WvI0rsI8tbiUVEq6qdUXBElfIe30PUhViv1IV3Vqyfoc5jmn9Jb5Hk4qvr6Z51+q2baBJJFObMVtMNdl19kuzDqBletSYKt8Et2IWScASCejUwpBrF9E9I6IK/K4ZeZwYdV620wkXTG3Ps6r2Z5zy1s1vkWq/tqadAG0XE4XQEQ8EhFDSNXk15EvaiJiUkT8KCJWIN0vPFLSVq1nHhHvk+6T7pmXNTKfiFvW6ZBW69QrIv4jaRDp6vvbkt7KPwnYDdheUn9SdeAikorf/6UL/eNJ1WdpZaRepGru2bLXavgNYLtW+VkgIsZFxOcRcVJErE6qVt+BfBFT4nkGUhXXbpKWJZV2riks99VWy+0TEdu3sT0/JZ1nvzmHZcwmL+tC0oXwovlc8wyz9u9bEXFQRCxFKuH+QdKKedy5EbEeqQpzZeDHpSyzYCKpJmuNwnr1jVkNkaCM78Fcg09EfEgq7v5e0s6SekvqLmk7Safnya4AjpO0WD6ITiAVlysxCviqpGUk9SUVVQGQNEDSEEkLkgLix6QrhtZuAlaW9C1J3STtSdrYN1aYJwAi4lVSdc3P2hjdh1Rl8A7QTdIJpPrjFm8Dy5XTok3SyqSbgN8mXdEfLWntEj9+FfANSVtJ6k66BzW
"text/plain": [
"<matplotlib.figure.Figure at 0x7fb2564c0e80>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"linear_clf = PassiveAggressiveClassifier(n_iter=50)\n",
"\n",
"linear_clf.fit(tfidf_train, y)\n",
"pred = linear_clf.predict(tfidf_test)\n",
"score = metrics.accuracy_score(yt, pred)\n",
"print(\"accuracy: %0.3f\" % score)\n",
"cm = metrics.confusion_matrix(yt, pred, labels=['FAKE', 'REAL'])\n",
"plot_confusion_matrix(cm, classes=['FAKE', 'REAL'], title= \"TFIDF Vectorite, PassiveAggressive Classifier\")"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jonas/.local/lib/python3.6/site-packages/sklearn/naive_bayes.py:472: UserWarning: alpha too small will result in numeric errors, setting alpha = 1.0e-10\n",
" 'setting alpha = %.1e' % _ALPHA_MIN)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Alpha: 0.00 Score: 0.87849\n",
"Alpha: 0.10 Score: 0.91215\n",
"Alpha: 0.20 Score: 0.90637\n",
"Alpha: 0.30 Score: 0.90005\n",
"Alpha: 0.40 Score: 0.89216\n",
"Alpha: 0.50 Score: 0.88795\n",
"Alpha: 0.60 Score: 0.88217\n",
"Alpha: 0.70 Score: 0.87217\n",
"Alpha: 0.80 Score: 0.86902\n",
"Alpha: 0.90 Score: 0.86113\n"
]
}
],
"source": [
"clf = MultinomialNB(alpha=0.1)\n",
"last_score = 0\n",
"for alpha in np.arange(0,1,.1):\n",
" nb_classifier = MultinomialNB(alpha=alpha)\n",
" nb_classifier.fit(tfidf_train, y)\n",
" pred = nb_classifier.predict(tfidf_test)\n",
" score = metrics.accuracy_score(yt, pred)\n",
" if score > last_score:\n",
" clf = nb_classifier\n",
" print(\"Alpha: {:.2f} Score: {:.5f}\".format(alpha, score))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* try to get most important features"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"FAKE -4.986418446992282 2016\n",
"FAKE -4.031739222213152 october\n",
"FAKE -3.2450737607438835 hillary\n",
"FAKE -3.163046832110649 article\n",
"FAKE -3.0797196307769865 november\n",
"FAKE -2.9126602525203786 election\n",
"FAKE -2.7767455973246777 share\n",
"FAKE -2.5799080044431215 establishment\n",
"FAKE -2.5391003972219663 wikileaks\n",
"FAKE -2.5124769239037335 mosul\n",
"FAKE -2.5044337732634636 source\n",
"FAKE -2.376392497005016 oct\n",
"FAKE -2.323456790625324 print\n",
"FAKE -2.296605039295202 advertisement\n",
"FAKE -2.1765893008482884 podesta\n",
"FAKE -2.1254730787507397 corporate\n",
"FAKE -2.1186888933652006 comments\n",
"FAKE -2.0814842675932406 russia\n",
"FAKE -1.9405914175220103 watch\n",
"FAKE -1.8706195854259284 war\n",
"FAKE -1.867386639102956 posted\n",
"FAKE -1.8056831543703649 com\n",
"FAKE -1.8054376409136181 navigation\n",
"FAKE -1.7877228165152776 26\n",
"FAKE -1.7685005227604957 stated\n",
"FAKE -1.7402325468939963 dakota\n",
"FAKE -1.7271994282637921 jewish\n",
"FAKE -1.7263968946984054 ayotte\n",
"FAKE -1.7200570381137112 donald\n",
"FAKE -1.6535827908878833 pipeline\n",
"\n",
"REAL 5.015301141104506 said\n",
"REAL 3.0775436014367448 says\n",
"REAL 2.6550720089727093 say\n",
"REAL 2.5730784312333856 gop\n",
"REAL 2.537624697551335 debate\n",
"REAL 2.37704059797269 islamic\n",
"REAL 2.3533175343115773 friday\n",
"REAL 2.324743477167939 jobs\n",
"REAL 2.285274612684968 conservative\n",
"REAL 2.2415729758446785 marriage\n",
"REAL 2.2145701146991454 rush\n",
"REAL 2.1909548260953593 tuesday\n",
"REAL 2.164804177946772 continue\n",
"REAL 2.1533786245547075 fox\n",
"REAL 2.134959398360091 cruz\n",
"REAL 1.9490044497446428 manafort\n",
"REAL 1.9042544756391377 candidates\n",
"REAL 1.8970529284348485 convention\n",
"REAL 1.8966333613592279 parties\n",
"REAL 1.8807450356106956 recounts\n",
"REAL 1.807064184984985 paris\n",
"REAL 1.8062833270371794 state\n",
"REAL 1.7996444826854578 decision\n",
"REAL 1.7900476239131307 prices\n",
"REAL 1.7517035944186097 shooting\n",
"REAL 1.7481252452408131 coverage\n",
"REAL 1.7370073447314753 nbc\n",
"REAL 1.716341346568611 security\n",
"REAL 1.664421277998289 wni9lmsppr\n",
"REAL 1.6244118186760108 baltimore\n"
]
}
],
"source": [
"def most_informative_feature_for_binary_classification(vectorizer, classifier, n=100):\n",
" \"\"\"\n",
" See: https://stackoverflow.com/a/26980472\n",
" \n",
" Identify most important features if given a vectorizer and binary classifier. Set n to the number\n",
" of weighted features you would like to show. (Note: current implementation merely prints and does not \n",
" return top classes.)\n",
" \"\"\"\n",
"\n",
" class_labels = classifier.classes_\n",
" feature_names = vectorizer.get_feature_names()\n",
" topn_class1 = sorted(zip(classifier.coef_[0], feature_names))[:n]\n",
" topn_class2 = sorted(zip(classifier.coef_[0], feature_names))[-n:]\n",
"\n",
" for coef, feat in topn_class1:\n",
" print(class_labels[0], coef, feat)\n",
"\n",
" print()\n",
"\n",
" for coef, feat in reversed(topn_class2):\n",
" print(class_labels[1], coef, feat)\n",
"\n",
"\n",
"most_informative_feature_for_binary_classification(tfidf_vectorizer, linear_clf, n=30)\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* another way to perform this"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[(5.015301141104506, 'said'),\n",
" (3.0775436014367448, 'says'),\n",
" (2.6550720089727093, 'say'),\n",
" (2.5730784312333856, 'gop'),\n",
" (2.537624697551335, 'debate'),\n",
" (2.37704059797269, 'islamic'),\n",
" (2.3533175343115773, 'friday'),\n",
" (2.324743477167939, 'jobs'),\n",
" (2.285274612684968, 'conservative'),\n",
" (2.2415729758446785, 'marriage'),\n",
" (2.2145701146991454, 'rush'),\n",
" (2.1909548260953593, 'tuesday'),\n",
" (2.164804177946772, 'continue'),\n",
" (2.1533786245547075, 'fox'),\n",
" (2.134959398360091, 'cruz'),\n",
" (1.9490044497446428, 'manafort'),\n",
" (1.9042544756391377, 'candidates'),\n",
" (1.8970529284348485, 'convention'),\n",
" (1.8966333613592279, 'parties'),\n",
" (1.8807450356106956, 'recounts')]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"[(-4.986418446992282, '2016'),\n",
" (-4.031739222213152, 'october'),\n",
" (-3.2450737607438835, 'hillary'),\n",
" (-3.163046832110649, 'article'),\n",
" (-3.0797196307769865, 'november'),\n",
" (-2.9126602525203786, 'election'),\n",
" (-2.7767455973246777, 'share'),\n",
" (-2.5799080044431215, 'establishment'),\n",
" (-2.5391003972219663, 'wikileaks'),\n",
" (-2.5124769239037335, 'mosul'),\n",
" (-2.5044337732634636, 'source'),\n",
" (-2.376392497005016, 'oct'),\n",
" (-2.323456790625324, 'print'),\n",
" (-2.296605039295202, 'advertisement'),\n",
" (-2.1765893008482884, 'podesta'),\n",
" (-2.1254730787507397, 'corporate'),\n",
" (-2.1186888933652006, 'comments'),\n",
" (-2.0814842675932406, 'russia'),\n",
" (-1.9405914175220103, 'watch'),\n",
" (-1.8706195854259284, 'war')]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"feature_names = tfidf_vectorizer.get_feature_names()\n",
"### Most real\n",
"display(sorted(zip(linear_clf.coef_[0], feature_names), reverse=True)[:20])\n",
"### Most fake\n",
"display(sorted(zip(linear_clf.coef_[0], feature_names))[:20])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* analyse token weights"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[('00', -10.806288039676962),\n",
" ('000', -8.4757571521028),\n",
" ('0000', -11.31983755527844),\n",
" ('00000031', -11.306788627092281),\n",
" ('000035', -11.375220169061265),\n",
" ('00006', -11.285092170704173),\n",
" ('0002', -11.375220169061265),\n",
" ('000ft', -11.020175960097573),\n",
" ('000x', -11.346623773737496),\n",
" ('001', -11.224561919245287),\n",
" ('0011', -11.375220169061265),\n",
" ('002', -11.253666274798343),\n",
" ('003', -11.226941303318013),\n",
" ('004', -11.303970982653903),\n",
" ('005', -11.375220169061265),\n",
" ('00684', -11.375220169061265),\n",
" ('006s', -11.375220169061265),\n",
" ('007', -11.375220169061265),\n",
" ('007s', -11.375220169061265),\n",
" ('008', -11.270210919871277)]"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tokens_with_weights = sorted(list(zip(feature_names, clf.coef_[0])))\n",
"tokens_with_weights[:20]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"----\n",
"## Building an own classifier for the 'pants on fire' dataset\n"
2018-05-09 19:13:08 +02:00
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
2018-05-09 10:47:51 +02:00
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}