nlp-lab/Jonas_Solutions/Task_02_JonasWeinz.ipynb

655 lines
23 KiB
Plaintext
Raw Normal View History

2018-05-09 10:47:51 +02:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# NLP-LAB Exercise 01 by jonas weinz\n",
"----\n",
2018-05-09 16:50:59 +02:00
"## links:\n",
"\n",
"* Article: https://miguelmalvarez.com/2017/03/23/how-can-machine-learning-and-ai-help-solving-the-fake-news-problem/\n",
" * corresponding code: https://github.com/kjam/random_hackery/blob/master/Attempting%20to%20detect%20fake%20news.ipynb\n",
"\n",
"* Tutorial on Datacamp: https://www.datacamp.com/community/tutorials/scikit-learn-fake-news\n",
"\n",
"* liar dataset paper: https://www.cs.ucsb.edu/~william/papers/acl2017.pdf\n",
2018-05-09 19:13:08 +02:00
" * dataset: https://www.cs.ucsb.edu/~william/data/liar_dataset.zip"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2018-05-09 16:50:59 +02:00
"## Dependencies for this Notebook:\n",
2018-05-09 10:47:51 +02:00
"* library [rdflib](https://github.com/RDFLib/rdflib)\n",
2018-05-09 19:13:08 +02:00
" * install: `pip3 install rdflib`\n"
2018-05-09 10:47:51 +02:00
]
},
{
"cell_type": "code",
2018-05-09 19:13:08 +02:00
"execution_count": 1,
2018-05-09 10:47:51 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Populating the interactive namespace from numpy and matplotlib\n"
]
}
],
"source": [
2018-05-09 16:50:59 +02:00
"%pylab ipympl"
]
},
{
"cell_type": "code",
2018-05-09 19:13:08 +02:00
"execution_count": 21,
2018-05-09 16:50:59 +02:00
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import itertools\n",
"import sklearn.utils as sku\n",
2018-05-09 19:13:08 +02:00
"from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, HashingVectorizer\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.linear_model import PassiveAggressiveClassifier\n",
"from sklearn.naive_bayes import MultinomialNB\n",
"from sklearn import metrics\n",
"import matplotlib.pyplot as plt\n",
"from pprint import pprint as pp\n",
2018-05-09 16:50:59 +02:00
"import os"
]
},
2018-05-09 19:13:08 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Tools used later"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [],
"source": [
"def plot_confusion_matrix(cm, classes,\n",
" title,\n",
" normalize=False,\n",
" cmap=plt.cm.Blues):\n",
" fig_1, ax_1 = plt.subplots()\n",
" \"\"\"\n",
" See full source and example: \n",
" http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html\n",
" \n",
" This function prints and plots the confusion matrix.\n",
" Normalization can be applied by setting `normalize=True`.\n",
" \"\"\"\n",
" plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
" plt.title('Confusion Matrix for: ' + title)\n",
" plt.colorbar()\n",
" tick_marks = np.arange(len(classes))\n",
" plt.xticks(tick_marks, classes, rotation=45)\n",
" plt.yticks(tick_marks, classes)\n",
"\n",
" if normalize:\n",
" cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
" print(\"Normalized confusion matrix\")\n",
" else:\n",
" print('Confusion matrix, without normalization')\n",
"\n",
" thresh = cm.max() / 2.\n",
" for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
" plt.text(j, i, cm[i, j],\n",
" horizontalalignment=\"center\",\n",
" color=\"white\" if cm[i, j] > thresh else \"black\")\n",
"\n",
" plt.tight_layout()\n",
" plt.ylabel('True label')\n",
" plt.xlabel('Predicted label')"
]
},
2018-05-09 16:50:59 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Generate/Download Datasets we are working on\n",
"\n",
"* running bash script to download all needed data and store it into the `data` subfolder"
]
},
{
"cell_type": "code",
2018-05-09 19:13:08 +02:00
"execution_count": 3,
2018-05-09 16:50:59 +02:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"================================================================================\n",
"downloading and unpacking https://www.cs.ucsb.edu/~william/data/liar_dataset.zip if not already existing\n",
"================================================================================\n",
"UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.\n",
"\n",
"Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;\n",
"see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.\n",
"\n",
"Compiled with gcc 6.3.0 20170415 for Unix (Linux ELF).\n",
"\n",
"UnZip special compilation options:\n",
" ACORN_FTYPE_NFS\n",
" COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)\n",
" SET_DIR_ATTRIB\n",
" SYMLINKS (symbolic links supported, if RTL and file system permit)\n",
" TIMESTAMP\n",
" UNIXBACKUP\n",
" USE_EF_UT_TIME\n",
" USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)\n",
" USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)\n",
" UNICODE_SUPPORT [wide-chars, char coding: UTF-8] (handle UTF-8 paths)\n",
" LARGE_FILE_SUPPORT (large files over 2 GiB supported)\n",
" ZIP64_SUPPORT (archives using Zip64 for large files supported)\n",
" USE_BZIP2 (PKZIP 4.6+, using bzip2 lib version 1.0.6, 6-Sept-2010)\n",
" VMS_TEXT_CONV\n",
" WILD_STOP_AT_DIR\n",
" [decryption, version 2.11 of 05 Jan 2007]\n",
"\n",
"UnZip and ZipInfo environment options:\n",
" UNZIP: [none]\n",
" UNZIPOPT: [none]\n",
" ZIPINFO: [none]\n",
" ZIPINFOOPT: [none]\n",
"================================================================================\n",
"successfully finished action: downloading and unpacking https://www.cs.ucsb.edu/~william/data/liar_dataset.zip if not already existing\n",
"================================================================================\n",
"================================================================================\n",
"downloading and unpacking https://raw.githubusercontent.com/GeorgeMcIntire/fake_real_news_dataset/master/fake_or_real_news.csv.zip if not already existing\n",
"================================================================================\n",
"================================================================================\n",
"successfully finished action: downloading and unpacking https://raw.githubusercontent.com/GeorgeMcIntire/fake_real_news_dataset/master/fake_or_real_news.csv.zip if not already existing\n",
"================================================================================\n"
]
}
],
"source": [
"%%bash\n",
"./Task_2_gen_data.sh"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Read in fake news table"
]
},
{
"cell_type": "code",
2018-05-09 19:13:08 +02:00
"execution_count": 4,
2018-05-09 16:50:59 +02:00
"metadata": {},
"outputs": [],
"source": [
"df_1 = pd.read_csv('data/fake_or_real_news.csv').set_index('Unnamed: 0')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* display first 10 entries"
]
},
{
"cell_type": "code",
2018-05-09 19:13:08 +02:00
"execution_count": 5,
2018-05-09 16:50:59 +02:00
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(6335, 3)"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>label</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Unnamed: 0</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>8476</th>\n",
" <td>You Can Smell Hillarys Fear</td>\n",
" <td>Daniel Greenfield, a Shillman Journalism Fello...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10294</th>\n",
" <td>Watch The Exact Moment Paul Ryan Committed Pol...</td>\n",
" <td>Google Pinterest Digg Linkedin Reddit Stumbleu...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3608</th>\n",
" <td>Kerry to go to Paris in gesture of sympathy</td>\n",
" <td>U.S. Secretary of State John F. Kerry said Mon...</td>\n",
" <td>REAL</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10142</th>\n",
" <td>Bernie supporters on Twitter erupt in anger ag...</td>\n",
" <td>— Kaydee King (@KaydeeKing) November 9, 2016 T...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>875</th>\n",
" <td>The Battle of New York: Why This Primary Matters</td>\n",
" <td>It's primary day in New York and front-runners...</td>\n",
" <td>REAL</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6903</th>\n",
" <td>Tehran, USA</td>\n",
" <td>\\nIm not an immigrant, but my grandparents ...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7341</th>\n",
" <td>Girl Horrified At What She Watches Boyfriend D...</td>\n",
" <td>Share This Baylee Luciani (left), Screenshot o...</td>\n",
" <td>FAKE</td>\n",
" </tr>\n",
" <tr>\n",
" <th>95</th>\n",
" <td>Britains Schindler Dies at 106</td>\n",
" <td>A Czech stockbroker who saved more than 650 Je...</td>\n",
" <td>REAL</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4869</th>\n",
" <td>Fact check: Trump and Clinton at the 'commande...</td>\n",
" <td>Hillary Clinton and Donald Trump made some ina...</td>\n",
" <td>REAL</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2909</th>\n",
" <td>Iran reportedly makes new push for uranium con...</td>\n",
" <td>Iranian negotiators reportedly have made a las...</td>\n",
" <td>REAL</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" title \\\n",
"Unnamed: 0 \n",
"8476 You Can Smell Hillarys Fear \n",
"10294 Watch The Exact Moment Paul Ryan Committed Pol... \n",
"3608 Kerry to go to Paris in gesture of sympathy \n",
"10142 Bernie supporters on Twitter erupt in anger ag... \n",
"875 The Battle of New York: Why This Primary Matters \n",
"6903 Tehran, USA \n",
"7341 Girl Horrified At What She Watches Boyfriend D... \n",
"95 Britains Schindler Dies at 106 \n",
"4869 Fact check: Trump and Clinton at the 'commande... \n",
"2909 Iran reportedly makes new push for uranium con... \n",
"\n",
" text label \n",
"Unnamed: 0 \n",
"8476 Daniel Greenfield, a Shillman Journalism Fello... FAKE \n",
"10294 Google Pinterest Digg Linkedin Reddit Stumbleu... FAKE \n",
"3608 U.S. Secretary of State John F. Kerry said Mon... REAL \n",
"10142 — Kaydee King (@KaydeeKing) November 9, 2016 T... FAKE \n",
"875 It's primary day in New York and front-runners... REAL \n",
"6903 \\nIm not an immigrant, but my grandparents ... FAKE \n",
"7341 Share This Baylee Luciani (left), Screenshot o... FAKE \n",
"95 A Czech stockbroker who saved more than 650 Je... REAL \n",
"4869 Hillary Clinton and Donald Trump made some ina... REAL \n",
"2909 Iranian negotiators reportedly have made a las... REAL "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(df_1.shape)\n",
"display(df_1[:10])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* create test dataset"
2018-05-09 10:47:51 +02:00
]
},
{
"cell_type": "code",
2018-05-09 19:13:08 +02:00
"execution_count": 6,
2018-05-09 10:47:51 +02:00
"metadata": {},
"outputs": [],
2018-05-09 16:50:59 +02:00
"source": [
2018-05-09 19:13:08 +02:00
"def create_training_and_test_set(dset, cutoff=0.7):\n",
" shuffled = sku.shuffle(dset) # shuffle dataset\n",
2018-05-09 16:50:59 +02:00
" y = shuffled.label\n",
2018-05-09 19:13:08 +02:00
" shuffled = shuffled.drop('label', axis=1)['text']\n",
" size = int(cutoff * shuffled.shape[0])\n",
" return shuffled[:size], y[:size], shuffled[size:], y[size:]\n",
2018-05-09 16:50:59 +02:00
" "
]
},
{
"cell_type": "code",
2018-05-09 19:13:08 +02:00
"execution_count": 7,
2018-05-09 16:50:59 +02:00
"metadata": {},
"outputs": [],
"source": [
2018-05-09 19:13:08 +02:00
"X,y, Xt,yt = create_training_and_test_set(df_1)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"#X2,y2, Xt2,yt2 = train_test_split(df_1['text'],df_1.label,test_size=0.3)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Unnamed: 0\n",
"5507 Anthony Weiner Sends Apology Sext To Entire Cl...\n",
"1440 John Kasich was killing it with these Iowa vot...\n",
"3172 Yeah, yeah, with the rise of Der Trumper and t...\n",
"6962 Vermont fights the opioid epidemic by limiting...\n",
"6970 — Adam Baldwin (@AdamBaldwin) October 28, 2016...\n",
"7209 Militias prepare for election unrest while Chr...\n",
"4781 Raleigh, North Carolina (CNN) As soon as the f...\n",
"6137 Yemens Hudaydah suffering from dire humanitar...\n",
"8799 Subscribe \\nIn politics, the Third Way is a po...\n",
"4374 Hanging in the balance is Obamas vision of Am...\n",
"9465 Under the Surface - Naomi Klein and the Great ...\n",
"10409 October 28, 2016 at 9:00 PM \\nWhy would Putin ...\n",
"2822 HAMPTON, N.H. -- Hillary Rodham Clinton said F...\n",
"5190 (CNN) Donald Trump on Wednesday refused to say...\n",
"2286 Gay rights won't fade as a political issue. Th...\n",
"4308 Fox Businesss Tuesday night Republican debate...\n",
"7305 Yes, There Are Paid Government Trolls On Socia...\n",
"6716 Vladimir Putin: The United States continues to...\n",
"6871 By wmw_admin on October 31, 2016 Sanchez Manni...\n",
"8527 Banana Republic Election in the United States?...\n",
"3658 Why are Americans expected to die sooner than ...\n",
"9721 Comments \\nFamous techno musician Moby tore in...\n",
"10001 Behind the headlines - conspiracies, cover-ups...\n",
"8920 Massachusetts +11% New York +10% \\nWhen home p...\n",
"8488 Ivana Says Young Donald Trump Was A Cry-Baby ...\n",
"8474 Recipient Email => \\nEnemies of the United S...\n",
"5789 Theresa May refuses to withdraw support for Sa...\n",
"1324 Even as Hillary Clinton has stepped up her rhe...\n",
"1744 Sally Kohn is an activist, columnist and telev...\n",
"3887 George Washington was the first President of t...\n",
" ... \n",
"643 Is anyone qualified to judge The Donald?\\n\\nDo...\n",
"6795 [Graphic: Calais street scene by Harriet Paint...\n",
"10444 By wmw_admin on October 30, 2016 By Timothy Fi...\n",
"8696 Unprecedented Surge In Election Fraud Incident...\n",
"5103 The Democratic National Committee is offering ...\n",
"8018 Comments \\nFOX News star Megyn Kelly has final...\n",
"2896 President Obama sent a draft Authorization for...\n",
"9188 Heseltine strangled dog as part of Thatcher ca...\n",
"6 It's been a big week for abortion news.\\n\\nCar...\n",
"9778 20 Foods That Naturally Unclog Arteries and Pr...\n",
"10062 FBI Releases Files on Bill Clinton's Cash for ...\n",
"8556 NTEB Ads Privacy Policy Black Americans Going ...\n",
"9757 10-27-1 6 The first Bill and Hillary Clinton c...\n",
"2524 President Obama's executive action preventing ...\n",
"8995 Rights? In The New America You Dont Get Any R...\n",
"8602 By Amanda Froelich For a long time, green juic...\n",
"2886 W hen trying to explain the current unrest in ...\n",
"3456 WASHINGTON, June 21 (Reuters) - Tensions are b...\n",
"6564 This is how it works in the Clinton Cabal...or...\n",
"8406 posted by Eddie Angelina Jolies father, Jon V...\n",
"1151 As the Democratic presidential contest reaches...\n",
"3233 With the Department of Homeland Securitys fun...\n",
"4601 Enough Is Enough\\n\\nIf we really want to do so...\n",
"305 Top Dems want White House to call off Part B d...\n",
"842 Texas Sen. Ted Cruzs popularity among Republi...\n",
"7085 Email \\nNot all invasions are hot — not all in...\n",
"530 President Obama's 2016 budget seeks higher spe...\n",
"5766 \\nThe decision of FBI Director Comey to go p...\n",
"976 Republican presidential candidate Ted Cruz is ...\n",
"2241 The simmering dispute over media access to Hil...\n",
"Name: text, Length: 4434, dtype: object"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"count_vectorizer = CountVectorizer(stop_words='english')\n",
"count_train = count_vectorizer.fit_transform(X)\n",
"count_test = count_vectorizer.transform(Xt)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"tfidf_vectorizer = TfidfVectorizer(stop_words='english', max_df=0.7)\n",
"tfidf_train = tfidf_vectorizer.fit_transform(X)\n",
"tfidf_test = tfidf_vectorizer.transform(Xt)"
2018-05-09 16:50:59 +02:00
]
2018-05-09 19:13:08 +02:00
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['00',\n",
" '000',\n",
" '0000',\n",
" '00000031',\n",
" '000035',\n",
" '00006',\n",
" '0001',\n",
" '0001pt',\n",
" '0002',\n",
" '000billion']\n",
"['حلب', 'عربي', 'عن', 'لم', 'ما', 'محاولات', 'من', 'هذا', 'والمرضى', 'ยงade']\n"
]
}
],
"source": [
"pp(count_vectorizer.get_feature_names()[0:10])\n",
"pp(count_vectorizer.get_feature_names()[-10:])\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['00',\n",
" '000',\n",
" '0000',\n",
" '00000031',\n",
" '000035',\n",
" '00006',\n",
" '0001',\n",
" '0001pt',\n",
" '0002',\n",
" '000billion']\n",
"['حلب', 'عربي', 'عن', 'لم', 'ما', 'محاولات', 'من', 'هذا', 'والمرضى', 'ยงade']\n"
]
}
],
"source": [
"pp(tfidf_vectorizer.get_feature_names()[:10])\n",
"pp(tfidf_vectorizer.get_feature_names()[-10:])"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"#count_df = pd.DataFrame(count_train.A, columns=count_vectorizer.get_feature_names())\n",
"#tfidf_df = pd.DataFrame(count_train.A, columns=tfidf_vectorizer.get_feature_names())\n",
"#diff = set(count_df.columns) - set(tfidf_df.columns)\n",
"#pp(count_df.equals(tfidf_df))"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"'score: 0.8527091004734351'\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "074a58de5e904298a7c212800fdc812b",
"version_major": 2,
"version_minor": 0
},
"text/html": [
"<p>Failed to display Jupyter Widget of type <code>FigureCanvasNbAgg</code>.</p>\n",
"<p>\n",
" If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
" that the widgets JavaScript is still loading. If this message persists, it\n",
" likely means that the widgets JavaScript library is either not installed or\n",
" not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
" Widgets Documentation</a> for setup instructions.\n",
"</p>\n",
"<p>\n",
" If you're reading this message in another frontend (for example, a static\n",
" rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
" it may mean that your frontend doesn't currently support widgets.\n",
"</p>\n"
],
"text/plain": [
"FigureCanvasNbAgg()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Confusion matrix, without normalization\n"
]
}
],
"source": [
"clf = MultinomialNB()\n",
"clf.fit(tfidf_train, y)\n",
"pred = clf.predict(tfidf_test)\n",
"score = metrics.accuracy_score(yt, pred)\n",
"pp(\"score: \" + str(score))\n",
"cm = metrics.confusion_matrix(yt, pred, labels=[\"FAKE\", \"REAL\"])\n",
"plot_confusion_matrix(cm, classes=[\"FAKE\", \"REAL\"], title= \"tfidf\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
2018-05-09 10:47:51 +02:00
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}