{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# NLP-LAB Exercise 01 by Jonas Weinz\n", "## links:\n", "\n", "* Article: https://miguelmalvarez.com/2017/03/23/how-can-machine-learning-and-ai-help-solving-the-fake-news-problem/\n", " * corresponding code: https://github.com/kjam/random_hackery/blob/master/Attempting%20to%20detect%20fake%20news.ipynb\n", "\n", "* Tutorial on Datacamp: https://www.datacamp.com/community/tutorials/scikit-learn-fake-news\n", "\n", "* liar dataset paper: https://www.cs.ucsb.edu/~william/papers/acl2017.pdf\n", " * dataset: https://www.cs.ucsb.edu/~william/data/liar_dataset.zip" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Dependencies for this Notebook:\n", "* library [rdflib](https://github.com/RDFLib/rdflib)\n", " * install: `pip3 install rdflib`\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Populating the interactive namespace from numpy and matplotlib\n" ] } ], "source": [ "%pylab inline" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import itertools\n", "import sklearn.utils as sku\n", "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, HashingVectorizer\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.linear_model import PassiveAggressiveClassifier\n", "from sklearn.naive_bayes import MultinomialNB\n", "from sklearn import metrics\n", "import matplotlib.pyplot as plt\n", "from pprint import pprint as pp\n", "import os" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tools used later" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "def plot_confusion_matrix(cm, classes,\n", " title,\n", " normalize=False,\n", " cmap=plt.cm.Blues):\n", " fig_1, ax_1 = plt.subplots()\n", " \"\"\"\n", " See full source and example: \n", " http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html\n", " \n", " This function prints and plots the confusion matrix.\n", " Normalization can be applied by setting `normalize=True`.\n", " \"\"\"\n", " plt.imshow(cm, interpolation='nearest', cmap=cmap)\n", " plt.title('Confusion Matrix for: ' + title)\n", " plt.colorbar()\n", " tick_marks = np.arange(len(classes))\n", " plt.xticks(tick_marks, classes, rotation=45)\n", " plt.yticks(tick_marks, classes)\n", "\n", " if normalize:\n", " cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n", " print(\"Normalized confusion matrix\")\n", " else:\n", " print('Confusion matrix, without normalization')\n", "\n", " thresh = cm.max() / 2.\n", " for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n", " plt.text(j, i, cm[i, j],\n", " horizontalalignment=\"center\",\n", " color=\"white\" if cm[i, j] > thresh else \"black\")\n", "\n", " plt.tight_layout()\n", " plt.ylabel('True label')\n", " plt.xlabel('Predicted label')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generate/Download Datasets we are working on\n", "\n", "* running bash script to download all needed data and store it into the `data` subfolder" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "================================================================================\n", "checking whether unzip is installed\n", "================================================================================\n", "UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.\n", "\n", "Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;\n", "see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.\n", "\n", "Compiled with gcc 6.3.0 20170415 for Unix (Linux ELF).\n", "\n", "UnZip special compilation options:\n", " ACORN_FTYPE_NFS\n", " COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)\n", " SET_DIR_ATTRIB\n", " SYMLINKS (symbolic links supported, if RTL and file system permit)\n", " TIMESTAMP\n", " UNIXBACKUP\n", " USE_EF_UT_TIME\n", " USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)\n", " USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)\n", " UNICODE_SUPPORT [wide-chars, char coding: UTF-8] (handle UTF-8 paths)\n", " LARGE_FILE_SUPPORT (large files over 2 GiB supported)\n", " ZIP64_SUPPORT (archives using Zip64 for large files supported)\n", " USE_BZIP2 (PKZIP 4.6+, using bzip2 lib version 1.0.6, 6-Sept-2010)\n", " VMS_TEXT_CONV\n", " WILD_STOP_AT_DIR\n", " [decryption, version 2.11 of 05 Jan 2007]\n", "\n", "UnZip and ZipInfo environment options:\n", " UNZIP: [none]\n", " UNZIPOPT: [none]\n", " ZIPINFO: [none]\n", " ZIPINFOOPT: [none]\n", "================================================================================\n", "successfully finished action: checking whether unzip is installed\n", "================================================================================\n", "================================================================================\n", "downloading and unpacking https://www.cs.ucsb.edu/~william/data/liar_dataset.zip if not already existing\n", "================================================================================\n", "================================================================================\n", "successfully finished action: downloading and unpacking https://www.cs.ucsb.edu/~william/data/liar_dataset.zip if not already existing\n", "================================================================================\n", "================================================================================\n", "downloading and unpacking https://raw.githubusercontent.com/GeorgeMcIntire/fake_real_news_dataset/master/fake_or_real_news.csv.zip if not already existing\n", "================================================================================\n", "================================================================================\n", "successfully finished action: downloading and unpacking https://raw.githubusercontent.com/GeorgeMcIntire/fake_real_news_dataset/master/fake_or_real_news.csv.zip if not already existing\n", "================================================================================\n" ] } ], "source": [ "%%bash\n", "./Task_2_gen_data.sh" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Read in fake news table" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "df_1 = pd.read_csv('data/fake_or_real_news.csv').set_index('Unnamed: 0')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* display first 10 entries" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(6335, 3)" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", " | title | \n", "text | \n", "label | \n", "
---|---|---|---|
Unnamed: 0 | \n", "\n", " | \n", " | \n", " |
8476 | \n", "You Can Smell Hillary’s Fear | \n", "Daniel Greenfield, a Shillman Journalism Fello... | \n", "FAKE | \n", "
10294 | \n", "Watch The Exact Moment Paul Ryan Committed Pol... | \n", "Google Pinterest Digg Linkedin Reddit Stumbleu... | \n", "FAKE | \n", "
3608 | \n", "Kerry to go to Paris in gesture of sympathy | \n", "U.S. Secretary of State John F. Kerry said Mon... | \n", "REAL | \n", "
10142 | \n", "Bernie supporters on Twitter erupt in anger ag... | \n", "— Kaydee King (@KaydeeKing) November 9, 2016 T... | \n", "FAKE | \n", "
875 | \n", "The Battle of New York: Why This Primary Matters | \n", "It's primary day in New York and front-runners... | \n", "REAL | \n", "
6903 | \n", "Tehran, USA | \n", "\\nI’m not an immigrant, but my grandparents ... | \n", "FAKE | \n", "
7341 | \n", "Girl Horrified At What She Watches Boyfriend D... | \n", "Share This Baylee Luciani (left), Screenshot o... | \n", "FAKE | \n", "
95 | \n", "‘Britain’s Schindler’ Dies at 106 | \n", "A Czech stockbroker who saved more than 650 Je... | \n", "REAL | \n", "
4869 | \n", "Fact check: Trump and Clinton at the 'commande... | \n", "Hillary Clinton and Donald Trump made some ina... | \n", "REAL | \n", "
2909 | \n", "Iran reportedly makes new push for uranium con... | \n", "Iranian negotiators reportedly have made a las... | \n", "REAL | \n", "