added simple approach

This commit is contained in:
Jonas Weinz 2018-06-05 11:34:14 +02:00
parent 12f99af9a9
commit f5e72e8d21
2 changed files with 996 additions and 0 deletions

View File

@ -0,0 +1,6 @@
# Simple Approach
----
just a slightly better approach than the naive one 💁

View File

@ -0,0 +1,990 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from IPython.display import clear_output, Markdown, Math\n",
"import ipywidgets as widgets\n",
"import os\n",
"import glob\n",
"import json"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# simple twitter approach\n",
"*for learning emoji usage by single (in the meaning of unconnected) twitter messages*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## loading train data"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"data_root_folder = \"./data_en/\" # i created a symlink here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* get all json files in `data_root_folder`"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"json_files = sorted(glob.glob(data_root_folder + \"/*.json\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"----"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* so far, only load the first file"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>EMOJI</th>\n",
" <th>datetime</th>\n",
" <th>id</th>\n",
" <th>lang</th>\n",
" <th>person</th>\n",
" <th>reply_to</th>\n",
" <th>text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>[🔥, 👏]</td>\n",
" <td>2017-11-01 13:29:00</td>\n",
" <td>925716304635547600</td>\n",
" <td>en</td>\n",
" <td>31507978</td>\n",
" <td>NaN</td>\n",
" <td>fashionbombdaily's photo &lt;EMOJI&gt;&lt;EMOJI&gt;🏼</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>[🤦]</td>\n",
" <td>2017-11-01 13:29:00</td>\n",
" <td>925716304664911900</td>\n",
" <td>en</td>\n",
" <td>231994649</td>\n",
" <td>NaN</td>\n",
" <td>Its scary how on point my horoscope be &lt;EMOJI...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>[🤧]</td>\n",
" <td>2017-11-01 13:29:00</td>\n",
" <td>925716304652308500</td>\n",
" <td>en</td>\n",
" <td>78123421</td>\n",
" <td>NaN</td>\n",
" <td>@mvyn_: Them boring ass licks, you supposed t...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>[🐯, 🐥, 💕]</td>\n",
" <td>2017-11-01 13:29:01</td>\n",
" <td>925716308846661600</td>\n",
" <td>en</td>\n",
" <td>860396470540705800</td>\n",
" <td>NaN</td>\n",
" <td>@taehyungpic: #vmin s unit shoot &lt;EMOJI&gt;&lt;EM...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>[👍, 👍, 👍]</td>\n",
" <td>2017-11-01 13:29:02</td>\n",
" <td>925716313015586800</td>\n",
" <td>en</td>\n",
" <td>775381892321333200</td>\n",
" <td>NaN</td>\n",
" <td>@risa_del: Can I just say how much Meng respe...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>[😒]</td>\n",
" <td>2017-11-01 13:29:02</td>\n",
" <td>925716313032364000</td>\n",
" <td>en</td>\n",
" <td>3244140547</td>\n",
" <td>NaN</td>\n",
" <td>@Melodia_Muse: omg did y'all just cut off Red...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>[✅, 🍟]</td>\n",
" <td>2017-11-01 13:29:02</td>\n",
" <td>925716313036722200</td>\n",
" <td>en</td>\n",
" <td>4891498023</td>\n",
" <td>NaN</td>\n",
" <td>@Harrys1DEmpire: Follow everyone who LIKES th...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>[👶]</td>\n",
" <td>2017-11-01 13:29:03</td>\n",
" <td>925716317243555800</td>\n",
" <td>en</td>\n",
" <td>2660072274</td>\n",
" <td>NaN</td>\n",
" <td>@taehyungpic: he is so smollll &lt;EMOJI&gt;©rocket...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>[😄]</td>\n",
" <td>2017-11-01 13:29:03</td>\n",
" <td>925716317214089200</td>\n",
" <td>en</td>\n",
" <td>2592765104</td>\n",
" <td>NaN</td>\n",
" <td>Woooaaaahhh &lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>[🌟, 🌟]</td>\n",
" <td>2017-11-01 13:29:03</td>\n",
" <td>925716317218508800</td>\n",
" <td>en</td>\n",
" <td>3011085514</td>\n",
" <td>NaN</td>\n",
" <td>@mymermaidIagoon: &lt;EMOJI&gt; HURRY! For your cha...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>[📷]</td>\n",
" <td>2017-11-01 13:29:04</td>\n",
" <td>925716321416949800</td>\n",
" <td>en</td>\n",
" <td>278737933</td>\n",
" <td>NaN</td>\n",
" <td>&lt;EMOJI&gt; vivalcli: Portraits by Zhao Guojing an...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>[🚨, 🚨, 🚨, 🚨, 🚨, 🚨]</td>\n",
" <td>2017-11-01 13:29:05</td>\n",
" <td>925716325615439900</td>\n",
" <td>en</td>\n",
" <td>876587129077477400</td>\n",
" <td>NaN</td>\n",
" <td>@BillRatchet: &lt;EMOJI&gt;&lt;EMOJI&gt;&lt;EMOJI&gt;NO NUT NOV...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>[💕]</td>\n",
" <td>2017-11-01 13:29:05</td>\n",
" <td>925716325615505400</td>\n",
" <td>en</td>\n",
" <td>2402588155</td>\n",
" <td>NaN</td>\n",
" <td>@mysweetapplexxx: ❤Happy Birthday, @GoodGalC...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>[🙏]</td>\n",
" <td>2017-11-01 13:29:06</td>\n",
" <td>925716329797054500</td>\n",
" <td>en</td>\n",
" <td>1546678166</td>\n",
" <td>NaN</td>\n",
" <td>@pourlui: @realDonaldTrump @SSNjl @foxandfrie...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>[😩, 😩]</td>\n",
" <td>2017-11-01 13:29:06</td>\n",
" <td>925716329801310200</td>\n",
" <td>en</td>\n",
" <td>824586253634981900</td>\n",
" <td>9.257162e+17</td>\n",
" <td>@hiphopphiIes i wanna know too&lt;EMOJI&gt;&lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>[😭, 💓]</td>\n",
" <td>2017-11-01 13:29:02</td>\n",
" <td>925716313019965400</td>\n",
" <td>en</td>\n",
" <td>1581953814</td>\n",
" <td>9.257088e+17</td>\n",
" <td>@WizMommma veda was yoda too &lt;EMOJI&gt;&lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>[🙏, 🤕, 😂]</td>\n",
" <td>2017-11-01 13:29:06</td>\n",
" <td>925716329801371600</td>\n",
" <td>en</td>\n",
" <td>1737878390</td>\n",
" <td>NaN</td>\n",
" <td>@TiagoIlori4: Me and my family are humbled by...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>[😂]</td>\n",
" <td>2017-11-01 13:29:05</td>\n",
" <td>925716325607133200</td>\n",
" <td>en</td>\n",
" <td>1001999683</td>\n",
" <td>NaN</td>\n",
" <td>Im less stressed about turning 30 now &lt;EMOJI&gt;...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>[💯]</td>\n",
" <td>2017-11-01 13:29:07</td>\n",
" <td>925716334008082400</td>\n",
" <td>en</td>\n",
" <td>745222369183043600</td>\n",
" <td>NaN</td>\n",
" <td>Full charged. &lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>[🙄]</td>\n",
" <td>2017-11-01 13:29:09</td>\n",
" <td>925716342401052700</td>\n",
" <td>en</td>\n",
" <td>798557155217539100</td>\n",
" <td>9.257147e+17</td>\n",
" <td>@SeaDimon Thats part of the problem, (they) @...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>[😟, 😥, 😢]</td>\n",
" <td>2017-11-01 13:29:10</td>\n",
" <td>925716346570240000</td>\n",
" <td>en</td>\n",
" <td>744396039126421500</td>\n",
" <td>9.250629e+17</td>\n",
" <td>@Ian_khetye got me emotional there&lt;EMOJI&gt;&lt;EMOJ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>[😂, 😂, 😂]</td>\n",
" <td>2017-11-01 13:29:10</td>\n",
" <td>925716346603753500</td>\n",
" <td>en</td>\n",
" <td>2428265479</td>\n",
" <td>NaN</td>\n",
" <td>@30SECONDFlGHTS: BRUHHH &lt;EMOJI&gt;&lt;EMOJI&gt;&lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>[👀]</td>\n",
" <td>2017-11-01 13:29:10</td>\n",
" <td>925716346595434500</td>\n",
" <td>en</td>\n",
" <td>2523596372</td>\n",
" <td>NaN</td>\n",
" <td>@BasebaIINation: Game 7 who you got?&lt;EMOJI&gt; f...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>[🔎]</td>\n",
" <td>2017-11-01 13:29:11</td>\n",
" <td>925716350764331000</td>\n",
" <td>en</td>\n",
" <td>2315896393</td>\n",
" <td>NaN</td>\n",
" <td>@JudicialWatch: JW says DOJ &amp;amp; Congress ar...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>[📣, 💖]</td>\n",
" <td>2017-11-01 13:29:12</td>\n",
" <td>925716354988236800</td>\n",
" <td>en</td>\n",
" <td>2316464875</td>\n",
" <td>NaN</td>\n",
" <td>@cssconfau: ✍🏻 &lt;EMOJI&gt; Ever thought about spe...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>[✨, 💖]</td>\n",
" <td>2017-11-01 13:29:12</td>\n",
" <td>925716354975465500</td>\n",
" <td>en</td>\n",
" <td>2614916743</td>\n",
" <td>NaN</td>\n",
" <td>@btspat_: Deal for Young Forever: Night Vers...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>[💪]</td>\n",
" <td>2017-11-01 13:29:13</td>\n",
" <td>925716359153041400</td>\n",
" <td>en</td>\n",
" <td>2887539044</td>\n",
" <td>NaN</td>\n",
" <td>@DFC_SARAVANA: @dhanushkraja Reached 6 Millio...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>[🌻]</td>\n",
" <td>2017-11-01 13:29:13</td>\n",
" <td>925716359182520300</td>\n",
" <td>en</td>\n",
" <td>721490010118205400</td>\n",
" <td>8.965900e+17</td>\n",
" <td>back to the yellow &lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>[🍁, 🌺, 🍂]</td>\n",
" <td>2017-11-01 13:29:15</td>\n",
" <td>925716367558545400</td>\n",
" <td>en</td>\n",
" <td>520536723</td>\n",
" <td>9.257159e+17</td>\n",
" <td>@Dimafadma Happy month to you and your loved o...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>[🍃]</td>\n",
" <td>2017-11-01 13:29:16</td>\n",
" <td>925716371735900200</td>\n",
" <td>en</td>\n",
" <td>850852815941517300</td>\n",
" <td>NaN</td>\n",
" <td>Maxx and Patricia. Family hangs at For Good He...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176302</th>\n",
" <td>[😭, 😭, 💘]</td>\n",
" <td>2017-11-01 07:23:31</td>\n",
" <td>925624327755591700</td>\n",
" <td>en</td>\n",
" <td>845085544589672400</td>\n",
" <td>9.256009e+17</td>\n",
" <td>@peachshua1230 Awww &lt;EMOJI&gt; Ajsksjdjd im smili...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176303</th>\n",
" <td>[🤣, 🤣, 🤣, 🤣]</td>\n",
" <td>2017-11-01 07:23:33</td>\n",
" <td>925624336156938200</td>\n",
" <td>en</td>\n",
" <td>1154786048</td>\n",
" <td>NaN</td>\n",
" <td>@playboinathann: MY LIL BRO ATE AN EDIBLE THI...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176304</th>\n",
" <td>[😫, ✋]</td>\n",
" <td>2017-11-01 07:23:34</td>\n",
" <td>925624340342812700</td>\n",
" <td>en</td>\n",
" <td>924752524871131100</td>\n",
" <td>NaN</td>\n",
" <td>I hate when I send a text or snap n I'm so anx...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176305</th>\n",
" <td>[👻, 🎃]</td>\n",
" <td>2017-11-01 07:23:34</td>\n",
" <td>925624340334501900</td>\n",
" <td>en</td>\n",
" <td>868681065308631000</td>\n",
" <td>NaN</td>\n",
" <td>@LDNNOISE: Happy Halloween &lt;EMOJI&gt;&lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176306</th>\n",
" <td>[😂, 🙁]</td>\n",
" <td>2017-11-01 07:23:34</td>\n",
" <td>925624340355280900</td>\n",
" <td>en</td>\n",
" <td>419493819</td>\n",
" <td>NaN</td>\n",
" <td>The answer is no I have no plans and I never l...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176307</th>\n",
" <td>[😭]</td>\n",
" <td>2017-11-01 07:23:34</td>\n",
" <td>925624340346937300</td>\n",
" <td>en</td>\n",
" <td>2457745952</td>\n",
" <td>9.256219e+17</td>\n",
" <td>@BeachBoy_Gab LMAOOO I'm so proud &lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176308</th>\n",
" <td>[😢, 💔]</td>\n",
" <td>2017-11-01 07:23:34</td>\n",
" <td>925624340338507800</td>\n",
" <td>en</td>\n",
" <td>1955767531</td>\n",
" <td>NaN</td>\n",
" <td>My cousin/little sister is leaving to San Fran...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176309</th>\n",
" <td>[🙃]</td>\n",
" <td>2017-11-01 07:23:35</td>\n",
" <td>925624344524361700</td>\n",
" <td>en</td>\n",
" <td>796490344581898200</td>\n",
" <td>NaN</td>\n",
" <td>Can't be alone w my thoughts tonight so just g...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176310</th>\n",
" <td>[✨]</td>\n",
" <td>2017-11-01 07:23:35</td>\n",
" <td>925624344511897600</td>\n",
" <td>en</td>\n",
" <td>825346617620328400</td>\n",
" <td>NaN</td>\n",
" <td>@ohmygirlpcs: Jine &lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176311</th>\n",
" <td>[😂, 😂, 😂, 😂, 😂]</td>\n",
" <td>2017-11-01 07:23:36</td>\n",
" <td>925624348710285300</td>\n",
" <td>en</td>\n",
" <td>907808317124177900</td>\n",
" <td>NaN</td>\n",
" <td>&lt;EMOJI&gt;&lt;EMOJI&gt;&lt;EMOJI&gt;&lt;EMOJI&gt;&lt;EMOJI&gt; ambot!!!</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176312</th>\n",
" <td>[🚀]</td>\n",
" <td>2017-11-01 07:23:37</td>\n",
" <td>925624352921608200</td>\n",
" <td>en</td>\n",
" <td>1605371160</td>\n",
" <td>NaN</td>\n",
" <td>@WestHamUtd: It wasn't the result we wanted, ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176313</th>\n",
" <td>[👌, 👊, 🙌]</td>\n",
" <td>2017-11-01 07:23:37</td>\n",
" <td>925624352929910800</td>\n",
" <td>en</td>\n",
" <td>262162415</td>\n",
" <td>NaN</td>\n",
" <td>&lt;EMOJI&gt;🏽&lt;EMOJI&gt;🏽 1st of the month!!Happy 1st o...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176314</th>\n",
" <td>[😴]</td>\n",
" <td>2017-11-01 07:23:41</td>\n",
" <td>925624369715515400</td>\n",
" <td>en</td>\n",
" <td>574882525</td>\n",
" <td>NaN</td>\n",
" <td>#WednesdayWisdom ... stay in bed &lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176315</th>\n",
" <td>[💀, 🖤]</td>\n",
" <td>2017-11-01 07:23:42</td>\n",
" <td>925624373905707000</td>\n",
" <td>en</td>\n",
" <td>2694136560</td>\n",
" <td>NaN</td>\n",
" <td>@Brandonwoelfel: Happy Halloween friends!&lt;EMO...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176316</th>\n",
" <td>[🔴]</td>\n",
" <td>2017-11-01 07:23:43</td>\n",
" <td>925624378078851100</td>\n",
" <td>en</td>\n",
" <td>799982767886872600</td>\n",
" <td>NaN</td>\n",
" <td>@carras16: Good win tonight and great to see ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176317</th>\n",
" <td>[😂, 😂, 😂, 😂]</td>\n",
" <td>2017-11-01 07:23:44</td>\n",
" <td>925624382269124600</td>\n",
" <td>en</td>\n",
" <td>556175173</td>\n",
" <td>9.254038e+17</td>\n",
" <td>@Louis_Tomlinson @NiallOfficial THIS IS GOLD. ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176318</th>\n",
" <td>[💖, 💗, 💞, 💕, 💘]</td>\n",
" <td>2017-11-01 07:23:44</td>\n",
" <td>925624382281547800</td>\n",
" <td>en</td>\n",
" <td>837324291637051400</td>\n",
" <td>NaN</td>\n",
" <td>@95duet: i love them sm☹&lt;EMOJI&gt;&lt;EMOJI&gt;&lt;EMOJI...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176319</th>\n",
" <td>[😀]</td>\n",
" <td>2017-11-01 07:23:45</td>\n",
" <td>925624386455031800</td>\n",
" <td>en</td>\n",
" <td>1610265588</td>\n",
" <td>NaN</td>\n",
" <td>&lt;EMOJI&gt; thank you for the kind compliment</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176320</th>\n",
" <td>[😎]</td>\n",
" <td>2017-11-01 07:23:45</td>\n",
" <td>925624386454937600</td>\n",
" <td>en</td>\n",
" <td>4760724450</td>\n",
" <td>NaN</td>\n",
" <td>Enjoyed the silence &lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176321</th>\n",
" <td>[✨]</td>\n",
" <td>2017-11-01 07:23:46</td>\n",
" <td>925624390657572900</td>\n",
" <td>en</td>\n",
" <td>882858115636514800</td>\n",
" <td>NaN</td>\n",
" <td>OS: Spiderman Homecoming &lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176322</th>\n",
" <td>[🤷]</td>\n",
" <td>2017-11-01 07:23:46</td>\n",
" <td>925624390682849300</td>\n",
" <td>en</td>\n",
" <td>188129628</td>\n",
" <td>NaN</td>\n",
" <td>Go to hell &lt;EMOJI&gt;🏽‍♀️</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176323</th>\n",
" <td>[😘]</td>\n",
" <td>2017-11-01 07:23:46</td>\n",
" <td>925624390666129400</td>\n",
" <td>en</td>\n",
" <td>2473135939</td>\n",
" <td>NaN</td>\n",
" <td>Thank you Yomi! &lt;EMOJI&gt;</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176324</th>\n",
" <td>[😂]</td>\n",
" <td>2017-11-01 07:23:46</td>\n",
" <td>925624390670106600</td>\n",
" <td>en</td>\n",
" <td>2373584209</td>\n",
" <td>9.256241e+17</td>\n",
" <td>@discopiggu Lol. Just enjoy the stars. Music K...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176325</th>\n",
" <td>[🙏]</td>\n",
" <td>2017-11-01 07:23:47</td>\n",
" <td>925624394860453900</td>\n",
" <td>en</td>\n",
" <td>40709255</td>\n",
" <td>NaN</td>\n",
" <td>@INS_Ecosystem: We are honored to have @eyal ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176326</th>\n",
" <td>[🔥]</td>\n",
" <td>2017-11-01 07:23:48</td>\n",
" <td>925624399063117800</td>\n",
" <td>en</td>\n",
" <td>889503742474768400</td>\n",
" <td>NaN</td>\n",
" <td>@EmiNiPr: I just love twitterMan's not hot &lt;E...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176327</th>\n",
" <td>[🙏]</td>\n",
" <td>2017-11-01 07:23:50</td>\n",
" <td>925624407459971100</td>\n",
" <td>en</td>\n",
" <td>181689756</td>\n",
" <td>NaN</td>\n",
" <td>Thoughts and prayers for NY&lt;EMOJI&gt;🏻 #NYCStrong</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176328</th>\n",
" <td>[💁]</td>\n",
" <td>2017-11-01 07:23:50</td>\n",
" <td>925624407460057100</td>\n",
" <td>en</td>\n",
" <td>60160788</td>\n",
" <td>9.254610e+17</td>\n",
" <td>@BrianyH I searched COCK, PENIS, SHLONG, WINKY...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176329</th>\n",
" <td>[🌠, 🎇, 🌠, 🌙]</td>\n",
" <td>2017-11-01 07:23:51</td>\n",
" <td>925624411620646900</td>\n",
" <td>en</td>\n",
" <td>4915042321</td>\n",
" <td>NaN</td>\n",
" <td>@eyesnoselips99: 170930 KMF &lt;EMOJI&gt;☄&lt;EMOJI&gt; ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176330</th>\n",
" <td>[🍃, 🌻, 🌻, 🍃, 🍃, 💐, 💐, 🍃, 🙋]</td>\n",
" <td>2017-11-01 07:23:53</td>\n",
" <td>925624420022063100</td>\n",
" <td>en</td>\n",
" <td>3792290725</td>\n",
" <td>9.256215e+17</td>\n",
" <td>@amitbarman520 Thank you so much&lt;EMOJI&gt;&lt;EMOJI&gt;...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>176331</th>\n",
" <td>[😻, 😫, 🤔, 💦, 💦, 💦]</td>\n",
" <td>2017-11-01 07:23:57</td>\n",
" <td>925624436799049700</td>\n",
" <td>en</td>\n",
" <td>612733752</td>\n",
" <td>NaN</td>\n",
" <td>@DarrionDaDon_: Vocals ? &lt;EMOJI&gt;High Notes ? ...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>176332 rows × 7 columns</p>\n",
"</div>"
],
"text/plain": [
" EMOJI datetime id \\\n",
"0 [🔥, 👏] 2017-11-01 13:29:00 925716304635547600 \n",
"1 [🤦] 2017-11-01 13:29:00 925716304664911900 \n",
"2 [🤧] 2017-11-01 13:29:00 925716304652308500 \n",
"3 [🐯, 🐥, 💕] 2017-11-01 13:29:01 925716308846661600 \n",
"4 [👍, 👍, 👍] 2017-11-01 13:29:02 925716313015586800 \n",
"5 [😒] 2017-11-01 13:29:02 925716313032364000 \n",
"6 [✅, 🍟] 2017-11-01 13:29:02 925716313036722200 \n",
"7 [👶] 2017-11-01 13:29:03 925716317243555800 \n",
"8 [😄] 2017-11-01 13:29:03 925716317214089200 \n",
"9 [🌟, 🌟] 2017-11-01 13:29:03 925716317218508800 \n",
"10 [📷] 2017-11-01 13:29:04 925716321416949800 \n",
"11 [🚨, 🚨, 🚨, 🚨, 🚨, 🚨] 2017-11-01 13:29:05 925716325615439900 \n",
"12 [💕] 2017-11-01 13:29:05 925716325615505400 \n",
"13 [🙏] 2017-11-01 13:29:06 925716329797054500 \n",
"14 [😩, 😩] 2017-11-01 13:29:06 925716329801310200 \n",
"15 [😭, 💓] 2017-11-01 13:29:02 925716313019965400 \n",
"16 [🙏, 🤕, 😂] 2017-11-01 13:29:06 925716329801371600 \n",
"17 [😂] 2017-11-01 13:29:05 925716325607133200 \n",
"18 [💯] 2017-11-01 13:29:07 925716334008082400 \n",
"19 [🙄] 2017-11-01 13:29:09 925716342401052700 \n",
"20 [😟, 😥, 😢] 2017-11-01 13:29:10 925716346570240000 \n",
"21 [😂, 😂, 😂] 2017-11-01 13:29:10 925716346603753500 \n",
"22 [👀] 2017-11-01 13:29:10 925716346595434500 \n",
"23 [🔎] 2017-11-01 13:29:11 925716350764331000 \n",
"24 [📣, 💖] 2017-11-01 13:29:12 925716354988236800 \n",
"25 [✨, 💖] 2017-11-01 13:29:12 925716354975465500 \n",
"26 [💪] 2017-11-01 13:29:13 925716359153041400 \n",
"27 [🌻] 2017-11-01 13:29:13 925716359182520300 \n",
"28 [🍁, 🌺, 🍂] 2017-11-01 13:29:15 925716367558545400 \n",
"29 [🍃] 2017-11-01 13:29:16 925716371735900200 \n",
"... ... ... ... \n",
"176302 [😭, 😭, 💘] 2017-11-01 07:23:31 925624327755591700 \n",
"176303 [🤣, 🤣, 🤣, 🤣] 2017-11-01 07:23:33 925624336156938200 \n",
"176304 [😫, ✋] 2017-11-01 07:23:34 925624340342812700 \n",
"176305 [👻, 🎃] 2017-11-01 07:23:34 925624340334501900 \n",
"176306 [😂, 🙁] 2017-11-01 07:23:34 925624340355280900 \n",
"176307 [😭] 2017-11-01 07:23:34 925624340346937300 \n",
"176308 [😢, 💔] 2017-11-01 07:23:34 925624340338507800 \n",
"176309 [🙃] 2017-11-01 07:23:35 925624344524361700 \n",
"176310 [✨] 2017-11-01 07:23:35 925624344511897600 \n",
"176311 [😂, 😂, 😂, 😂, 😂] 2017-11-01 07:23:36 925624348710285300 \n",
"176312 [🚀] 2017-11-01 07:23:37 925624352921608200 \n",
"176313 [👌, 👊, 🙌] 2017-11-01 07:23:37 925624352929910800 \n",
"176314 [😴] 2017-11-01 07:23:41 925624369715515400 \n",
"176315 [💀, 🖤] 2017-11-01 07:23:42 925624373905707000 \n",
"176316 [🔴] 2017-11-01 07:23:43 925624378078851100 \n",
"176317 [😂, 😂, 😂, 😂] 2017-11-01 07:23:44 925624382269124600 \n",
"176318 [💖, 💗, 💞, 💕, 💘] 2017-11-01 07:23:44 925624382281547800 \n",
"176319 [😀] 2017-11-01 07:23:45 925624386455031800 \n",
"176320 [😎] 2017-11-01 07:23:45 925624386454937600 \n",
"176321 [✨] 2017-11-01 07:23:46 925624390657572900 \n",
"176322 [🤷] 2017-11-01 07:23:46 925624390682849300 \n",
"176323 [😘] 2017-11-01 07:23:46 925624390666129400 \n",
"176324 [😂] 2017-11-01 07:23:46 925624390670106600 \n",
"176325 [🙏] 2017-11-01 07:23:47 925624394860453900 \n",
"176326 [🔥] 2017-11-01 07:23:48 925624399063117800 \n",
"176327 [🙏] 2017-11-01 07:23:50 925624407459971100 \n",
"176328 [💁] 2017-11-01 07:23:50 925624407460057100 \n",
"176329 [🌠, 🎇, 🌠, 🌙] 2017-11-01 07:23:51 925624411620646900 \n",
"176330 [🍃, 🌻, 🌻, 🍃, 🍃, 💐, 💐, 🍃, 🙋] 2017-11-01 07:23:53 925624420022063100 \n",
"176331 [😻, 😫, 🤔, 💦, 💦, 💦] 2017-11-01 07:23:57 925624436799049700 \n",
"\n",
" lang person reply_to \\\n",
"0 en 31507978 NaN \n",
"1 en 231994649 NaN \n",
"2 en 78123421 NaN \n",
"3 en 860396470540705800 NaN \n",
"4 en 775381892321333200 NaN \n",
"5 en 3244140547 NaN \n",
"6 en 4891498023 NaN \n",
"7 en 2660072274 NaN \n",
"8 en 2592765104 NaN \n",
"9 en 3011085514 NaN \n",
"10 en 278737933 NaN \n",
"11 en 876587129077477400 NaN \n",
"12 en 2402588155 NaN \n",
"13 en 1546678166 NaN \n",
"14 en 824586253634981900 9.257162e+17 \n",
"15 en 1581953814 9.257088e+17 \n",
"16 en 1737878390 NaN \n",
"17 en 1001999683 NaN \n",
"18 en 745222369183043600 NaN \n",
"19 en 798557155217539100 9.257147e+17 \n",
"20 en 744396039126421500 9.250629e+17 \n",
"21 en 2428265479 NaN \n",
"22 en 2523596372 NaN \n",
"23 en 2315896393 NaN \n",
"24 en 2316464875 NaN \n",
"25 en 2614916743 NaN \n",
"26 en 2887539044 NaN \n",
"27 en 721490010118205400 8.965900e+17 \n",
"28 en 520536723 9.257159e+17 \n",
"29 en 850852815941517300 NaN \n",
"... ... ... ... \n",
"176302 en 845085544589672400 9.256009e+17 \n",
"176303 en 1154786048 NaN \n",
"176304 en 924752524871131100 NaN \n",
"176305 en 868681065308631000 NaN \n",
"176306 en 419493819 NaN \n",
"176307 en 2457745952 9.256219e+17 \n",
"176308 en 1955767531 NaN \n",
"176309 en 796490344581898200 NaN \n",
"176310 en 825346617620328400 NaN \n",
"176311 en 907808317124177900 NaN \n",
"176312 en 1605371160 NaN \n",
"176313 en 262162415 NaN \n",
"176314 en 574882525 NaN \n",
"176315 en 2694136560 NaN \n",
"176316 en 799982767886872600 NaN \n",
"176317 en 556175173 9.254038e+17 \n",
"176318 en 837324291637051400 NaN \n",
"176319 en 1610265588 NaN \n",
"176320 en 4760724450 NaN \n",
"176321 en 882858115636514800 NaN \n",
"176322 en 188129628 NaN \n",
"176323 en 2473135939 NaN \n",
"176324 en 2373584209 9.256241e+17 \n",
"176325 en 40709255 NaN \n",
"176326 en 889503742474768400 NaN \n",
"176327 en 181689756 NaN \n",
"176328 en 60160788 9.254610e+17 \n",
"176329 en 4915042321 NaN \n",
"176330 en 3792290725 9.256215e+17 \n",
"176331 en 612733752 NaN \n",
"\n",
" text \n",
"0 fashionbombdaily's photo <EMOJI><EMOJI>🏼 \n",
"1 Its scary how on point my horoscope be <EMOJI... \n",
"2 @mvyn_: Them boring ass licks, you supposed t... \n",
"3 @taehyungpic: #vmin s unit shoot <EMOJI><EM... \n",
"4 @risa_del: Can I just say how much Meng respe... \n",
"5 @Melodia_Muse: omg did y'all just cut off Red... \n",
"6 @Harrys1DEmpire: Follow everyone who LIKES th... \n",
"7 @taehyungpic: he is so smollll <EMOJI>©rocket... \n",
"8 Woooaaaahhh <EMOJI> \n",
"9 @mymermaidIagoon: <EMOJI> HURRY! For your cha... \n",
"10 <EMOJI> vivalcli: Portraits by Zhao Guojing an... \n",
"11 @BillRatchet: <EMOJI><EMOJI><EMOJI>NO NUT NOV... \n",
"12 @mysweetapplexxx: ❤Happy Birthday, @GoodGalC... \n",
"13 @pourlui: @realDonaldTrump @SSNjl @foxandfrie... \n",
"14 @hiphopphiIes i wanna know too<EMOJI><EMOJI> \n",
"15 @WizMommma veda was yoda too <EMOJI><EMOJI> \n",
"16 @TiagoIlori4: Me and my family are humbled by... \n",
"17 Im less stressed about turning 30 now <EMOJI>... \n",
"18 Full charged. <EMOJI> \n",
"19 @SeaDimon Thats part of the problem, (they) @... \n",
"20 @Ian_khetye got me emotional there<EMOJI><EMOJ... \n",
"21 @30SECONDFlGHTS: BRUHHH <EMOJI><EMOJI><EMOJI> \n",
"22 @BasebaIINation: Game 7 who you got?<EMOJI> f... \n",
"23 @JudicialWatch: JW says DOJ &amp; Congress ar... \n",
"24 @cssconfau: ✍🏻 <EMOJI> Ever thought about spe... \n",
"25 @btspat_: Deal for Young Forever: Night Vers... \n",
"26 @DFC_SARAVANA: @dhanushkraja Reached 6 Millio... \n",
"27 back to the yellow <EMOJI> \n",
"28 @Dimafadma Happy month to you and your loved o... \n",
"29 Maxx and Patricia. Family hangs at For Good He... \n",
"... ... \n",
"176302 @peachshua1230 Awww <EMOJI> Ajsksjdjd im smili... \n",
"176303 @playboinathann: MY LIL BRO ATE AN EDIBLE THI... \n",
"176304 I hate when I send a text or snap n I'm so anx... \n",
"176305 @LDNNOISE: Happy Halloween <EMOJI><EMOJI> \n",
"176306 The answer is no I have no plans and I never l... \n",
"176307 @BeachBoy_Gab LMAOOO I'm so proud <EMOJI> \n",
"176308 My cousin/little sister is leaving to San Fran... \n",
"176309 Can't be alone w my thoughts tonight so just g... \n",
"176310 @ohmygirlpcs: Jine <EMOJI> \n",
"176311 <EMOJI><EMOJI><EMOJI><EMOJI><EMOJI> ambot!!! \n",
"176312 @WestHamUtd: It wasn't the result we wanted, ... \n",
"176313 <EMOJI>🏽<EMOJI>🏽 1st of the month!!Happy 1st o... \n",
"176314 #WednesdayWisdom ... stay in bed <EMOJI> \n",
"176315 @Brandonwoelfel: Happy Halloween friends!<EMO... \n",
"176316 @carras16: Good win tonight and great to see ... \n",
"176317 @Louis_Tomlinson @NiallOfficial THIS IS GOLD. ... \n",
"176318 @95duet: i love them sm☹<EMOJI><EMOJI><EMOJI... \n",
"176319 <EMOJI> thank you for the kind compliment \n",
"176320 Enjoyed the silence <EMOJI> \n",
"176321 OS: Spiderman Homecoming <EMOJI> \n",
"176322 Go to hell <EMOJI>🏽‍♀️ \n",
"176323 Thank you Yomi! <EMOJI> \n",
"176324 @discopiggu Lol. Just enjoy the stars. Music K... \n",
"176325 @INS_Ecosystem: We are honored to have @eyal ... \n",
"176326 @EmiNiPr: I just love twitterMan's not hot <E... \n",
"176327 Thoughts and prayers for NY<EMOJI>🏻 #NYCStrong \n",
"176328 @BrianyH I searched COCK, PENIS, SHLONG, WINKY... \n",
"176329 @eyesnoselips99: 170930 KMF <EMOJI>☄<EMOJI> ... \n",
"176330 @amitbarman520 Thank you so much<EMOJI><EMOJI>... \n",
"176331 @DarrionDaDon_: Vocals ? <EMOJI>High Notes ? ... \n",
"\n",
"[176332 rows x 7 columns]"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"twitter_data = pd.read_json(json_files[0])\n",
"twitter_data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* extracting emojis and text"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"emojis = twitter_data['EMOJI']\n",
"plain_text = twitter_data['text']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* make our plain text more \"plain\":\n",
" * removing the keyword `<EMOJI>` (just for the beginning)\n",
" * removing remaining useless emojis, like skin modifier etc."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"emoji_blacklist = set([\n",
" chr(0x1F3FB),\n",
" chr(0x1F3FC),\n",
" chr(0x1F3FD),\n",
" chr(0x1F3FE),\n",
" chr(0x1F3FF),\n",
" chr(0x2642),\n",
" chr(0x2640)\n",
"])"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}