Term_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Term Assignment - Introduction to Computational Social Science\n",
    "=============================\n",
    "\n",
    "--------\n",
    "\n",
    "\n",
    "**Student** : Nicolas Michel 48-179727\n",
    "\n",
    "\n",
    "--------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This Jupyter Notebook contains my term assignement. I decided to work on the 3rd subject given in the notice.\n",
    "\n",
    "I chose to compare two categories : **science** vs **health**\n",
    "\n",
    "Indeed in this notebook, I first operate webscraping over several websites to collect my dataset of articles.\n",
    "\n",
    "Then, I work on the parsing of the obtained text to get usable data as 'Bag of Words' for my algorithms.\n",
    "\n",
    "Finally I compare several algorithms on classification task over my dataset.\n",
    "\n",
    "It is important to note that for this exercice I might have misunderstood the subject. My dataset is composed of bag of words from the *entire* text body of each articles (not the summary). I hope this is not a problem.\n",
    "\n",
    "\n",
    "----------\n",
    "\n",
    "\n",
    "Scraping\n",
    "====\n",
    "\n",
    "------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# urls for news feed scraping\n",
    "rss_urls = {\n",
    "    'science': {\n",
    "        'bbc':\"http://feeds.bbci.co.uk/news/science_and_environment/rss.xml\",\n",
    "        'reuters':\"http://feeds.reuters.com/reuters/scienceNews\",\n",
    "        'cnn':\"http://rss.cnn.com/rss/edition_space.rss\",\n",
    "        'guardian':\"https://www.theguardian.com/science/rss\",\n",
    "        'dm':\"http://www.dailymail.co.uk/sciencetech/index.rss\",\n",
    "        'toi':\"https://timesofindia.indiatimes.com/rssfeeds/-2128672765.cms\"\n",
    "        },\n",
    "    'health': {\n",
    "        'bbc':\"http://feeds.bbci.co.uk/news/health/rss.xml\",\n",
    "        'reuters':\"http://feeds.reuters.com/reuters/healthNews\",\n",
    "        'guardian':\"https://www.theguardian.com/society/health/rss\",\n",
    "        'dm':\"http://www.dailymail.co.uk/health/index.rss\",\n",
    "        'toi':\"https://timesofindia.indiatimes.com/rssfeeds/3908999.cms\",\n",
    "        'h1': \"http://www.health.com/news/feed\",\n",
    "        'h2': \"http://www.health.com/nutrition/feed\",\n",
    "        'h3': \"http://www.health.com/food/feed\",\n",
    "        'h4': \"http://www.health.com/healthday/feed\",\n",
    "        'h5': \"http://www.health.com/mind-body/feed\",\n",
    "        'h6': \"http://www.health.com/weight-loss/feed\",\n",
    "        'h7': \"http://www.health.com/style/feed\"\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we check that the number of collected articles is big enough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number articles for category science is 420\n",
      "Number articles for category health is 358\n"
     ]
    }
   ],
   "source": [
    "import feedparser\n",
    "\n",
    "for cat in rss_urls.keys():\n",
    "    s = 0\n",
    "    for key in rss_urls[cat].keys():\n",
    "        url = rss_urls[cat][key]\n",
    "        parsed = feedparser.parse(url)\n",
    "        entries = parsed.entries\n",
    "        s += len(entries)\n",
    "    print('Number articles for category {} is {}'.format(cat,s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following two functions are used to get the main text body and to make it a bag of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "\n",
    "def get_main_body_text(entry, news_website):\n",
    "    link = entry.link\n",
    "    res = requests.get(link)\n",
    "    soup = BeautifulSoup(res.text, \"lxml\")\n",
    "    \n",
    "    body_text = \"\"\n",
    "    if (news_website == 'bbc'):\n",
    "        tags = soup.find('body').find_all('p', {'class':'', 'style':''})\n",
    "        for i in range(len(tags)-1):\n",
    "            tag_text = tags[i].get_text()\n",
    "            # Some articles have twitter suggestion at the end of main body\n",
    "            if not (i==len(tags)-2 and ('Twitter' in tag_text)):\n",
    "                body_text += \" \"+tag_text\n",
    "    \n",
    "    elif (news_website == 'reuters'):\n",
    "        tags = soup.find('body').find_all('p', {'class':'', 'style':'', 'id':''})\n",
    "        for i in range(len(tags)-2):\n",
    "             body_text += \" \"+tags[i].get_text()\n",
    "    \n",
    "    elif (news_website == 'cnn'):\n",
    "        tags = soup.find('body').find_all('p', {'class':'', 'style':'', 'id':''})\n",
    "        for i in range(len(tags)):\n",
    "            body_text += \" \"+tags[i].get_text()\n",
    "    \n",
    "    elif (news_website =='dm'):\n",
    "        tags0 = soup.find_all('div', {'itemprop':'articleBody'})\n",
    "        for tag0 in tags0:\n",
    "            tags1 = tag0.find_all('p')\n",
    "            for tag1 in tags1:\n",
    "                body_text += \" \"+tag1.get_text()\n",
    "    \n",
    "    elif(news_website == 'toi'):\n",
    "        tags = soup.find('body').find_all('div', {'class':'Normal', 'style':''})\n",
    "        for i in range(len(tags)):\n",
    "            body_text += \" \"+tags[i].get_text()\n",
    "    \n",
    "    else:\n",
    "        tags = soup.find('body').find_all('p', {'class':'', 'style':''})\n",
    "        for i in range(len(tags)):\n",
    "            body_text += \" \"+tags[i].get_text()\n",
    "\n",
    "    return body_text\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*text_to_bag()* transforms a raw string text into a bag of words.\n",
    "\n",
    "Package unidecode is needed to remove accents, just in case.\n",
    "\n",
    "Run\n",
    "\n",
    "    conda install unidecode\n",
    "    \n",
    "to install the package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import unidecode\n",
    "\n",
    "def text_to_bag(body_text, news_website):\n",
    "    if (news_website == 'reuters'):\n",
    "        body_text = body_text.split('Reuters')[1]\n",
    "    elif (news_website == 'cnn'):\n",
    "        split = body_text.split('CNN')\n",
    "        if not (len(split) == 1):\n",
    "            body_text = body_text.split('CNN')[1]\n",
    "    elif(news_website == 'h1' or 'h2' or 'h3' or 'h4' or 'h5' or 'h6' or 'h7'):\n",
    "        split = body_text.split('HealthDay News')\n",
    "        if not (len(split) == 1):   \n",
    "            body_text = split[1]\n",
    "        \n",
    "    body_text = body_text.replace(',', '')\\\n",
    "    .replace('.', '')\\\n",
    "    .replace('?', '')\\\n",
    "    .replace('!', '')\\\n",
    "    .replace('&', '')\\\n",
    "    .replace('$', '')\\\n",
    "    .replace(':', '')\\\n",
    "    .replace(';', '')\\\n",
    "    .replace('\\'s', '')\\\n",
    "    .replace('\"', '')\\\n",
    "    .replace(')', '')\\\n",
    "    .replace('(', '')\\\n",
    "    .replace('[', '')\\\n",
    "    .replace(']', '')\\\n",
    "    .replace('{', '')\\\n",
    "    .replace('}', '')\\\n",
    "    .replace('%', '')\\\n",
    "    .replace('/', '')\n",
    "    body_text = unidecode.unidecode(body_text) # remove accents\n",
    "    body_text = body_text.replace('\\n', '')\\\n",
    "    .replace('\\\\', '')\\\n",
    "    .replace('-', '')\\\n",
    "    .replace(\"'\", \"\")\\\n",
    "    .strip()\\\n",
    "    .lower()\n",
    "\n",
    "    bag = body_text.split(' ')\n",
    "\n",
    "    for word in bag:\n",
    "        word = word.strip()\n",
    "\n",
    "    while('' in bag):\n",
    "        bag.remove('')\n",
    "    \n",
    "    if (news_website == 'toi' and len(bag) is not 0):\n",
    "        #specific case for new york and new delhi\n",
    "        if (bag[0] == 'new'):\n",
    "            bag.pop(0)\n",
    "            bag.pop(0)\n",
    "        else:\n",
    "            bag.pop(0)\n",
    "            \n",
    "    return bag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stack each elements of the dataset in a dataframe.\n",
    "The code below takes at least **15 min** to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Website</th>\n",
       "      <th>Article_url</th>\n",
       "      <th>Bag</th>\n",
       "      <th>Category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bbc</td>\n",
       "      <td>http://www.bbc.co.uk/news/science-environment-...</td>\n",
       "      <td>[more, than, 11, billion, items, of, plastic, ...</td>\n",
       "      <td>science</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>bbc</td>\n",
       "      <td>http://www.bbc.co.uk/news/science-environment-...</td>\n",
       "      <td>[telemetry, from, the, vehicle, was, lost, abo...</td>\n",
       "      <td>science</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bbc</td>\n",
       "      <td>http://www.bbc.co.uk/news/science-environment-...</td>\n",
       "      <td>[new, dating, of, fossils, from, israel, indic...</td>\n",
       "      <td>science</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>bbc</td>\n",
       "      <td>http://www.bbc.co.uk/news/science-environment-...</td>\n",
       "      <td>[of, the, 18, resident, species, most, are, gr...</td>\n",
       "      <td>science</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>bbc</td>\n",
       "      <td>http://www.bbc.co.uk/news/health-42809445</td>\n",
       "      <td>[identical, longtailed, macaques, zhong, zhong...</td>\n",
       "      <td>science</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>bbc</td>\n",
       "      <td>http://www.bbc.co.uk/news/science-environment-...</td>\n",
       "      <td>[that, the, finding, of, a, study, that, track...</td>\n",
       "      <td>science</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>bbc</td>\n",
       "      <td>http://www.bbc.co.uk/news/uk-wales-south-east-...</td>\n",
       "      <td>[mathematicians, think, they, have, devised, a...</td>\n",
       "      <td>science</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bbc</td>\n",
       "      <td>http://www.bbc.co.uk/news/science-environment-...</td>\n",
       "      <td>[in, what, is, known, as, a, static, firing, a...</td>\n",
       "      <td>science</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bbc</td>\n",
       "      <td>http://www.bbc.co.uk/news/science-environment-...</td>\n",
       "      <td>[akin, to, a, giant, disco, ball, the, object,...</td>\n",
       "      <td>science</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>bbc</td>\n",
       "      <td>http://www.bbc.co.uk/news/science-environment-...</td>\n",
       "      <td>[the, seabed, investigation, coordinated, by, ...</td>\n",
       "      <td>science</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Website                                        Article_url  \\\n",
       "0     bbc  http://www.bbc.co.uk/news/science-environment-...   \n",
       "1     bbc  http://www.bbc.co.uk/news/science-environment-...   \n",
       "2     bbc  http://www.bbc.co.uk/news/science-environment-...   \n",
       "3     bbc  http://www.bbc.co.uk/news/science-environment-...   \n",
       "4     bbc          http://www.bbc.co.uk/news/health-42809445   \n",
       "5     bbc  http://www.bbc.co.uk/news/science-environment-...   \n",
       "6     bbc  http://www.bbc.co.uk/news/uk-wales-south-east-...   \n",
       "7     bbc  http://www.bbc.co.uk/news/science-environment-...   \n",
       "8     bbc  http://www.bbc.co.uk/news/science-environment-...   \n",
       "9     bbc  http://www.bbc.co.uk/news/science-environment-...   \n",
       "\n",
       "                                                 Bag Category  \n",
       "0  [more, than, 11, billion, items, of, plastic, ...  science  \n",
       "1  [telemetry, from, the, vehicle, was, lost, abo...  science  \n",
       "2  [new, dating, of, fossils, from, israel, indic...  science  \n",
       "3  [of, the, 18, resident, species, most, are, gr...  science  \n",
       "4  [identical, longtailed, macaques, zhong, zhong...  science  \n",
       "5  [that, the, finding, of, a, study, that, track...  science  \n",
       "6  [mathematicians, think, they, have, devised, a...  science  \n",
       "7  [in, what, is, known, as, a, static, firing, a...  science  \n",
       "8  [akin, to, a, giant, disco, ball, the, object,...  science  \n",
       "9  [the, seabed, investigation, coordinated, by, ...  science  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import time\n",
    "import feedparser\n",
    "\n",
    "data = []\n",
    "for cat in rss_urls.keys():\n",
    "    for site in rss_urls[cat].keys():\n",
    "        rss = rss_urls[cat][site]\n",
    "        parsed = feedparser.parse(rss)\n",
    "        entries = parsed.entries\n",
    "\n",
    "        for entry in entries:\n",
    "            body_text = get_main_body_text(entry=entry, news_website=site)\n",
    "            bag = text_to_bag(body_text=body_text, news_website=site)\n",
    "            row = [site, entry.link, bag, cat]\n",
    "            data.append(row)\n",
    "            time.sleep(1)\n",
    "\n",
    "feeds_df = pd.DataFrame(data, columns=['Website', 'Article_url', 'Bag', 'Category'])\n",
    "feeds_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of the bag\n",
      "476\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['more',\n",
       " 'than',\n",
       " '11',\n",
       " 'billion',\n",
       " 'items',\n",
       " 'of',\n",
       " 'plastic',\n",
       " 'were',\n",
       " 'found',\n",
       " 'on',\n",
       " 'a',\n",
       " 'third',\n",
       " 'of',\n",
       " 'coral',\n",
       " 'reefs',\n",
       " 'surveyed',\n",
       " 'in',\n",
       " 'the',\n",
       " 'asiapacific',\n",
       " 'region']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A bag of words\n",
    "print(\"Size of the bag\")\n",
    "print(len(feeds_df['Bag'].iloc[0]))\n",
    "feeds_df['Bag'].iloc[0][:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Making a copy because collecting all the data is time consuming\n",
    "feeds_copy = feeds_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---------------\n",
    "\n",
    "Cleaning\n",
    "====\n",
    "\n",
    "-------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "feeds_df = feeds_copy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We remove all bag-of-words that are not big enough"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def remove_bag_under(size_min, feeds_df=feeds_df, category=True):\n",
    "    to_drop = []\n",
    "    for i in range(len(feeds_df)): \n",
    "        if (len(feeds_df.iloc[i]['Bag']) < size_min):\n",
    "            to_drop.append(i)\n",
    "    feeds_df = feeds_df.drop(feeds_df.index[to_drop])\n",
    "    if (category):\n",
    "        print('Number of science articles')\n",
    "        print(feeds_df[feeds_df['Category'] == 'science'].shape[0])\n",
    "        print('Number of health articles')\n",
    "        print(feeds_df[feeds_df['Category'] == 'health'].shape[0])\n",
    "    return feeds_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of science articles\n",
      "395\n",
      "Number of health articles\n",
      "332\n"
     ]
    }
   ],
   "source": [
    "feeds_df = remove_bag_under(200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "import nltk.stem\n",
    "\n",
    "def preprocess(bag):\n",
    "    stops = stopwords.words('english')\n",
    "    stemmer = nltk.stem.porter.PorterStemmer()\n",
    "    processed = [stemmer.stem(token) for token in bag if token not in stops]\n",
    "    return processed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    [11, billion, item, plastic, found, third, cor...\n",
       "1    [telemetri, vehicl, lost, nine, minut, flight,...\n",
       "2    [new, date, fossil, israel, indic, speci, homo...\n",
       "3    [18, resid, speci, grow, number, stabl, evid, ...\n",
       "4    [ident, longtail, macaqu, zhong, zhong, hua, h...\n",
       "5    [find, studi, track, danc, death, fastest, lan...\n",
       "6    [mathematician, think, devis, way, calcul, siz...\n",
       "7    [known, static, fire, 27, engin, launcher, fir...\n",
       "8    [akin, giant, disco, ball, object, visibl, nak...\n",
       "9    [seab, investig, coordin, campaign, group, gre...\n",
       "Name: Bag, dtype: object"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feeds_df['Bag'] = feeds_df['Bag'].apply(preprocess)\n",
    "feeds_df['Bag'].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---------------------\n",
    "\n",
    "\n",
    "Learning\n",
    "=====\n",
    "\n",
    "-------------------------\n",
    "\n",
    "\n",
    "### 1. Creating the dataset\n",
    "--------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Website                                        Article_url  \\\n",
      "0     bbc  http://www.bbc.co.uk/news/science-environment-...   \n",
      "1     bbc  http://www.bbc.co.uk/news/science-environment-...   \n",
      "2     bbc  http://www.bbc.co.uk/news/science-environment-...   \n",
      "3     bbc  http://www.bbc.co.uk/news/science-environment-...   \n",
      "4     bbc          http://www.bbc.co.uk/news/health-42809445   \n",
      "\n",
      "                                                 Bag Category  \n",
      "0  [11, billion, item, plastic, found, third, cor...        0  \n",
      "1  [telemetri, vehicl, lost, nine, minut, flight,...        0  \n",
      "2  [new, date, fossil, israel, indic, speci, homo...        0  \n",
      "3  [18, resid, speci, grow, number, stabl, evid, ...        0  \n",
      "4  [ident, longtail, macaqu, zhong, zhong, hua, h...        0  \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Website</th>\n",
       "      <th>Article_url</th>\n",
       "      <th>Bag</th>\n",
       "      <th>Category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>773</th>\n",
       "      <td>h7</td>\n",
       "      <td>http://www.health.com/syndication/madewell-bes...</td>\n",
       "      <td>[articl, origin, appear, peoplecom, holiday, d...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>774</th>\n",
       "      <td>h7</td>\n",
       "      <td>http://www.health.com/syndication/amazon-12-da...</td>\n",
       "      <td>[articl, origin, appear, travelandleisurecom, ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>775</th>\n",
       "      <td>h7</td>\n",
       "      <td>http://www.health.com/style/best-workout-under...</td>\n",
       "      <td>[nobodi, like, midsquat, wedgi, panti, line, p...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>776</th>\n",
       "      <td>h7</td>\n",
       "      <td>http://www.health.com/style/lindsey-vonn-under...</td>\n",
       "      <td>[come, girl, boss, alpin, skier, lindsey, vonn...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>777</th>\n",
       "      <td>h7</td>\n",
       "      <td>http://www.health.com/style/plus-size-sports-i...</td>\n",
       "      <td>[ever, sinc, hit, puberti, bath, suit, shop, d...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Website                                        Article_url  \\\n",
       "773      h7  http://www.health.com/syndication/madewell-bes...   \n",
       "774      h7  http://www.health.com/syndication/amazon-12-da...   \n",
       "775      h7  http://www.health.com/style/best-workout-under...   \n",
       "776      h7  http://www.health.com/style/lindsey-vonn-under...   \n",
       "777      h7  http://www.health.com/style/plus-size-sports-i...   \n",
       "\n",
       "                                                   Bag Category  \n",
       "773  [articl, origin, appear, peoplecom, holiday, d...        1  \n",
       "774  [articl, origin, appear, travelandleisurecom, ...        1  \n",
       "775  [nobodi, like, midsquat, wedgi, panti, line, p...        1  \n",
       "776  [come, girl, boss, alpin, skier, lindsey, vonn...        1  \n",
       "777  [ever, sinc, hit, puberti, bath, suit, shop, d...        1  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here the labels are science and health\n",
    "# science is category 0 and health category 1\n",
    "labels = ['science', 'health']\n",
    "\n",
    "feeds_df['Category'] = feeds_df['Category'].where(feeds_df['Category'] == 'science', other=1)\n",
    "feeds_df['Category'] = feeds_df['Category'].where(feeds_df['Category'] == 1, other=0)\n",
    "print(feeds_df.head(5))\n",
    "feeds_df.tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this exercice I decided to take the same number of words for every article as input of my algorithm to not give too much credits to longer articles, so I take the maximum number of word as possible which is the size of the smallest bag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "107"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_size = 500\n",
    "for i in range(len(feeds_df['Bag'])):\n",
    "    if(len(feeds_df['Bag'].iloc[i]) <= input_size):\n",
    "        input_size = len(feeds_df['Bag'].iloc[i])\n",
    "input_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function is used to take a random crop of a bag with the necessary number of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def get_random_words(bag, nb_words=input_size, random_seed=2):\n",
    "    indices = list(range(len(bag)))\n",
    "    np.random.seed(random_seed)\n",
    "    np.random.shuffle(indices)\n",
    "    bag_crop = []\n",
    "    for i in range(input_size):\n",
    "        bag_crop.append(bag[indices[i]])\n",
    "    return bag_crop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I use join_bag() later to transform my bag of word into an input vector. The package used must be applied to a bag of word in the form of one string, each word separated by a space ' '."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def join_bag(bag):\n",
    "    return ' '.join(bag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "feeds_df['Bag'] = feeds_df['Bag'].apply(get_random_words).apply(join_bag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of the entire dataset: 727\n",
      "Size of the training set: 545\n",
      "Size of the test set: 182\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "dataset = feeds_df[['Bag', 'Category']]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    dataset['Bag'].values, dataset['Category'].values, stratify=dataset['Category'].values, random_state=2)\n",
    "\n",
    "# Conversion to float type\n",
    "y_train = y_train.astype(float)\n",
    "y_test = y_test.astype(float)\n",
    "\n",
    "print('Size of the entire dataset: %i' % len(dataset['Bag']))\n",
    "print('Size of the training set: %i' % len(X_train))\n",
    "print('Size of the test set: %i' % len(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As briefly explained before, the transform used below must be applied to a numpy array of bag of word, each bag in the form of a string (and not in the form of a list of string).\n",
    "\n",
    "The use of the package HashingVectorizer create a float vector of size 4096 which is unique for each bag-of-words. I need to use this package to apply some algorithms on my data. We can see here that most of these values are 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(545, 2048)\n",
      "(182, 2048)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[ 0.        ,  0.        ,  0.        , ...,  0.        ,\n",
       "        -0.0836242 ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        , ..., -0.08421519,\n",
       "         0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        , ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       ..., \n",
       "       [ 0.        ,  0.        ,  0.        , ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        , ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        , ...,  0.        ,\n",
       "         0.        ,  0.        ]])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "vectorizer = HashingVectorizer(n_features=2 ** 11)\n",
    "\n",
    "X_train = vectorizer.transform(X_train)\n",
    "X_test = vectorizer.transform(X_test)\n",
    "\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "X_train.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Algorithms comparison\n",
    "-------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section I apply 4 basical algorithms to my dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            presort=False, random_state=241, splitter='best')"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "svc = SVC()\n",
    "forest = RandomForestClassifier(n_estimators=100, n_jobs=-1, max_depth=4)\n",
    "logreg = LogisticRegression()\n",
    "tree = DecisionTreeClassifier(random_state=241)\n",
    "\n",
    "svc.fit(X_train, y_train)\n",
    "forest.fit(X_train, y_train)\n",
    "logreg.fit(X_train, y_train)\n",
    "tree.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear Regression accuracy : 0.81868\n",
      "Support Vector Machine accuracy : 0.54396\n",
      "Decision Tree accurcacy : 0.67582\n",
      "Random Forest accuracy : 0.73626\n"
     ]
    }
   ],
   "source": [
    "# Prediction and evaluation of its accuracy\n",
    "pred_svc = svc.predict(X_test)\n",
    "pred_forest = forest.predict(X_test)\n",
    "pred_logreg = logreg.predict(X_test)\n",
    "pred_tree = tree.predict(X_test)\n",
    "\n",
    "accuracy_svc = (pred_svc == y_test).sum() / float(len(y_test))\n",
    "accuracy_forest = (pred_forest == y_test).sum() / float(len(y_test))\n",
    "accuracy_logreg = (pred_logreg == y_test).sum() / float(len(y_test))\n",
    "accuracy_tree = (pred_tree == y_test).sum() / float(len(y_test))\n",
    "\n",
    "print('Linear Regression accuracy : {}'.format(round(accuracy_logreg, 5)))\n",
    "print('Support Vector Machine accuracy : {}'.format(round(accuracy_svc, 5)))\n",
    "print('Decision Tree accurcacy : {}'.format(round(accuracy_tree, 5)))\n",
    "print('Random Forest accuracy : {}'.format(round(accuracy_forest, 5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see most algorithm is found out to be pretty accurate, above 67% accuracy except the Support Vector Machine algorithm which has around 50% accuracy.\n",
    "\n",
    "I realized that having the same number of words as input for each bag was not necessary using HasingVectorizer, so I tried without using get_random_words(), and the result is mostly better that way:\n",
    "\n",
    "    Linear Regression accuracy : 0.86624\n",
    "    Support Vector Machine accuracy : 0.50318\n",
    "    Decision Tree accurcacy : 0.76433\n",
    "    Random Forest accuracy : 0.86624\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Further experiments\n",
    "---------\n",
    "\n",
    "Just for fun, I tried using the Linear Regression Algorithm on 70 articles from the daily mail, category 'Money'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of articles : 70\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Website</th>\n",
       "      <th>Article_url</th>\n",
       "      <th>Bag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>dm</td>\n",
       "      <td>http://www.dailymail.co.uk/money/markets/artic...</td>\n",
       "      <td>[budget, hotel, business, easyhotel, is, boost...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>dm</td>\n",
       "      <td>http://www.dailymail.co.uk/money/diyinvesting/...</td>\n",
       "      <td>[the, recent, peril, of, carillion, is, a, tim...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>dm</td>\n",
       "      <td>http://www.dailymail.co.uk/news/article-531569...</td>\n",
       "      <td>[britain, economy, grew, by, 05, per, cent, in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dm</td>\n",
       "      <td>http://www.dailymail.co.uk/money/markets/artic...</td>\n",
       "      <td>[host, commentator, host, commentator, tequila...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>dm</td>\n",
       "      <td>http://www.dailymail.co.uk/money/cars/article-...</td>\n",
       "      <td>[want, to, be, the, coolest, people, on, the, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>dm</td>\n",
       "      <td>http://www.dailymail.co.uk/money/markets/artic...</td>\n",
       "      <td>[telecoms, giant, bt, has, been, slapped, with...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>dm</td>\n",
       "      <td>http://www.dailymail.co.uk/money/saving/articl...</td>\n",
       "      <td>[natwest, coop, and, barclays, had, the, bigge...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>dm</td>\n",
       "      <td>http://www.dailymail.co.uk/money/bills/article...</td>\n",
       "      <td>[andrea, caldwell, travel, insurance, claim, w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>dm</td>\n",
       "      <td>http://www.dailymail.co.uk/money/investing/art...</td>\n",
       "      <td>[recently, this, is, money, received, an, emai...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>dm</td>\n",
       "      <td>http://www.dailymail.co.uk/money/news/article-...</td>\n",
       "      <td>[the, us, has, locked, horns, with, world, ban...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Website                                        Article_url  \\\n",
       "0      dm  http://www.dailymail.co.uk/money/markets/artic...   \n",
       "1      dm  http://www.dailymail.co.uk/money/diyinvesting/...   \n",
       "2      dm  http://www.dailymail.co.uk/news/article-531569...   \n",
       "3      dm  http://www.dailymail.co.uk/money/markets/artic...   \n",
       "4      dm  http://www.dailymail.co.uk/money/cars/article-...   \n",
       "5      dm  http://www.dailymail.co.uk/money/markets/artic...   \n",
       "6      dm  http://www.dailymail.co.uk/money/saving/articl...   \n",
       "7      dm  http://www.dailymail.co.uk/money/bills/article...   \n",
       "8      dm  http://www.dailymail.co.uk/money/investing/art...   \n",
       "9      dm  http://www.dailymail.co.uk/money/news/article-...   \n",
       "\n",
       "                                                 Bag  \n",
       "0  [budget, hotel, business, easyhotel, is, boost...  \n",
       "1  [the, recent, peril, of, carillion, is, a, tim...  \n",
       "2  [britain, economy, grew, by, 05, per, cent, in...  \n",
       "3  [host, commentator, host, commentator, tequila...  \n",
       "4  [want, to, be, the, coolest, people, on, the, ...  \n",
       "5  [telecoms, giant, bt, has, been, slapped, with...  \n",
       "6  [natwest, coop, and, barclays, had, the, bigge...  \n",
       "7  [andrea, caldwell, travel, insurance, claim, w...  \n",
       "8  [recently, this, is, money, received, an, emai...  \n",
       "9  [the, us, has, locked, horns, with, world, ban...  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = 'http://www.dailymail.co.uk/money/index.rss'\n",
    "site = 'dm'\n",
    "parsed = feedparser.parse(url)\n",
    "entries = parsed.entries\n",
    "\n",
    "print(\"Number of articles : {}\".format(len(entries)))\n",
    "\n",
    "data = []\n",
    "for entry in entries:\n",
    "            body_text = get_main_body_text(entry=entry, news_website=site)\n",
    "            bag = text_to_bag(body_text=body_text, news_website=site)\n",
    "            row = [site, entry.link, bag]\n",
    "            data.append(row)\n",
    "            time.sleep(1)\n",
    "\n",
    "money_df = pd.DataFrame(data, columns=['Website', 'Article_url', 'Bag'])\n",
    "money_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Data processing\n",
    "money_df['Bag'] = money_df['Bag'].apply(preprocess).apply(join_bag)\n",
    "\n",
    "# bag of word conversion to numbers\n",
    "X_test = vectorizer.transform(money_df['Bag'].values)\n",
    "\n",
    "# Prediction\n",
    "predictions = logreg.predict(X_test)\n",
    "predictions[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8\n"
     ]
    }
   ],
   "source": [
    "pred_df = pd.DataFrame(predictions)\n",
    "nb_science = pred_df[pred_df.iloc[:,0] == 0].shape[0]\n",
    "nb_total = pred_df.shape[0]\n",
    "\n",
    "print(nb_science/nb_total)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apparently our algrorithm thinks that Money category is closer to Science category than Health category, which seems to be a understandable opinion.\n",
    "\n",
    "Let's take a look at some articles that have been categorized as beeing healh-related articles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      0\n",
       "7   1.0\n",
       "20  1.0\n",
       "30  1.0\n",
       "32  1.0\n",
       "33  1.0"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_df[pred_df.iloc[:,0] == 1].head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://www.dailymail.co.uk/money/bills/article-5085511/Why-did-travel-insurer-reject-claim.html?ITO=1490&ns_mchannel=rss&ns_campaign=1490'"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "money_df.iloc[7]['Article_url']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I chose this one because it was in my opinion the more relevent. Traveling and insurance are common subjects in health category which can explain why the algorithm made this decision.\n",
    "\n",
    "Conclusion\n",
    "-----\n",
    "\n",
    "In this work I was able to implement a classifier that can correctly make the distinction between a scientific news article and a health news article, in 85% of the cases.\n",
    "\n",
    "To some extend, this algorithm seems to be able to find some categories more scientific or healthy-related than others.\n",
    "\n",
    "I also wanted to apply this algorithm to another source of feed : Reddit feeds. Unfortunately most of the summary given by those feed was empty so I could not have any interesting results"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}