moocs/fast.ai/deep-learning-2018/lesson11.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 11"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 00:00:07 - Blog recap\n",
    "\n",
    "* Blog on super convergence: [The 1cycle policy](https://sgugger.github.io/the-1cycle-policy.html)\n",
    "  * 5x faster than stepwise approaches.\n",
    "  * Let's you have massively high learning rates (somewhere between 1 and 3).\n",
    "  * Trains at high learning rates for lots of the epochs: loss doesn't improve much but it's doing a lot of searching to find generalisable areas.\n",
    "  * When learning rate is high, momentum is lower.\n",
    "* Hamel Husain's blog on [sequence-to-sequence data products](https://towardsdatascience.com/how-to-create-data-products-that-are-magical-using-sequence-to-sequence-models-703f86a231f8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 00:05:42 - Building a sequence-to-sequence model using machine translation\n",
    "\n",
    "* Neural translation has surpased typical translations techniques as of 2016.\n",
    "  * Path of neural translation similar to image classification in 2012: just surpased state of the art and now moving past it rapidly.\n",
    "* Four big wins of Neural MT:\n",
    "  1. End-to-end training: all params are optimised to minimise loss function (less hyperparams)\n",
    "  2. Distributed rep share strength: better exploit word and phrase similarities.\n",
    "  3. Better exploitation of context: NMT can use a much bigger context to translate more accurately.\n",
    "  4. More fluent text generation.\n",
    "  \n",
    "* Models use Bidirectional LSTM with Attention (which is obviously not just useful for machine translation)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 00:10:05 - Translate French into English\n",
    "\n",
    "* Basic idea: make it look like a standard NN problem, need 3 things:\n",
    "  1. Data (x, y pairs)\n",
    "  2. Architecture\n",
    "  3. Loss function\n",
    "  \n",
    "* Lots of parallel corpuses (some language -> some other language), especially for European documents.\n",
    "* For bounding boxes, all interesting stuff is in loss function, for translation, it's all in the arch.\n",
    "\n",
    "## 00:13:16 - Neural translation walkthrough\n",
    "\n",
    "* Take a sentence in English and put it through an RNN/Encoder.\n",
    "* Encoder: piece of NN architecture that takes input and turns into some representation.\n",
    "* Decoder: take the encoder / RNN output and convert into a sequence of French tokens.\n",
    "\n",
    "* For translating language, you don't know how many words should be outputted from a sentence.\n",
    "  * Key issue: arbitrary length output which don't correspond to the input length.\n",
    " \n",
    "## 00:18:19 - RNN revision\n",
    "\n",
    "* Need to understand Lesson 6 if the lesson doesn't make sense.\n",
    "* RNN is a standard fully connected network, which takes an input to a linear layer which is fed into another layer and so on. However, it has one key difference: the second layer can also accept and concat another input.\n",
    "\n",
    "<img src=\"https://i.gyazo.com/900233717de09d0ac63b4330a2c6b877.gif\" width=\"400px\">\n",
    "\n",
    "* Use the same weight matrix for each of the layer outputs and the same weight matrix for each input.\n",
    "\n",
    "* The diagram can be refactored to be a for loop:\n",
    "\n",
    "<img src=\"https://i.gyazo.com/a53c737b2b3c325112430d9d3ad4b6a5.gif\" width=\"400px\">\n",
    "\n",
    "<img src=\"https://i.gyazo.com/82ecea54084aab349d420720a0caa647.gif\" width=\"400px\">\n",
    "\n",
    "  * The refactoring is basically what makes it an RNN.\n",
    "  \n",
    "* RNNs can be stacked: output of one RNN can be fed into another:\n",
    "\n",
    "<img src=\"https://i.gyazo.com/0383008c47d943200ea38423ffcb3071.gif\" width=\"400px\">\n",
    "\n",
    "  * Need to be able to write it from scratch to really understand it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from pathlib import Path\n",
    "import pickle\n",
    "from collections import Counter, defaultdict\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "from fastai.text import Tokenizer, partition_by_cores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH = Path('data/translate')\n",
    "TMP_PATH = PATH / 'tmp'\n",
    "TMP_PATH.mkdir(exist_ok=True)\n",
    "fname = 'giga-fren.release2.fixed'\n",
    "en_fname = PATH / f'{fname}.en'\n",
    "fr_fname = PATH / f'{fname}.fr'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Start do not rerun**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2018-08-04 15:34:53--  http://www.statmt.org/wmt10/training-giga-fren.tar\n",
      "Resolving www.statmt.org (www.statmt.org)... 129.215.197.184\n",
      "Connecting to www.statmt.org (www.statmt.org)|129.215.197.184|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2595102720 (2.4G) [application/x-tar]\n",
      "Saving to: 'data/translate/training-giga-fren.tar’\n",
      "\n",
      "training-giga-fren. 100%[===================>]   2.42G  4.03MB/s    in 20m 9s  \n",
      "\n",
      "2018-08-04 15:55:03 (2.05 MB/s) - 'data/translate/training-giga-fren.tar’ saved [2595102720/2595102720]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget http://www.statmt.org/wmt10/training-giga-fren.tar --directory-prefix={PATH}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "giga-fren.release2.fixed.en.gz\n",
      "giga-fren.release2.fixed.fr.gz\n"
     ]
    }
   ],
   "source": [
    "!cd {PATH} && tar -xvf training-giga-fren.tar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd {PATH} && gunzip giga-fren.release2.fixed.en.gz && gunzip giga-fren.release2.fixed.fr.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Training translation models takes a long time.\n",
    "  * No conceptual difference between 2 and 8 layers: use 2 layers because we think it should be enough.\n",
    "* Find questions that start with Wh (what, where, when etc) and match with French questions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "re_eng_questions = re.compile('^(Wh[^?.!]+\\?)')\n",
    "re_french_questions = re.compile('^([^?.!]+\\?)')\n",
    "\n",
    "en_fh = open(en_fname, encoding='utf-8')\n",
    "fr_fh = open(fr_fname, encoding='utf-8')\n",
    "\n",
    "lines = []\n",
    "\n",
    "for eq, fq in zip(en_fh, fr_fh):\n",
    "    lines.append((\n",
    "        re_eng_questions.search(eq),\n",
    "        re_french_questions.search(fq)\n",
    "    ))\n",
    "\n",
    "questions = [(e.group(), f.group()) for e, f in lines if e and f]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(questions, (PATH / 'fr-en-qs.pkl').open('wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **End do not rerun**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = pickle.load((PATH / 'fr-en-qs.pkl').open('rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We now have 52k sentence pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([('What is light ?', 'Qu’est-ce que la lumière?'),\n",
       "  ('Who are we?', 'Où sommes-nous?'),\n",
       "  ('Where did we come from?', \"D'où venons-nous?\"),\n",
       "  ('What would we do without it?', 'Que ferions-nous sans elle ?'),\n",
       "  ('What is the absolute location (latitude and longitude) of Badger, Newfoundland and Labrador?',\n",
       "   'Quelle sont les coordonnées (latitude et longitude) de Badger, à Terre-Neuve-etLabrador?')],\n",
       " 52331)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "questions[:5], len(questions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Separate questions into each language:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_qs, fr_qs = zip(*questions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Start do no rerun**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Tokenise using English and French tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting https://github.com/explosion/spacy-models/releases/download/fr_core_news_sm-2.0.0/fr_core_news_sm-2.0.0.tar.gz\n",
      "\u001b[?25l  Downloading https://github.com/explosion/spacy-models/releases/download/fr_core_news_sm-2.0.0/fr_core_news_sm-2.0.0.tar.gz (39.8MB)\n",
      "\u001b[K    100% |████████████████████████████████| 39.8MB 2.4MB/s ta 0:00:011\n",
      "\u001b[?25hInstalling collected packages: fr-core-news-sm\n",
      "  Running setup.py install for fr-core-news-sm ... \u001b[?25ldone\n",
      "\u001b[?25hSuccessfully installed fr-core-news-sm-2.0.0\n",
      "\n",
      "\u001b[93m    Linking successful\u001b[0m\n",
      "    /home/lex/anaconda3/envs/fastai/lib/python3.6/site-packages/fr_core_news_sm\n",
      "    -->\n",
      "    /home/lex/anaconda3/envs/fastai/lib/python3.6/site-packages/spacy/data/fr\n",
      "\n",
      "    You can now load the model via spacy.load('fr')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!python -m spacy download fr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_tok = Tokenizer.proc_all_mp(partition_by_cores(en_qs))\n",
    "fr_tok = Tokenizer.proc_all_mp(partition_by_cores(fr_qs), 'fr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['what', 'is', 'light', '?'],\n",
       " ['qu’', 'est', '-ce', 'que', 'la', 'lumière', '?'])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_tok[0], fr_tok[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Want to find the largest 90th sequence sentences, and make that the max sequence length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(23.0, 28.0)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.percentile([len(o) for o in en_tok], 90), np.percentile([len(o) for o in fr_tok], 90)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "keep = np.array([len(o) < 30 for o in en_tok])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_tok = np.array(en_tok)[keep]\n",
    "fr_tok = np.array(fr_tok)[keep]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(en_tok, (PATH / 'en_tok.pkl').open('wb'))\n",
    "pickle.dump(fr_tok, (PATH / 'fr_tok.pkl').open('wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **End do no rerun**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_tok = pickle.load((PATH / 'en_tok.pkl').open('rb'))\n",
    "fr_tok = pickle.load((PATH / 'fr_tok.pkl').open('rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Don't need to know a lot of NLP stuff for deep learning on text, but the basics are useful: particurally tokenising.\n",
    "* 00:28:37 - some students in the study group are trying to build language models for Chinese, need a tokeniser like [sentence piece](https://github.com/google/sentencepiece), since it doesn't have individual words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Next, turn tokens into numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "def toks2ids(tok, pre):\n",
    "    freq = Counter(p for o in tok for p in o)\n",
    "    itos = [o for o, c in freq.most_common(40000)]\n",
    "    itos.insert(0, '_bos_')\n",
    "    itos.insert(1, '_pad_')\n",
    "    itos.insert(2, '_eos_')\n",
    "    itos.insert(3, '_unk')\n",
    "    stoi = defaultdict(lambda: 3, {v: k for k, v in enumerate(itos)})\n",
    "    ids = np.array([([stoi[o] for o in p] + [2]) for p in tok])\n",
    "    np.save(TMP_PATH / f'{pre}_ids.npy', ids)\n",
    "    pickle.dump(itos, open(TMP_PATH / f'{pre}_itos.pkl', 'wb'))\n",
    "    return ids, itos, stoi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_ids, en_itos, en_stoi = toks2ids(en_tok, 'en')\n",
    "fr_ids, fr_itos, fr_stoi = toks2ids(fr_tok, 'fr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_ids(pre):\n",
    "    ids = np.load(TMP_PATH / f'{pre}_ids.npy')\n",
    "    itos = pickle.load(open(TMP_PATH / f'{pre}_itos.pkl', 'rb'))\n",
    "    stoi = defaultdict(lambda: 3, {v:k for k, v in enumerate(itos)})\n",
    "    return ids, itos, stoi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_ids, en_itos, en_stoi = load_ids('en')\n",
    "fr_ids, fr_itos, fr_stoi = load_ids('fr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['qu’', 'est', '-ce', 'que', 'la', 'lumière', '?', '_eos_'], 17573, 24793)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[fr_itos[o] for o in fr_ids[0]], len(en_itos), len(fr_itos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 00:33:01 - Word vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Seq-to-seq with language models hasn't been explored yet in academia: lots of potential papers to be written.\n",
    "* Word2Vec has been surpased by a number of word vectors: FastText is a good choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Start do not rerun**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/facebookresearch/fastText.git\n",
      "  Cloning https://github.com/facebookresearch/fastText.git to /tmp/pip-req-build-erf46hfl\n",
      "Requirement already satisfied (use --upgrade to upgrade): fasttext==0.8.22 from git+https://github.com/facebookresearch/fastText.git in /home/lex/anaconda3/envs/fastai/lib/python3.6/site-packages\n",
      "Requirement already satisfied: pybind11>=2.2 in /home/lex/anaconda3/envs/fastai/lib/python3.6/site-packages (from fasttext==0.8.22) (2.2.3)\n",
      "Requirement already satisfied: setuptools>=0.7.0 in /home/lex/anaconda3/envs/fastai/lib/python3.6/site-packages (from fasttext==0.8.22) (40.0.0)\n",
      "Requirement already satisfied: numpy in /home/lex/anaconda3/envs/fastai/lib/python3.6/site-packages (from fasttext==0.8.22) (1.14.5)\n",
      "Building wheels for collected packages: fasttext\n",
      "  Running setup.py bdist_wheel for fasttext ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /tmp/pip-ephem-wheel-cache-s3ecl52n/wheels/69/f8/19/7f0ab407c078795bc9f86e1f6381349254f86fd7d229902355\n",
      "Successfully built fasttext\n"
     ]
    }
   ],
   "source": [
    "!pip install git+https://github.com/facebookresearch/fastText.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Need to also download the fasttext word vectors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.en.zip --directory-prefix={PATH}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.fr.zip  --directory-prefix={PATH}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  wiki.en.zip\n",
      "  inflating: wiki.en.vec             \n",
      "  inflating: wiki.en.bin             \n",
      "Archive:  wiki.fr.zip\n",
      "  inflating: wiki.fr.vec             \n",
      "  inflating: wiki.fr.bin             \n"
     ]
    }
   ],
   "source": [
    "!cd {PATH} && unzip wiki.en.zip && unzip wiki.fr.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **End do not rerun**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fastText as ft"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "english_vecs = ft.load_model(str((PATH / 'wiki.en.bin')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "french_vecs = ft.load_model(str((PATH / 'wiki.fr.bin')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Turn it into a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_vecs(lang, ft_vecs):\n",
    "    vecd = {w: ft_vecs.get_word_vector(w) for w in ft_vecs.get_words()}\n",
    "    pickle.dump(vecd, open(PATH / 'wiki.{lang}.pkl'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Preparing data for PyTorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_ids_tr = np.array([o[:enlen_90] for o in en_ids])\n",
    "fr_ids_tr = np.array([o[:rnlen_90] for o in fr_ids])\n",
    "enlen_90, frlen_90"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def iter_to_numpy(*a):\n",
    "    \"\"\"convert iterable object into numpy array\"\"\"\n",
    "    return np.array(a[0]) if len(a)==1 else [np.array(o) for o in a]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqDataset(Dataset):\n",
    "    def __init__(self, x, y):\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "        \n",
    "    def __getitem__(self, idx):\n",
    "        return iter_to_numpy(self.x[idx], self.y[idx])\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "trn_keep = np.random.rand(len(en_ids_tr)) > 0.1\n",
    "en_trn, fr_trn = es_ids_tr[trn_keep], fr_ids_tr[trn_keep]\n",
    "en_val, fr_val = en_ids_tr[~trn_keep], fr_ids_tr[~trn_keep]\n",
    "len(en_trn), len(en_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_ds = Seq2SeqDataset(fr_trn, en_trn)\n",
    "val_ds = Seq2SeqDataset(fr_val, en_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 125"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}