{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0c97d6cd",
   "metadata": {},
   "source": [
    "# ランダムフォレスト\n",
    "\n",
    "ランダムフォレストについては以下を参照してください。\n",
    "[https://scikit-learn.org/stable/modules/ensemble.html#forest](https://scikit-learn.org/stable/modules/ensemble.html#forest)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beaeafb3",
   "metadata": {},
   "source": [
    "**データとモジュールのロード**\n",
    "\n",
    "学習に使うデータをロードします。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "aa65ec88",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn import model_selection\n",
    "\n",
    "data = pd.read_csv(\"input/pn_same_judge_preprocessed.csv\")\n",
    "train, test = model_selection.train_test_split(data, test_size=0.1, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a8500c05",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics import PrecisionRecallDisplay"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c69e19a",
   "metadata": {},
   "source": [
    "## 決定木\n",
    "\n",
    "ランダムフォレストは決定木の\n",
    "[バギング](https://ja.wikipedia.org/wiki/バギング)\n",
    "によりアンサンブル学習する手法なので、\n",
    "まずは決定木から始めましょう。\n",
    "\n",
    "決定木を学習するには\n",
    "[sklearn.tree.DecisionTreeClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html)\n",
    "を使います。\n",
    "ここでは正則化のために `max_depth`, `min_samples_leaf` パラメータを指定しています。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d0497415",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "817bb85b",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_dt = Pipeline([\n",
    "    (\"vect\", TfidfVectorizer(tokenizer=str.split)),\n",
    "    (\"clf\", DecisionTreeClassifier(max_depth=2, min_samples_leaf=10, random_state=0)),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "77ec26e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                 TfidfVectorizer(tokenizer=&lt;method &#x27;split&#x27; of &#x27;str&#x27; objects&gt;)),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 DecisionTreeClassifier(max_depth=2, min_samples_leaf=10,\n",
       "                                        random_state=0))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                 TfidfVectorizer(tokenizer=&lt;method &#x27;split&#x27; of &#x27;str&#x27; objects&gt;)),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 DecisionTreeClassifier(max_depth=2, min_samples_leaf=10,\n",
       "                                        random_state=0))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(tokenizer=&lt;method &#x27;split&#x27; of &#x27;str&#x27; objects&gt;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">DecisionTreeClassifier</label><div class=\"sk-toggleable__content\"><pre>DecisionTreeClassifier(max_depth=2, min_samples_leaf=10, random_state=0)</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('vect',\n",
       "                 TfidfVectorizer(tokenizer=<method 'split' of 'str' objects>)),\n",
       "                ('clf',\n",
       "                 DecisionTreeClassifier(max_depth=2, min_samples_leaf=10,\n",
       "                                        random_state=0))])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_dt.fit(train[\"tokens\"], train[\"label_num\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6fd0041d",
   "metadata": {},
   "outputs": [],
   "source": [
    "score_dt = pipe_dt.predict_proba(test[\"tokens\"])[:,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "940d6cec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.precision_recall_curve.PrecisionRecallDisplay at 0x7f0f7578eeb0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhvUlEQVR4nO3deZgV1Z3/8fdHQBEVRME8yhJAiQuoPdhGjRuJC+ogZIxGcF+i0eCSmDjjL04QDZNHYzSZqBPEiFsUFzQ2JkQSdxMVbRJEQVRUoq1m7EEBFVn9/v6o6k7TNNxquuteuu/n9Tz9dFWdc6u+1SzfPudUnaOIwMzMytcmpQ7AzMxKy4nAzKzMORGYmZU5JwIzszLnRGBmVuY6ljqA5urRo0f069ev1GGYmbUpM2fO/L+I6NlUWZtLBP369aO6urrUYZiZtSmS/r6uMncNmZmVOScCM7My50RgZlbmnAjMzMqcE4GZWZnLLRFImiTpA0kvr6Nckn4pab6k2ZKG5BWLmZmtW54tgluBI9ZTfiQwMP06G/hVjrGYmdk65PYeQUQ8JanfeqqMBG6PZB7s5yRtLWn7iHg/j3guf2gOc99bksepW2RkRS9O2KdvqcMwszJWyjGCXsA7DfZr0mNrkXS2pGpJ1bW1tUUJrhjmvr+EqlnvljoMMytzbeLN4oiYCEwEqKys3KCVdC47elCrxtQajr/x2VKHYGZW0hbBu0CfBvu902NmZlZEpUwEU4FT0qeH9gUW5zU+YGZm65Zb15CkycBQoIekGuAyoBNAREwApgFHAfOBpcDpecViZmbrludTQ6MLlAcwJq/rm5lZNn6z2MyszDkRmJmVOScCM7My50RgZlbmnAjMzMqcE4GZWZlzIjAzK3NOBGZmZc6JwMyszDkRmJmVOScCM7My50RgZlbmnAjMzMqcE4GZWZlzIjAzK3NOBGZmZc6JwMyszDkRmJmVOScCM7My50RgZlbmnAjMzMqcE4GZWZlzIjAzK3NOBGZmZc6JwMyszDkRmJmVOScCM7My50RgZlbmOmapJGk7YH9gB+Az4GWgOiI+zzE2MzMrgvUmAklfBS4BtgH+BnwAdAa+DuwoaQpwTUQsyTlOMzPLSaEWwVHAWRHxduMCSR2B4cBhwP05xGZmZkWw3kQQERevp2wV8GBrB2RmZsW1wYPFkk5vzUDMzKw0WvLU0OWtFoWZmZVMocHi2esqAr7Q+uGYmVmxFRos/gIwDPio0XEBz+QSkZmZFVWhRPA7YMuImNW4QNITeQRkZmbFVeipoTPXU3ZC64djZmbFlusUE5KOkPSqpPmSLmmivK+kxyX9TdJsSUflGY+Zma0tt0QgqQNwA3AksBswWtJujar9J3BvRPwLMAr4n7ziMTOzpuXZIvgyMD8i3oyIFcDdwMhGdQLomm53A97LMR4zM2tCnomgF/BOg/2a9FhD44CTJNUA04DzmzqRpLMlVUuqrq2tzSNWM7OylTkRSJq4vv0NNBq4NSJ6k8xrdIektWKKiIkRURkRlT179myFy5qZWZ3mtAhuLLDf2LtAnwb7vdNjDZ0J3AsQEc+SzGzaoxkxmZlZC2VOBBExc337TXgBGCipv6RNSQaDpzaq8zZwCICkXUkSgft+zMyKqNAUEw+RDOg2KSJGrKdslaTzgOlAB2BSRMyRdAXJojZTge8DN0n6Xnqd0yJindczM7PWV+jN4p+15OQRMY1kELjhsbENtueSrHxmZmYlUujN4ifrtiVtDvSNiFdzj8rMzIom0xiBpKOBWcDD6X6FpMb9/WZm1gZlHSweR/KC2CKAdBK6/rlEZGZmRZU1EayMiMWNjnlQ18ysHSg0WFxnjqQTgA6SBgIX4PUIzMzahawtgvOBQcByYDKwBPhuTjGZmVkRZWoRRMRS4FJJVyW78XG+YZmZWbFkfWpob0kvAbOBlyS9KGmvfEMzM7NiyDpGcDPwnYh4GkDSAcAtwB55BWZmZsWRdYxgdV0SAIiIPwOr8gnJzMyKqdBcQ0PSzScl3UgyUBzA8cAT+YZmZmbFUKhr6JpG+5c12PZ7BGZm7UChuYa+WqxAzMysNLIOFiPpX0neJehcdywirsgjKDMzK56sj49OIBkXOB8QcBzwxRzjMjOzIsn61NBXIuIU4KOIuBzYD/hSfmGZmVmxZE0En6Xfl0raAVgJbJ9PSGZmVkxZxwh+J2lr4GrgryRPDP06r6DMzKx4ss419ON0835JvwM6NzEttZmZtUGFXig7Zj1lRMQDrR+SmZkVU6EWwdHrKQvAicDMrI0r9ELZ6cUKxMzMSiPrU0NmZtZOORGYmZU5JwIzszKXdYqJLpJ+JOmmdH+gpOH5hmZmZsWQtUVwC8nC9ful++8C43OJyMzMiiprItgxIn5KMrVE3WL2yi0qMzMrmqyJYIWkzUkXo5G0I0kLwczM2riscw2NAx4G+ki6E9gfOC2nmMzMrIiyzjX0R0kzgX1JuoQujIj/yzUyMzMrikyJQNJDwF3A1Ij4NN+QzMysmLKOEfwMOBCYK2mKpGMldS70ITMz2/hl7Rp6EnhSUgfga8BZwCSga46xmZlZETRn8frNSWYjPR4YAtyWV1BmZlY8WccI7gW+TPLk0PXAkxHxeZ6BmZlZcWRtEdwMjI6I1XkGY2ZmxVdohbKvRcRjwBbASGnNl4m9QpmZWdtXqEVwMPAYTa9UVnCFMklHAP8NdAB+HRFXNlHnmyQvrAXwYkScUDhsMzNrLYVWKLss3bwiIt5qWCap//o+mz5hdANwGFADvCBpakTMbVBnIPD/gP0j4iNJ223APZiZWQtkfY/g/iaOTSnwmS8D8yPizYhYAdwNjGxU5yzghoj4CCAiPsgYj5mZtZJCYwS7AIOAbpKOaVDUFSj0Qlkv4J0G+zXAPo3qfCm9zl9Iuo/GRcTDTcRxNnA2QN++fQtc1szMmqPQGMHOwHBga9YcJ/iY5Lf51rj+QGAo0Bt4StLuEbGoYaWImAhMBKisrIxWuK6ZmaUKjRFUAVWS9ouIZ5t57neBPg32e6fHGqoBZkTESuAtSa+RJIYXmnktMzPbQIW6hv49XZDmBEmjG5dHxAXr+fgLwMB0UPldYBTQ+ImgB4HRwC2SepB0Fb2ZPXwzM2upQl1Dr6Tfq5t74ohYJek8YDpJ//+kiJgj6QqgOiKmpmWHS5oLrAYujoiFzb2WmZltuEJdQw+l3+vnFZK0CbBlRCwpdPKImAZMa3RsbIPtAC5Kv8zMrAQyPT4q6S5JXSVtAbxMMh31xfmGZmZmxZD1PYLd0hbA14E/AP2Bk/MKyszMiidrIugkqRNJIpiaPuXjxzjNzNqBrIngRmAByeRzT0n6IlBwjMDMzDZ+WVco+yXwywaH/i7pq/mEZGZmxZR1sLibpGslVadf15C0DszMrI3L2jU0iWRaiW+mX0uAW/IKyszMiifrCmU7RsQ3GuxfLmlWDvGYmVmRZW0RfCbpgLodSfsDn+UTkpmZFVPWFsE5wO2SuqX7HwGn5hOSmZkVU8FEIKkC2Ilk0rh3AbJML2FmZm3DeruGJI0F7gW+AfweON5JwMysfSnUIjgeqIiIpZK2BR4Gbso/LDMzK5ZCg8XLI2IpQDo9dNbBZTMzayMKtQgGSJqabgvYscE+ETEit8jMzKwoCiWCkY32f5ZXIGZmVhqFFqZ5sliBmJlZaRR6aughSUenU1A3Lhsg6QpJZ+QXnpmZ5a1Q19BZJMtI/kLSh0At0BnoB7wBXB8RVblGaGZmuSrUNfQP4N+Bf5fUD9ieZGqJ1+qeJjIzs7Yt6xQTRMQCksVpzMysHfF7AWZmZc6JwMyszDkRmJmVuUxjBOn6A+OAL6afERARMSC/0MzMrBiyDhbfDHwPmAmszi8cMzMrtqyJYHFE/CHXSMzMrCSyJoLHJV0NPAAsrzsYEX/NJSozMyuarIlgn/R7ZYNjAXytdcMxM7Niy5QIIuKreQdiZmalkenxUUndJF0rqTr9uqbBQvZmZtaGZX2PYBLwMfDN9GsJcEteQZmZWfFkHSPYMSK+0WD/ckmzcojHzMyKLGuL4DNJB9TtpC+YfZZPSGZmVkxZWwTnArel4wICPgROyysoMzMrnqxPDc0C9pTUNd1fkmdQZmZWPOtNBJJOiojfSLqo0XEAIuLaHGMzM7MiKNQi2CL9vlXegZiZWWkUWqryxvT75RtycklHAP8NdAB+HRFXrqPeN4ApwN4RUb0h1zIzsw2T9YWyn0rqKqmTpEcl1Uo6qcBnOgA3AEcCuwGjJe3WRL2tgAuBGc0P38zMWirr46OHpwPEw0nWLd4JuLjAZ74MzI+INyNiBXA3MLKJej8GrgKWZYzFzMxaUdZEUNeF9K/AfRGxOMNnegHvNNivSY/VkzQE6BMRv1/fiSSdXTe9RW1tbcaQzcwsi6yJ4HeS5gF7AY9K6kkLf4OXtAlwLfD9QnUjYmJEVEZEZc+ePVtyWTMzayRTIoiIS4CvAJURsRL4lKa7eRp6F+jTYL93eqzOVsBg4AlJC4B9gamSGk51bWZmOSv0HsHXIuIxScc0ONawygPr+fgLwEBJ/UkSwCjghLrCtHupR4PzPgH8wE8NmZkVV6H3CA4GHgOObqIsWE8iiIhVks4DppM8PjopIuZIugKojoipGxizmZm1okLvEVyWfj99Q04eEdOAaY2OjV1H3aEbcg0zM2uZrO8R/ETS1g32u0san1tUZmZWNFlnHz0yIn5YtxMRH0k6CvjPfMIqH3PfX8LxNz5b6jDM1jCyohcn7NO31GFYkWRNBB0kbRYRywEkbQ5sll9Y5WFkRa/ClcyKbO77yeTCTgTlI2siuJPk/YG65SlPB27LJ6TyccI+ff2PzTY6bqGWn6zrEVwl6UXg0PTQjyNien5hmZlZsWRtEQC8AqyKiEckdZG0VUR8nFdgZmZWHFmfGjqLZJroG9NDvYAHc4rJzMyKKOtcQ2OA/YElABHxOrBdXkGZmVnxZE0Ey9OppAGQ1JHkzWIzM2vjsiaCJyX9ENhc0mHAfcBD+YVlZmbFkjUR/AdQC7wEfJtk2gi/TGZm1g4UfGooXXJyTkTsAtyUf0hmZlZMBVsEEbEaeFWS33wyM2uHsr5H0B2YI+l5kkVpAIiIEblEZWZmRZM1Efwo1yjMzKxkCq1Q1hk4B9iJZKD45ohYVYzAzMysOAqNEdwGVJIkgSOBa3KPyMzMiqpQ19BuEbE7gKSbgefzD8nMzIqpUItgZd2Gu4TMzNqnQi2CPSUtSbdF8mbxknQ7IqJrrtGZmVnuCi1e36FYgZiZWWlknWLCzMzaKScCM7My50RgZlbmnAjMzMqcE4GZWZlzIjAzK3NOBGZmZc6JwMyszDkRmJmVOScCM7My50RgZlbmnAjMzMqcE4GZWZlzIjAzK3NOBGZmZc6JwMyszDkRmJmVuUJLVbaIpCOA/wY6AL+OiCsblV8EfAtYBdQCZ0TE35t7nZUrV1JTU8OyZctaIWqzDdO5c2d69+5Np06dSh2KWbPklggkdQBuAA4DaoAXJE2NiLkNqv0NqIyIpZLOBX4KHN/ca9XU1LDVVlvRr18/JLVG+GbNEhEsXLiQmpoa+vfvX+pwzJolz66hLwPzI+LNiFgB3A2MbFghIh6PiKXp7nNA7w250LJly9h2222dBKxkJLHtttu6VWptUp6JoBfwToP9mvTYupwJ/KGpAklnS6qWVF1bW9vkh50ErNT8d9Daqo1isFjSSUAlcHVT5RExMSIqI6KyZ8+exQ3OzKydyzMRvAv0abDfOz22BkmHApcCIyJieY7x5KpDhw5UVFQwaNAg9txzT6655ho+//zzDTrX2LFjeeSRR9ZZPmHCBG6//fYNDRWAl156iYqKCioqKthmm23o378/FRUVHHrooS06b2MPPvggV1xxxRrHKioqGDVq1BrHTjvttPoYhgwZwrPPPtviaz/88MPsvPPO7LTTTlx55ZVN1pkwYQK77747FRUVHHDAAcydmwxh3XnnnfU/n4qKCjbZZBNmzZoFwKGHHspHH33U4vjMNhoRkcsXyUD0m0B/YFPgRWBQozr/ArwBDMx63r322isamzt37lrHim2LLbao3/7f//3fOOSQQ2Ls2LEljCi7U089Ne677761jq9cubLF595vv/2itra2fn/u3LkxePDg2GGHHeKTTz5pMobp06fH7rvv3qLrrlq1KgYMGBBvvPFGLF++PPbYY4+YM2fOWvUWL15cv11VVRXDhg1bq87s2bNjwIAB9fu33nprjB8/vsnrbgx/F1vqmxOeiW9OeKbUYVgrA6pjHf+v5vbUUESsknQeMJ3k8dFJETFH0hVpQFNJuoK2BO5L+1ffjogRLbnu5Q/NYe57S1oY/Zp226Erlx09KHP97bbbjokTJ7L33nszbtw4Pv/8cy655BKeeOIJli9fzpgxY/j2t78NwFVXXcVvfvMbNtlkE4488kiuvPJKTjvtNIYPH86xxx7LJZdcwtSpU+nYsSOHH344P/vZzxg3bhxbbrklP/jBD5g1axbnnHMOS5cuZccdd2TSpEl0796doUOHss8++/D444+zaNEibr75Zg488MCCsQ8dOpSKigr+/Oc/M3r0aIYOHcpFF13EJ598Qo8ePbj11lvZfvvteeONNxgzZgy1tbV06dKFm266iV122WWNc7322mtsttlm9OjRo/7Y5MmTOfnkk3nllVeoqqrihBNOWCuGgw46iPnz52f+eTfl+eefZ6eddmLAgAEAjBo1iqqqKnbbbbc16nXt2rV++9NPP22yn3/y5MlrtGBGjBjBgQceyKWXXtqiGM02Frm+RxAR04BpjY6NbbDduv0QG5EBAwawevVqPvjgA6qqqujWrRsvvPACy5cvZ//99+fwww9n3rx5VFVVMWPGDLp06cKHH364xjkWLlzIb3/7W+bNm4ckFi1atNZ1TjnlFK677joOPvhgxo4dy+WXX84vfvELAFatWsXzzz/PtGnTuPzyy9fb3dTQihUrqK6uZuXKlRx88MFUVVXRs2dP7rnnHi699FImTZrE2WefzYQJExg4cCAzZszgO9/5Do899tga5/nLX/7CkCFD1jh2zz338Kc//Yl58+Zx3XXXNZkIHnroIXbfffe1jt95551cffXaw0g77bQTU6ZMWePYu+++S58+/+yZ7N27NzNmzGjyfm+44QauvfZaVqxYsdY91MVcVVVVv9+9e3eWL1/OwoUL2XbbbZs8p1lbkmsiKIXm/OZeLH/84x+ZPXt2/X9Wixcv5vXXX+eRRx7h9NNPp0uXLgBss802a3yuW7dudO7cmTPPPJPhw4czfPjwNcoXL17MokWLOPjggwE49dRTOe644+rLjznmGAD22msvFixYkDne449PXuV49dVXefnllznssMMAWL16Ndtvvz2ffPIJzzzzzBrXWr587eGd999/n4aD+9XV1fTo0YO+ffvSq1cvzjjjDD788MP6+7744osZP348PXv25Oabb17rfCeeeCInnnhi5vvIasyYMYwZM4a77rqL8ePHc9ttt9WX1SXpwYMHr/GZ7bbbjvfee8+JwNqFdpcINhZvvvkmHTp0YLvttiMiuO666xg2bNgadaZPn77ec3Ts2JHnn3+eRx99lClTpnD99dc3+Rvrumy22WZAMpC9atWqzJ/bYostgGT8aNCgQWsN3C5ZsoStt966fvB0XTbffHMWL15cvz958mTmzZtHv3796s9z//33c9ZZZwFw9dVXc+yxx67zfM1pEfTq1Yt33vnn08s1NTX06rW+p5eT7qNzzz13jWN33303o0ePXqvusmXL2Hzzzdd7PrO2YqN4fLS9qa2t5ZxzzuG8885DEsOGDeNXv/oVK1euBJK+808//ZTDDjuMW265haVLk3fqGncNffLJJyxevJijjjqKn//857z44otrlHfr1o3u3bvz9NNPA3DHHXfUtw5aw84770xtbW19Ili5ciVz5syha9eu9O/fn/vuuw9IEkbj2AB23XXX+r7+zz//nHvvvZeXXnqJBQsWsGDBAqqqqpg8eXLmeE488URmzZq11lfjJACw99578/rrr/PWW2+xYsUK7r77bkaMWHv46fXXX6/f/v3vf8/AgQPr9+tibvyEU0Twj3/8oz6hmbV1bhG0ks8++4yKigpWrlxJx44dOfnkk7nooosA+Na3vsWCBQsYMmQIEUHPnj158MEHOeKII5g1axaVlZVsuummHHXUUfzkJz+pP+fHH3/MyJEjWbZsGRHBtddeu9Z1b7vttvrB4gEDBnDLLbe02j1tuummTJkyhQsuuIDFixezatUqvvvd7zJo0CDuvPNOzj33XMaPH8/KlSsZNWoUe+655xqfP+igg/j+979PRPD000/Tq1cvdthhhzXK586dy/vvv99qMdfp2LEj119/PcOGDWP16tWcccYZDBqUdBuOHTuWyspKRowYwfXXX88jjzxCp06d6N69+xrdQk899RR9+vSpH3CuM3PmTPbdd186dmy//3zmvr+E429s+SO81rqa++BKVkqeKmo7Kisro7q6eo1jr7zyCrvuumuJIrL1ufDCCzn66KNb/f2EUrrwwgsZMWIEhxxyyFpl7eHv4l0z3qZq1lqv/NhGoCWJQNLMiKhsqqz9/kpjG4Uf/vCH63xap60aPHhwk0mgvThhn76csE/fUodhReQxAsvVF77whSb75tuyusFts/ai3SSCttbFZe2P/w5aW9UuEkHnzp1ZuHCh/yFayUS6HkHnzp1LHYpZs7WLMYLevXtTU1PDuqaoNiuGuhXKzNqadpEIOnXq5FWhzMw2ULvoGjIzsw3nRGBmVuacCMzMylybe7NYUi3w9w38eA/g/1oxnLbA91wefM/loSX3/MWIaHKt3zaXCFpCUvW6XrFur3zP5cH3XB7yumd3DZmZlTknAjOzMlduiWBiqQMoAd9zefA9l4dc7rmsxgjMzGxt5dYiMDOzRpwIzMzKXLtMBJKOkPSqpPmSLmmifDNJ96TlMyT1K0GYrSrDPV8kaa6k2ZIelfTFUsTZmgrdc4N635AUktr8o4ZZ7lnSN9M/6zmS7ip2jK0tw9/tvpIel/S39O/3UaWIs7VImiTpA0kvr6Nckn6Z/jxmSxrS4otGRLv6AjoAbwADgE2BF4HdGtX5DjAh3R4F3FPquItwz18FuqTb55bDPaf1tgKeAp4DKksddxH+nAcCfwO6p/vblTruItzzRODcdHs3YEGp427hPR8EDAFeXkf5UcAfAAH7AjNaes322CL4MjA/It6MiBXA3cDIRnVGAnWrlE8BDpGkIsbY2grec0Q8HhFL093ngLY+X3KWP2eAHwNXAcuKGVxOstzzWcANEfERQER8UOQYW1uWew6ga7rdDXiviPG1uoh4CvhwPVVGArdH4jlga0nbt+Sa7TER9ALeabBfkx5rsk5ErAIWA9sWJbp8ZLnnhs4k+Y2iLSt4z2mTuU9E/L6YgeUoy5/zl4AvSfqLpOckHVG06PKR5Z7HASdJqgGmAecXJ7SSae6/94LaxXoElp2kk4BK4OBSx5InSZsA1wKnlTiUYutI0j00lKTV95Sk3SNiUSmDytlo4NaIuEbSfsAdkgZHxOelDqytaI8tgneBPg32e6fHmqwjqSNJc3JhUaLLR5Z7RtKhwKXAiIhYXqTY8lLonrcCBgNPSFpA0pc6tY0PGGf5c64BpkbEyoh4C3iNJDG0VVnu+UzgXoCIeBboTDI5W3uV6d97c7THRPACMFBSf0mbkgwGT21UZypwarp9LPBYpKMwbVTBe5b0L8CNJEmgrfcbQ4F7jojFEdEjIvpFRD+ScZEREVFdmnBbRZa/2w+StAaQ1IOkq+jNIsbY2rLc89vAIQCSdiVJBO153dqpwCnp00P7Aosj4v2WnLDddQ1FxCpJ5wHTSZ44mBQRcyRdAVRHxFTgZpLm43ySQZlRpYu45TLe89XAlsB96bj42xExomRBt1DGe25XMt7zdOBwSXOB1cDFEdFmW7sZ7/n7wE2SvkcycHxaW/7FTtJkkmTeIx33uAzoBBARE0jGQY4C5gNLgdNbfM02/PMyM7NW0B67hszMrBmcCMzMypwTgZlZmXMiMDMrc04EZmZlzonAcidptaRZkl6W9JCkrVv5/AvSZ+aR9Mk66mwu6UlJHST1k/RZGtNcSRPSN5Gbc81KSb9Mt4dK+kqDsnMkndKSe0rPM07SDwrUuVXSsc04Z791zWrZqN5/SXqn8c9T0nmSzsh6PWsbnAisGD6LiIqIGEzy3saYEsRwBvBARKxO99+IiApgD5IZK7/enJNFRHVEXJDuDgW+0qBsQkTc3tKAS+whkgnfGptE+5/Lp+w4EVixPUs6QZakHSU9LGmmpKcl7ZIe/4Kk30p6Mf36Snr8wbTuHElnN/O6JwJVjQ+mkw4+A+yU/rb8mP65ZkPf9LrHpa2ZFyU9lR4bKul3StayOAf4XtrCOLDuN3lJu0h6vu5a6flfSrf3SlsoMyVNV4HZIyWdJemFNIb7JXVpUHyopGpJr0kantbvIOnq9DOzJX27OT+siHiuqbdV0xlsF0hqKklYG+VEYEUjqQPJVAB1b/1OBM6PiL2AHwD/kx7/JfBkROxJMi/7nPT4GWndSuACSZlmjE2nJhgQEQuaKOuSxvQScB1wW0TsAdyZxgEwFhiWxrPG29jpOScAP09bPU83KJsHbCqpf3roeOAeSZ3Sax2b3s8k4L8K3MYDEbF3GsMrJPPr1OlH8tv7vwITJHVOyxdHxN7A3sBZDeKou/cdJE0rcN2mVAMHbsDnbCPV7qaYsI3S5pJmkbQEXgH+JGlLku6UuikvADZLv38NOAUg7cpZnB6/QNK/pdt9SCZTyzJ9Qg9gUaNjO6YxBVAVEX+QdAdwTFp+B/DTdPsvwK2S7gUeyHC9hu4lSQBXpt+PB3YmmRDvT+m9dwAKzRUzWNJ4YGuSqUKmN7xGOtPm65LeBHYBDgf2aDB+0I3k5/Va3Yci4j2SqQqa64P0GtZOOBFYMXwWERXpb9/TScYIbgUWpf30BUkaChwK7BcRSyU9QTK5WKbrN1H3jazXjohzJO1D8hv3TEl7ZbwuwD0kye6B5FTxuqTdgTkRsV8zznMr8PWIeFHSaaQTy9WF2DhkktWrzo+IhgkDtc6yrJ1JfqbWTrhryIom7V++gGSSsKXAW5KOg/p1WPdMqz5KspxmXV93N5LfaD9Kk8AuJNNKZ73uR0CHtMtkfZ7hnxMQngg8ncawY0TMiIixJLNa9mn0uY9Jpr1u6tpvkEz+9iOSpADwKtBTydz5SOokaVCB2LYC3k+7lU5sVHacpE0k7UiypOOrJAn33LQ+kr4kaYsC18jqS0DBJ4+s7XAisKKKiL8Bs0kWEzkROFPSiyTjAHVLEF4IfDUdWJ1J8lTPw0BHSa+QdLM818xL/xE4oECd84HTJc0GTk7jALha0kvpY5fPkKyb29BDwL/VDRY3cd57gJP455z5K0imP78qvfdZNHjqaB1+BMwg6aaa16jsbeB5klXnzomIZcCvgbnAX9O4b6RRD8D6xggk/VTJzJddJNVIGtegeH/gTwXitTbEs49aWVCybOX3IuLkUsfSlilZ1+Ii/xzbF7cIrCxExF+Bx9Mnl2zD9SBpnVg74haBmVmZc4vAzKzMORGYmZU5JwIzszLnRGBmVuacCMzMytz/BzTyrgjq3mGqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "PrecisionRecallDisplay.from_predictions(\n",
    "    y_true=test[\"label_num\"],\n",
    "    y_pred=score_dt,\n",
    "    name=\"Decision Tree\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45f8e117",
   "metadata": {},
   "source": [
    "木を表示してみましょう。\n",
    "\n",
    "```{note}\n",
    "sklearn 0.21 から\n",
    "[plot_tree](https://scikit-learn.org/stable/modules/generated/sklearn.tree.plot_tree.html#sklearn.tree.plot_tree)\n",
    "で決定木を表示できるようになりました。\n",
    "\n",
    "以下のドキュメントが参考になります。\n",
    "\n",
    "* [https://scikit-learn.org/stable/auto_examples/tree/plot_iris_dtc.html](https://scikit-learn.org/stable/auto_examples/tree/plot_iris_dtc.html)\n",
    "* [https://scikit-learn.org/stable/auto_examples/tree/plot_unveil_tree_structure.html](https://scikit-learn.org/stable/auto_examples/tree/plot_unveil_tree_structure.html)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d7fb67cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADnCAYAAAC9roUQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABZzklEQVR4nO3dd3gUVRfA4d/dlt4rJYVeQgsgvXfpXQHxUxFFVKqiiL2hKAhKU0AQEBtFikjvXXpJ6ISEQHrvye58f2xYEhIggZANyX2fh8dkZnbmzLg5e3fmnnuFoihIkiRJxUNl7gAkSZLKEpl0JUmSipFMupIkScVIJl1JkqRiJJOuJElSMZJJV5IkqRjJpCtJklSMZNKVJEkqRjLpSpIkFSOZdCVJkoqRTLqSJEnFSCZdSZKkYiSTriRJUjGSSVeSJKkYyaQrSZJUjGTSlSRJKkYy6UqSJBUjmXQlSZKKkcbcAUgli5VOE5aWqfcwdxyliaVWHZ6akeVp7jikkkHIOdKknIQQSvjcoeYOo1TxGL0CRVGEueOQSgZ5e0GSJKkYyaQrSZJUjGTSlSRJKkYy6UoFduBiOJP/+A+AS2HxjFy4j+DoJOq+u5pf918hNjmdft9to8+MrYz6eT96gwGA3tO3Umn8n9yMTTHtq/rEv+j33TYGf78DgEy9gZEL99Hvu23M2RoAwKbTN2j8/loOX44o0vOY/MdRek/fysxN5/Ks2xlwkx7fbGHYnF0kpGYAMGPjGfrM2MqbvxwkS2/gcngCPb/dQu/pW/lg5bEijU0q/WTSlQqsRXUPbsWlEhSZyLQNZ3inVz0AOtWpwLCWVbDQqlnwcivWTuiMr5ste8+HA7Dw5Vb08vfKta963s6sGd+JP8d0AOCfEyE0ruzKmvGdOBUcQ1RiGt3qVeSZZpUeGFem3kCm3lCgczh5PRqNWrBuYmfOhMQQkZCaa/28bedZObYDz7WqyvJ9VwiPT+VMSCxrJ3SmgY8zW86E4mxjwR9vtGfdxM7EJKVzJTyhQMeWJJBJVyqkST3r8erP+3G2saCqh32uddY6Da52lgCohUBkP693d7DKs5+A0Dh6T9/KvG2BAARHJ1GrvCMA1T0dOHk9+oGx3IpL4ev1pxkwc7upVfogx4OiaVXd2COueVV3TgfHmNYlp2dha6nFSqehdQ0PTlyPJiQmmRrlHQCoVd6Ro1ejcLa1wMZSazxP1Z3zlKSCkP10pUKp5mlPWFwq3ft43XObiIRU9l4IZ0L3Ovfc5sDHvbCz1DBiwT7a1S5HVQ97Dl6KoHUNDw5djqCyu909X3v4cgSLdl8EYFiLKkzqWRchBNejkhi37FCubZtXc2dSz3qm3+NTMqhZzphEbS21xKdmmtYlpGZgl51MbSw0JKRm4ONqy7Fr0RgMCocuR5CQdmf78zfjiE5Kp7J77g8fSbofmXSlQlm27zJDWlRmwY7ztK2Zt79/lt7A+GWH+erZp1Cr7v1FytFaB0BHv/JcvBVPT38vdgXeYtD3O3CxszS1mPOz+XQo0UnpjOlSmzY1PRHZTU0fV1vWjO903/gdrHUkZifOpLRMfN1sTevsre6sS07Pwt5Kh5udJd3rV2TArO3ULO+Aq50FAImpmUz+4yjzXmxx3+NJ0t1k0pUKLDktk3XHglk1rgOfrjnJ3gth+Lja5trmw1XHGdjUl9oVHO+5n5SMLCw0KtQqFUevRfFC62qoVSqmDWmCoii8+ctBGld2vefrP+zvT1hcCsv2Xea7f8/RuW55Xm5Xg7D41Ae2dP19XFhzNIiu9Spy6HIEvRt5m9bZWGhISsskLVPPvovh+Pu4APBi2+q82LY6c7YG0KSKG4qiMG75Id7pVQ9PR+vCXEJJkklXKrh5288zol111CoVY7v68erP+5k+rIlpfVBUEiv2XyEwNI6ley8zulMtOtetwOjFB9h3MZygyCTe6lEXVztLxi49iKVOg7+PM/6+LtyISebNXw4ihOCVDjWw1t3/renpaM3bPesxXm/gn5MhpGRkFail6+/rwh+HrtJ7+lba1S6Hu70VZ0NiCbwZx6CmlRjVsSb9Z27D0VrHvBdbAvDij3tITMukWVV3nqrsxr4L4ey7EE5MUjoAHw9oSH1v50e8ulJZIcuApVwKWwZ8MzaFIXN28kr7mgxrWaVIY9l0+gbTNpxmxrCmNMhudT6JZBmwlJNMulIucuyFoieTrpST7DImSZJUjGTSlYrN2ZBY/jp87Z7rP1p1/KH2e78Ks8Hf76D39K0Mm7OL5Bzdvc6GxFLhjd/yxHf3MkkqajLpSsWmjpcTg5reu8LskwENC73PB1WYfTusCesmdqZ7Ay9WHb1uWr5k7yXq3vXwK79lklTUZO8F6bGISUpnxIK9WOnUaNQqXu9UiyyDwuErkUx4ug6dpv5LFQ97LoXF88PzzfGr6MTAWdtZObZjoY6TX4VZpzoVTOu9XYxd2tQqgUFvfH5xNSIBF1sLbC3uvP3zWyZJj4N8h0mPxa8HrjCiXXV6+nszZPbOPOujE9PZ+HYzTl6PYeWRIPwqOuXZ5lErzG5Lychi+f7LrHi9HQALdl5gfLc6HLsWZdomv2WS9DjIpCs9FsFRSfRqaCw8yC+hVvawQ6dR4+loRfw9xk141Aqz2yatOMLbPepib6UjLM440lnO8SDyWyZJj4u8pys9Ft6utgTciAUgMDQuz/qc/afu1WvxelQS/b7bluvftA2nc23j7+PC/ovG0cwOXY6g3l33ZOdvD6R6OQfa1ioHwIWwBM7eiOPZ2Ts5dyOO9/48mu8ySXpcZEtXeiyGtajCiAV7WbrvMmoh0KhVZBn0hdrHo1aY9WnkzRdrT9G4kis7A24xsEklhrWsYhozYuCs7Xw5uDFAvssk6XGQxRFSLkVVHKE3GFAJgRCCYXN38d1zTXG3L5tf32VxhJSTbOlKj0VcSgYv/LiXLL2BdrXKldmEK0l3k0lXeixcbC1ZP7GzucOQpBJHJl2pxHmY/roFERqTzCuL9qNRC3xd7Zg5vCk7A27xwxbjnGyBN+NYOaYjdbyc+PafMxy8HIG9pZbFr7bh+83n2Blwy7Td+W8GFnl8Utkgk65UZng6WrHhrc4IIRi37BABoXF08CtPB7/yKIpCt68341fRkSNXIrHQqlmVI/GP6erHmK5+XA5P4Nt/zpjxLKQnnUy60kM7GxLLW78dwUqr5plmlenX2Iehc3eRkWWgRjkHvh3ahP0Xw5mzNQCBIMtgoINfedYeu06H2uV5q0ddxiw9iFat4vzNeAY08eWlttVN+794K57Jfx4lPVPP0BZVGNqiCu/89h/nb8WhEuKBPRvulnMmC4vsPsK3nbgeTX0fZ4QQbD93k7jkDGM3tcY+PN+6mmm7Tadu0K1exUe4alJZJ/vpSg9tz/kwJjxdhzXjO/FMs0po1Sp+Hd2O9RM7k5aRRVBkIgA2Flp+fb0dXi62WGjUbHy7K7vPh5n209GvPOsndubvo9fJyjGr77cbzzDvxRasn9iZtceuYzAoXAyL5+/xnVg9Lvfth9SMrDx9et9acSRPzHsvhNHu841EJKRib6UzLd90KtSUTKMS03B3sGTV2I5sOBFCVGKaabsdAbfo6Fe+aC6gVCbJlq700AY3q8T0jWdYe+w6I9vXoKqHPeOXHyYyIY0bMclEJBiTVa3s2XQ97C2pmf2zhebO533tik6oVAIvFxvTbAwAVyMSeXXRfgCik9KJTUnnf62r8fqSg3i72vBOz3qm+dGsdJoCtXxb1/Bk1/vd+WDlMXacu0nX7ES7/2I4b/UwTqRpb6WjeVV3VCpBw0ouXI9KwtXOksjENCy1KuystI966aQyTCZd6aHZWGiY+sxTRMSnMun3/xjwlC/1vJ15s0ttRi8+gKkPeI45yvPrrBoYGoe3sw0h0ck421qYllfxsGfq4MY421qQqTegUQmerl+Rvo19eHvFEQJC40wlxqkZWQydsyvXfqt42PPt0DvTCWVk6dFp1ADYW2qx1Bp/DopKoryTtWld40quBN6Mo0V1Dy7cjOeF7NsLW8+E0jnHYDqS9DBk0pUe2qr/gvjr8DVS0rMY280Pfx8XZm46x5ErkYXaz+7AW/yw5Rz9G/uiUd9pAb/VvQ6vLd5Ppl7BxdaCuS+24NnZO8kyKNhbaanicWfq84K0dE8ERfPlulOohMDXzY7WNYxVaJtO3aBrvTvJtEu9Coxbdoi107fSpqanafLJzadDmfqMrFaTHo2sSJNyKe7pesYsPci7vepT3qn0zqorK9KknOSDNEmSpGIkby9IZvX9883NHYIkFSvZ0pUkSSpGsqUrFYnHVboL8PvBq8zeEsCcF1ugNxj4cOVxFAVa1fBgcu/6rD8ezJxtgQhgZPsa9H/KlyV7LrHmaBBpmXrGdvWjewMv1hwN4ou1p2hdw5Pvnmt6z+ONWXqQy+EJWGjUTOheh9Y1PHlu7i6S0rJQqwU/jWiJi60lx65F8eW6UxgMCh/0a4C7vRXPz99NT39vJjxd57FcC+nJJ5Ou9EQY182P+t7OZOoNbHirCwADZm0nMTUTf18XNr7VBYOi0Pe7bfR/ypdhLavwQptqJKdnMWzOLro38KJtzXI08Hbh++yxFu5n4cutcz3cW/xqG7RqFX8dvsbq/67zcrvqzN9+nhWj22GR3fUM4LOBjThcyN4bUtkiby9I9zXh18MERycB8NnfJzkVHMPy/Zfp9902un69idPBMbm2H7P0IDdjjdPfDJy1HTCW8w6YtZ2e325hxYErjxSPNrtLmd5gwNPBCiudmorONqhUArVKmIolbm+XlplF9XLGrmXOthao1Q/uRCAQvLJoHyMX7iM2Of2u/emp7mlPUGQSqRlZPD9/N68t3k9yetYjnZdUdsikK91X9/oV+ffUDQDOhsRQ39uZ/k/5smZ8J34c0Yofd5x/4D7yK+e9rSBT8txt1X9BtPr0H+ytdLn69S7ff4VOOUp0P119gvZf/Evzau6FOudPBjZkw1td6NPIm+83G1vFscnp9PhmCwt3XaCapwNRiWlcj0pi6ai2tKnpyfL9lwt1DKnskrcXpPtqU9OTJXsu0a5WOapnz7q77exNFu26gCp7Gp6cRI6as9upNb9yXhdbS6BgU/LcbcBTvvRr5MPLC/cREBpH7QqOnLsRy+bTN/hlVBvTdh/292dC9zr0/W4b/Rr7Fnj/jtbGMRm61qtoapk72Vjwz9td+PfUDX7ccZ4hzSvT0NcVC62a5lXdmbstsFDnIJVdMulK96XTqHGw1rFs32V6NPAC4Kcd51k9riOhMSm8/VvuQWXsrLSExafgYmvBtQjjgDd3l/NqcyTqgkyznlN6ph4LrRqVSmBrqcFSqyImKZ0pfx5j4chWppHEbm9nqVVjZ3nvsRJik9Ox0mlMJcFgnFXY1lLL0atReLvYYjAoGBQFjVqFvZUWC42aKh72hCekoigKAaFxeLnknYVYkvIjk670QF3rVeT9v47x6YCGgLH122fGNlpU98iz7eCmlRi77BC1KzjiZm9szd5dzrvg5Vam7Qvb0t18JpTFuy9iUBSaVXWnsrs90zacJjQ2mZEL9wGwcmwHvt14hqNXo8jUG3itYy0AdgXcYvq/ZwmOSuLNXw7yw/+aM3/7ebo38KJ+jlmE3/jlINFJ6eg0Kr4f3oyUjCyem7sbIcBCq+b74c3QqlUMeMqXPjO2YaVTM/+lloW/sFKZJMuApVyKuwy4INYfD2b2lgCmDW2SKzkWhSl/HuXzQY1MD+AexY0YY+If2MSXEe1qmJbLMmApJ5l0pVxKYtJ90smkK+Ukey9IkiQVI5l0JUmSipG8vSDlYqXThKVl6vM+IZMemqVWHZ6akeVp7jikkkEmXemRCCEqAv8Ah4DXFUUplaVZQog3gPeAvoqi5J18TZIKSN5ekB6aEKIBcBBYDowqrQkXQFGU2cCrwD9CiH7mjkd6csmWrvRQhBBPA79gbN3+Ze54iosQohGwDvgGmKXIPyCpkGTSlQpNCPEq8DEwQFGUA2YOp9gJIXww3lLZCYxTFEVv5pCkJ4hMulKBCSFUwFSgH9BdUZQyO8qLEMIBWAWkAEMURUk2c0jSE0Le05UKRAhhBfwOtACal+WEC6AoSjzQHYgCdgshypk5JOkJIZOu9EBCCDdgO6AHOiuKEm3mkEoERVEygBHAGuCgEMLPzCFJTwCZdKX7EkJUx9hDYScwTFGUNDOHVKIoRl8AU4CdQojCjVMplTky6Ur3JIRoDewBpiqKMkVRFIO5YyqpFEX5FRgE/CqEeNHc8Ugll3yQJuVLCDEEmAU8pyjKFnPH86QQQtQANgIrgA9llzLpbjLpSrkI4xiH7wKjgJ6Kopwxc0hPHCGEO8a+vJeBEYqipJs5JKkEkbcXJBMhhBZYgPFrcnOZcB+OoigRQHvAEtgihCjaQYClJ5pMuhJg6nf6D+AJtFEU5aaZQ3qiKYqSCgwGjgAHhBCVzRySVELIpCshhPAC9gKXMA7okmTmkEoFRVEMiqK8DXwP7BdCNDN3TJL5yaRbxgkhGmLsErYEeKM0D1pjLoqizAVeBtYLIQaYOx7JvOSDtDJMCNEDWIxxhLDV5o6ntBNC+APrge+AGbJnQ9kkk24ZJYQYDbwP9FcU5dCDtpeKRvatnH+AfcAY+c2i7JFJt4zJHrRmGtAT46A1V80cUpkjhLAH/gKygGfkPfSyRd7TLQOEEBWEEI7Zg9b8CTQGWsiEax6KoiRg/NC7CewRQpQXQmizS66lUk4m3bLhN+BpYAeQBnRVFCXGvCGVbYqiZAKvYGzxHgRaAfuEEJZmDUx67OTthVJOCFEX2AYkAb8CH8kHOCWLEOJZjN3KgoDZiqIsNW9E0uMkk24pJ4RYA/QAjgEVgZcURdlq3qik24QQ5YGjwFWMt31CFUWpYt6opMdJJt1STghxGbiOcQDyHcBV2dItWYQQnkA7oC/QFSifXdEmlUIy6UqSJBUj+SBNkiSpGGnMHUBRsNKpw9IyDR7mjqO0sNSqwlMz9J7mjqM0U+sswwyZ6fI9ew8qrUW4PiOtVL4HS8XtBSGEEv7DIHOHUWp4vPkXiqIIc8dRmgkhlHZL5EBu97LrhfKl9j0oby9IkiQVI5l0JUmSipFMupIkScWozCTdA5cimfzncQAuhSUw8ueDBEcnU3fKen49eA2AVxYfovqkvzl8Jcr0uhmbAugzcydvLjtClt6AoiiM/uUwfWft5Pmf9pORZeB4UAwtP9vEyv+uF2nMMzcH0vu7nUz+60SededuxNFzxg76fb+L0NgUAPp9v4t+s4z/LtxKIDY5g36zdtFn5k5GLTmE3vDk378vjeLOH+TisvcASL55iXNzXiU1MoQDYxtwa/cKAI5MbsOJqQM4MXUAmUmxAJz4si97Xq1KWozx3rCiKAT8+AYnpvbnzKwXMGRlkHD1BIffbU3YgVVFGvP19d9z4su+XFo2Jc+6pOBzHP+8Nye/GkhadCgAJ78aaIo/OfTiPc+pLCgzSbdFNTduxacSFJnEtI3neKdHHQA6+ZVjWPNKAHw+oAEj21UzvSY8PpUzIXGsHdeeBt7ObDl7i4DQeKx0av4e256GPs7sOh9GQ19n3uxc84ExGAwKaZn6AsUbkZDGmRuxrBvfHo1KcDI491AJ0zcFsHhkCz7v34DZW88DoBaCNWPbsWZsO2qUs8dCq2LBS81ZO649vq627L0QXqBjS8XLsWZzMmLDSI0IImjNt1Tq/zYAzvU7Uq7tUAB0Tp74T16F/+RVaG2dAKg9+ifcGvc07Sc5JAC1zgr/yauxr9yQmLO7sa/sj3ePNx4Yg2IwoM9IK1C8GfGRJF4/g/97fyPUGhKuncq1Pmjtd9QZ8zNVh35C8MY5xoUqlSl+mwrV73lOZUGZSboAk3rU4dUlh3C2saCqh12e9e72uccaCYlJoUY5ewBqlXfg6LVoPBysTC3GxLRMnKx1DzxuTHI6c7ZdoO+sXVyPSi5QrKeCY2le1Q2AVtXdORGUO+mmZGThZmeJX0VHrkUZRwZUgH6zdjF+xVHSMvVY6zS42lkAoFYJjBP9SiWRb7+3CZg3Gq2dM9blquZZn5kQxYkv+3Hlj8+43ePIwtE91zY6R08Ug/FDPSs1Aa2N4wOPm5kUQ/DGuZz8qj9pkcEFijXx2ikcaxhnHnKs1ZLEq7m/ienTU9DZu2Lr7UdqeJBxoQInpg7g/M8TTck9v3MqC0pFP92CquZhR1h8Kt3rVyjQ9j6uNhwLisZgUDh0JZKEtEycbHQkpmXS6vNNuNpZMKVX3Xu+PuBmPD/tvEhMcgbPNPFh9Zi2aNQqUjP0DJ23N9e2VTzs+PbZRqbf41MzsLPUAmBrqSE+NTPX9jnvFNz+edGI5jha65i3/QIrDl7jpTbGP96IhDT2XohgQrfaBTpvqfhZl6tKemwYlQa+m+/6BpNXo7Vx5NLy94k5tR2XBp3ybKO1dSQrNZEjk9ugtXel8sD37nm8pJBAbmxZQGZSDJ4tB1P/nZWo1Br0Gamcnv7cXbFVocYL00y/Z6UkoLEyNlrUlrZkpSTk3rliyPOz3xs/obVxJOTf+YTt+Y0KnV4s0DmVRmUq6S7bf5UhzSqxYPcl2tZ8cL90NztLuterwIAfdlOznD2uthbsCgzD28WGRSNaMH/HRdYcD2FAY+98X7//YgQXwhJ4o2MNutYtj0Zt/GJhpVOzZmy7+x7bwUrH9WhjCzYpLQsHK22u9SqR92fH7FZ317rl+XHnJQCy9AbG//ofXz3TELVKtnRLqlu7llOu9bOEblmIs1+bPOtvt1pdGnQmKfhsvgkq5uxurFy9qPPGAkI2/0TE4b/xaN4/3+PFnd9PcuhFvLuPxsW/Cyq1MRUYb0/c//6vxtqe1Ejj8wt9WhIaa/vcGwhVnp9N8ft34caWBQU+p9KozNxeSE7PYt2JEN7u7kcVNzv2Xogo0OtebFOVNWPbUdHZho5+5TAod5Kbk42OhJSMe752ZLtq/PZaa4Kjkxk4ezdf/3OW+JQMUjP0pgdet/+99fuxXK+t7+3EocvGB3r7L0Xg7+uca72VVkNUYjrnQuOo5GoLQFKasTV89Fo03i42AHy45hQDm/hQu7xDgc5XKn5ZaclE/Lce334TsfKsQmzAvlzrDVmZGDLTAUi4cgxL1/w/5DEY0GTfG9XaOOZtgeZQsfPL1Ju4nLTIYE5NG8y11dPITI5Hn5Fqerh1+9+FJZNyvdbWtx7xFw4DEHf+AHaV/XOtV1tYkZEQTVJIAFYevsZzTE3KEb9Xwc+pFCozLd152y8wok011CrB2K41eXXxIaYPaZxrm49Wn2Lj6VC2nbvFyHbVGPiUDy8uOEBiWibNqrjyVCUXMvUGVhy8Rr9Zu1CrBAtHNL/vcR2tdbzWsQajOlRnZ2A4sSkZ+LraPrCl625vSe0KDvT+bie1KzjQwNuZiIQ0/jpyndc71WBCt1q8sGA/WrWKH4Y3AWDg7D3o1CrsrbXMfb4pQVFJrDhwjcDQeJbuu8rojjXoXKfcI11Hqejd2DSfCp1eQqjU+PR6k4B5r1H9hW9N67NSEzj97VDUFtZYOJfHu+cYAAJ+fIO4wP2kRQbh23ciTnXacmvvb5yYOgChVuP3+k/3Pa7WxhGvp0dRsdurxJzZRVZyHFbuPg9s6Vo4umPjVYsTX/bFpmIt7CvVJz0ugvADK/HuPhqf3uM4+/2LCLWWWq/MAuDUN8+g0mjRWNlT69XZ9zynsqBMlwHfjE1hyLy9vNK+uqkHw8M4HhTDu38eZ1zXWgW+X1ySyTLgx+9BZcBpMTc5M+M5KnZ+2dSD4WEkXD3BxaWT8ek1FrdGTz/0fopbaS4DLtNJV8qfTLqPnxx74f5Kc9ItM7cXHtbZG3EE3oxnUBOffNd/tPoUn/SvX+j9Tv7rBOduxNGhtifjutbKtW7wnD2kZeixs9Tw00vNsbHQ8MriQ+wKDGPZq61oWsUVgFafb8LNztjN7eeXW+Bk8+Dua1LpkXj9LMk3zuPZcmC+6y//9glVh3xU6P1eWjaFpJBzONftgE+v3F/7Y87sImjtdDTWjtQeNQeNtT0JV45zdeVXoBioPHgKQq3l8grjcVPDr1Ft+BdPVCv7cZNJ9wHqVHSkTkXHe65/mIR7MjgGjUqwbnx7Riw6QERCWq4+wt8+2whvFxt+PXiNVUeDeb5lZT4f0IBf9l3JtR9PBytWvtm20MeXSgc7nzrY+dS55/qHSbgJ104h1Br83/ubs7NHkhEfic7BzbQ+ZNN86k/6k9izu7m5+1e8uo0iZNOP1JuwDJXWwrTd7fvCJ6YOwKl260LHUZrJpJtDTHI6IxYdxEqrRqNW8XrHGmQZDBy+EsWEbrXp9PVWqrjbcSk8gR+GN8GvgiMDf9hd6MR3PCiGVtWNHdubV3XjdEgsnfzuPOC63fNALQQGjLd/7i7cAIhMTKPPzJ008nXhgz51ZfFDKZaZFMO52a+g0lki1Bq8nn4NRa8n/tIRfHuP4+hHXbD2rELyzUvUeuV7bL1qc3LaYBpM+rNQx0m8chzHWi0BcKzRjMSg07jU7wgYix7UlraodVY41mpF+ME1pEYEYchI5cysF9DaOlPjxW9QW1gDkB4XgdrCGo2VbdFejCecTLo5/HrgGiPaVKVng4oMmbs3z/ropHQ2TuzIyeAYVv53Hb8Kjnm2uR6VzLhf/8u1rHlVNyb18DP9Hp+aSc1y2YUPFto8hQ9grDhbfuAqK167dyth7bj2OFrreG/lCbadC5M9E0qxW7tXUKHji7g91YPT04flWZ+ZGE3NDzaQeO0k4ftXYvvsh3m2SY0M5vzC8bmWOdZsRqV+b5t+z0pJwKaisaTdWPgQn2NdvCmBqi1tyEpNIDMhitTI6zT+ZAvhh9Zwc9eveHUdCUD0ya24NOj86Cdfysikm0NwdDK9/CsC4JfPLYXK7nboNCo8HayIT8mbKMFYxfbgwgctidl9apPSM/G1y9sSmPT7cd7u7of9XUUROZmKIeqU58yNWJl0S7HUyBDcnuoFgK23X571Vh5VUGl06BzL3bN/rpWbd4EKH7JSEwFj4YPO3TfHOgdTf1t9WjIaK3s01vbYV/ZHpbXAsUYzQv6db9o++uQWqv/v60KdZ1kgk24O3i42BITG4+tqS2BoHJ39ciexnF/e79XnoyAtXX8fZ9YcC6Zr3fIcuhxFb3+vXNvP33GR6p72962ay9QbMBgULLRqjl6Lpppn3rEkpNLDys3LWGzg7kNSSCDO2V/5TXK8OZV7vDsL0tK1q+xPxOG/cfXvQvyFw7g36W1ap7awRp+WhD4jjbjz+7Gr3AArj8qkx0WgKApJIYFYuhobLfr0FLJSE7FwKpUz7jwSmXRzGNaiEiMWHWTp/iuohUCjFmQZHvy6nArS0vX3ceaPw0H0/m4n7Wp54G5vaeol0aehF1+sP0NjXxd2BoYxsIkPw5pXylO40b6WJ8/O3YO1TkN5J2vGdnnwKGfSk6tc26Gcm/0KN3cuQ6jUqNRa9PqCjVh3W0FauvaVGxC27w9OfNkXpzrt0Dm45eolUbHrq5z6ehAaGwdqvToHlUaLR/P+nJzaD5XOitqj5gLGkmRnP/mQNz+yn24OeoOCSoAQgmHz9/Hd0Mb5PsAq7WQ/3cevsP10FYMehAohBKe/e56aL03P1augtJH9dMuIuJQMXliwnyyDQruaHmUy4UolU2ZyHGe/fwlFr8e5TptSnXBLO5l0c3CxtWD9+A7mDkOS8tDZudBwylpzhyEVgTIzypgkSVJJIFu6ReBhCiQKIjQ2hVcWH0KjEvi62jJzWGOEECzdd4W1J25gMCj8+UYbYpMzeHXxIbIMBlpUc2dyzzokpGby2pJDxKVm8tbTtWlfSz5FlowepmiiIBRFIfCnN0mPuYnG2h6/138iNmA/wf/MBiD5xnnqT/rjvlV0ZYFMuiWYp4MVG8a3RwjBuF//I+BmPA5WOgJvxrMqR5L/+3gIz7WszIDG3jwzZw9JaZksP3CV51pWpl1NT/63YL9MutJjl3OOtuvrvyfm7G5cG3TGpV57FEXh+Kfd8+1jXNaUqaR79kYcb/1+DCudmmea+NKvkRdD5+8jI0tPjXIOfPtsI/ZfimDOtgsIIMug0KG2J2uPh9ChlidvdfdjzLIjaDUqzt9KYEBjb9OUOAAXwxKY/NcJ0jP1DG1eiaHNK/HOH8c5fyselUqwZky7QsWbc6YHC40aTwcr/j0dSkqGnv7f76J5VTfe7u5HZTdbQmNTMGTP26NVqzgeFMPzLStjpVNja6EhJSMLa12Z+t/9xEu8fpaLv7yDWmeFR8tBeDTry+kZwzFkpWNToQY1XphGbOABQv6dCwgUQxbOdTsQeWQdznXb4dt3IoELxqHSaEkOPY9Hs/5U6PSiaf/JNy9xadkUDJlplGszhHJthnBx6WSSb5xHqNQ0eHdloeK93xxtiddOYudbX5aqU8aS7p4L4UzoVosudcqjKAqKAr+OaoWlVs0bS48QFGmstrGx0LDgpea89fsxLDRqNk7sSK/vdvBWd+OndMfa5fjmmUb0nbWL51tWNu3/238DmPe/prjZWfDs3L0829SXi+EJ/J1Pv92CzJMGsPdCBB+sPkklV1vsrbREJaaDgNVj2vHmsiOcvRFHfW8nPlt7mnk7LtLbvyIWWjWJaZnY3p5jzUJDQmqmTLpPmNiAvfj0Hodrg87GiRsVhbrjl6LWWRL40xhSI4IAY0mu3+gfubBkEiqtjoYfrOf4F33w7TsRAOd6Haj+v685+VV/yrUfbtp/0N/TqT1qDlp7V05PH4pnq2dIuXmJBpNX54mlIHOn3W+Otqjjm3Hx71qUl+eJVab+Cgc38WX6v+dYezyEke2qUdXdjvErjhKZmMaNmBQiEo2zlNbKntrGw96SmtmzAVto1Kb91K7ggEol8HKxJib5znQ9VyMSeXXxIQCik9OJTcngfy2r8PrSI3i72PBODz/TJ31B5kkDaF3DnV2Tu/DBqpPsCAjD3kpLi+xZgptVdeNyeCJrjgXzfu96dPLzZMSig4TGpmBnqSUpO/EmpWfdt5xYKpk8Ww4maO10Ig+vo0KXl7H2rMKFxRPJiI8kLeoGGfHGKadsKhgLY3QO7qafc474ZetVG6FSYeniRWbinVmlU8Ovcm7uKMA4oE5mcizl2w8n8Kc3sXL1xrf/26b3a0HmTrvfHG1x5/fj23dCEV2ZJ1uZSro2FhqmDm5IREIak/44xoDGPtTzcuLNzjUZ/cth8qsTye/rUODNeLydbQiJTsE5xxi2VdztmDrYH2cbCzL1BjQqwdP1ytO3kRdv/36MgNB405gOBWnpZmQZ0GmMHUzsrbRYatU0ruTC6qPBpjj6NfLiVEgMjjY6hBDZyTYLfx9n9l2KpF1ND5LS5a2FJ5Ha0prqw78kPS6CS0vfxb15f+x86uLd4w0CfnzjzrTlOd+j+Xx7T74RiKWrF2nRIWjt7sy1Z+1ZhWrDv0Br64whKxOh1uDasBvuTftwYck7JIcEmO7BFqSle6852lIjrmPhVB6VRo73DGUs6a46ep2/jlwnJSOLsV1q4e/jzMwtgRy5GlWo/ew+H84PW8/Tv7G3aYZfgLeers1rSw6TqTfgYmvB3P815dm5e8kyGLC30lHF4874CAVp6Z64HsOX68+gEgJfN1taV3dHpRL8eeQ6/WbtorK7LY18XXCxtWDcr/9hMEA1TztqlLPH08GSUUsO88PW80yUU68/kcIPriZ8/0r06Sn49ByDXeUGBK+fRfyl/x784hxizu4m+J85uDfra5r1F8CnzwQC5r+Oos9Ca+tMrVdnc+rboSiGLDTWDlh5VjFtW5CW7r3maIs6sRlXeWvBRJYBF9KYZUd4t2cdyjtZF8vxzEGWAT9+xTVdT+CCcVQaMAlL5/KP/VhFqTSXAcviCEmSpGJUpm4vFIXvs6c7l6QnQa2RM80dgnSXMtvSHfjD7se2798PBdHq802cCo7leFAMPWfsoMeMHUzdcBaA9Sdu0O3b7Tw9fbvpodjC3Zd4evp2uk/fzt4LxqfSBy9H0u3b7XT7djv/ng697zEjEtLwmbCKm7EpAPT7fhf9Zhn/XbiVQGxyBv1m7aLPzJ2MWnIIvUHhRkwKHb7awoxNAY/tWkiPz8lpgx/bvm/t/YMjk9uQGHSahKsnOP55b45/3ourq+49KHly6EWOf96LY5/2JHTH0lzrjn7YmbADxnvCQWtnsO+NslskIVu6j8m4rrWo7+1Ept7AhgnGQXQG/LCbxNRM/H2c2TihAwZFoe+sXfRv7E2XOuV5uW01ElMzeWnRAVrXcOfHnRdZMrIFjtY6hs3fx9P1KtzzeAt3X6Khj4vpd7UQrBxzp2otJSOLBS81x9XOgq82nGXvhXDa1fLkswENOHylcA8SpbLBu9dY7HzrYcjKpOH76wA4+fVgslIT0VjlHTT/1u5fqfLMh9hXbcyJL/pQocPzAESf3oHGxsm0nW+fCcRdOFQ8J1EClbqW7oQVRwmOTgbgs7WnORUcy/IDV+k3axddv9nG6ZDYXNuPWXbE1Dq83fq9GJbAgB9203PGDlYcvPZI8WizezfoDQqeDpZY6dRUdLZGpRKoVcLU2+f2ZJQatcrUda2ymx0JqZmkZGRha3Hvz8f4lAxikjPwcr7zcE8B+s3axfgVR0nL1GOt0+BqZ+y7aTxuqXxGUSpc+PktUiNDALjy5xckBp3m5u5fOTF1AMc+6U5i0Olc2wcuGEdajPGh3O3Wb/LNS5z8ejDHP+/NrT2/PVI8Ko2xj7di0GPh6IFKZ5XvdlYelchKTcSQmY7a4s424QdX496szyPFUJqUupZu9/oV+Pd0KK+2r87ZG3F80Kce1TzteK5FZYKikvhm4znmPN/0vvvIr7JMlV2SW5DpeO626mgw3248R7tanrm6mC0/cC3XLMAA320OYGjzSgB0q1eeZ+fuwaDAjCGN77n/n/de4X8tK7Ng1yXTskUjmuNorWPe9gusOHjNVK4ckZDG3gsRTJDdyEos10ZPE3V8E15dR5IUfJYqg6dgXa4q5dsOIzXiOkFrvqXWqz/cdx/5VZsJlfG9V5Bpe+4WfnA1QX/PwLlO21zdznJy8mvN6enPoRj0+PQaC0D8pSPG8l9VqUs1D63UXYk2NTxYsu8K7Wp6Ut3TWE227dwtFu2+jCp7Cp6ccrb4bneey6+yzMXW2EosyHQ8dxvQ2Jt+Db14+eeDBNyMp3Z5B87diGPz2Zv8MrKlabsdAWGEx6fxXq+6AHy14Syb3uqEtU7N8J/206F23kFrUjP0XAxLYHzXWrmWmyatrFueH3cak3GW3sD4X//jq2ca5hrXQSpZnPxac3PHLzjXaYtN+eoARJ/aTui2n0GoUGly/9nm+tKS/SbOr9pMZ2e8/VSQaXvu5tG8P+5N+3JuziskhQRi61UrzzbXVk+j7rglWHlU5vT0oXi0GMDNncup9vxUIv/bUKjjlWalLunqNCocrLQsO3CVHvWN90B/2nmJ1WPaERqbwtu/H8u1vZ2VlrD4NFxsLbgWaZwF9e7KMm2O1mlhW7rpmXostGpUKoGthQZLjYqY5HSmrDrBwpdamJJfUGQSc7ZfYPmrrUyvVauMFWY6jYqM7MnaIhPTcLaxML0uJCaZoKgknp27h8Cb8cSlZrL0lZamEuCj16JNty4+XHOKgU18qJ1d5iyVTCqNDo21Azd3LcetcXcAbmxZQIN3V5IWHcrFX97Jtb3ayp6MuHB0di6khl8F8lab3b5FAIVv6Roy01FpLRAqFWpLW1RaC/QZqRgy03MNaoNinDHYeCyBYtCTGnmdc7NHkhEXBkKFY/Wmpskry6pSl3TB2Lp7f+VJPu1XHzC2fvvM2mkasyCnwU18GLv8P2pXcMDNzjg9z92VZQteam7avrAt3c1nb7J4zxUMikKzqm5Udrdj2j/nCI1NZeTPBwFY+WZbpm8KICIhlaHz9uJia8HCEc15pV11+v+wC0WB51oYbzl8vOYUn/ZvYGp5V/e059+JxplhbxduAAycvQedWoW9tZa5zzclKCqJFQeuERgaz9J9VxndsYacsr0Ec/HvwuUVH1J1yMeAsfV74sv+ONZsnmdbz5YDOb9oArYVa6FzcAfyVpv5vf6jafvCtnSjTmwhdPsSUAw41GiGtWdlYs7tITk4AK+nR5m283r6Nc7NGQkKONdth8bSxjTbxa29fyDUmjKfcEFWpD0W60/cYPa280x7phH1vZ0e/IJCmPzXCaYO8i+Sfd2ISWHk4oMMbOzDiLZ3hqiUFWmPX3FVpD2siP82ELJxDtX/9zV2vvXyrA/Z/BOuDbth5eZd6H0HrZ1BxJH1NPli5z23Kc0VaTLpSnnIpPv4lfSka26lOemWui5jkiRJJZlMupIkScWoVNxesNKpw9IyDR7mjqO0sNSqwlMz9HJStcdIrbMMM2Smy/fsPai0FuH6jLRS+R4sFUm3uAghVMABYL6iKEuK6ZizARRFeaM4jic9uYQQLwIvA62UYvjDFkL4A/8CNRVFiXvcxystZNItBCHEc8BYoKmiKIZiOqYLEAh0UBTlbHEcU3ryCCHsgfNAH0VRCjfK+aMddwGQqCiKnIungGTSLSAhhC3G5PeMoigHivnYbwJ9gM7F0YKRnjxCiK8AD0VRXnzgxkV7XA/gHMbW9fniPPaTSibdAhJCfAZUURRlqBmOrQFOAe8pirK2uI8vlWxCiKrAIaCuoii3zHD8iRi/ifUo7mM/iWTSLQAhhC9wDKivKMoNM8XQGZgP1FYUJd0cMUglkxDib+Cgoij3Huz28R5fB5wFximKstEcMTxJZJexgvkGmGmuhAugKMpWst/Y5opBKnmyP4zrADPNFYOiKBnAeGBGdgKW7kO2dB9ACNEW+AWopShKqpljuf01so6iKGHmjEUyv+zbTieB9xVF+dvMsQhgI7BFUZTvzBlLSSeT7n0IIdQYbyt8qSjKn+aOB0AIMQ1wVRTlJXPHIpmXEOJ1oB8l5AGrEKIWsAfjLbBIc8dTUsmkex9CiFeA54C2JeFNDaauQReA3sXZNUgqWUpqV0IhxEzAUlGUUQ/atqySSfcehBCOGPs9Pq0oygkzh5OLEOIljJ3gW5aUDwOpeAkhfsD491uiimaEEE4Y/266Kopy0szhlEgy6d6DEGIGYKsoyivmjuVu2ZVxR4AZiqKsMHc8UvESQtQBdmB8zhBt7njuJoQYBTwLtJeNgrxk0s2HEKIGsA/wUxQlwtzx5EcI0RL4HWMJZrK545GKR/YDq63AWkVR7j9RmplkPws5DnymKMpKc8dT0sguY/mbAXxVUhMugKIo+zF+MEwydyxSseoNlMPYZ7tEUhRFj7Fr47dCiPynDi7DZEv3LkKI7hj7PNbJ7n9YYgkhvIETQENFUa6bOx7p8RJCWGAsuX0tu992iSaEWAUcVxTlC3PHUpLIpJuDEEILnAEmKoryj7njKQghxEcYu+g8Y+5YpMdLCDEJ48PTPuaOpSCEEJWB/4B6iqKEmjuekkIm3RyEEOOArkD3J+UBgBDCGmPXoecURdlr7nikx0MI4YmxIrG5oiiXzB1PQQkhvgC8FUUZbu5YSgqZdLMJIdyAAKCNoiiB5o6nMIQQzwDvAE9l30+TShkhxM9AlKIoT9Q9/OzR+S4AAxVFOWjueEoCmXSzCSHmA6mKoow3dyyFlf1Eew/wi6IoC80dj1S0hBCNgfVADUVREswdT2EJIYYDbwLNimsc6pJMJl1ACNEA2Iyx+1WsmcN5KEKIhhhr32soihJv7nikopH9gboPWKQoys/mjudhZPcrPwjMURRlqbnjMbcy32Us+009E/joSU24AIqiHAc2AB+YOxapSD0LWABLzBzHQ8tu3Y4Fpgoh7Mwdj7mV+ZauEGIA8CHGbldP9P3Q7FH8z2J8wn3R3PFIj0YIYYOxpHaIoij7zB3PoxJCLAVuKIrynrljMacynXSzO24HAC8pirLT3PEUBSHEWxgH6Oll7likRyOE+ASorijKEHPHUhSEEBWA0xgf+F41dzzmUtaT7hTAX1GUgeaOpajkGMV/jKIom8wdj/RwhBA+GEtp/RVFCTZ3PEVFCPEe0FhRlP7mjsVcylzSzb6H+yLGB2enMb4Brpk3qqIlhOgFTAOaYuxz/LuZQ5IKSAjRDggGpgIBiqJ8YtaAipgQwhJjv/IRgDPGQc+fuB4Zj6IsJl174CawBghWFGWKmUMqctkfLJuAoxj7R9Ywc0hSAQkhfgMuYmwY1FQUJcXMIRW57OcoHwFxwIeKouwya0DFrCz2XnAAUoCOQF0hRGl82v8XcAt4FXA0byhSITlgHDh/O/BfdnerUkMI0RNjIU8axnN1MG9Exa9U/Q8tIEfABbDBOK35NLNG83iMAnTZ/1zMHItUONWBCtn/7V8Kiwn+wThCWlWME2pWMG84xa8sJl1LIArjAMsflMbpzBVFiVIUZSjG5BubfbtBejKkAT9gLEe/YO5gippi9DNQF2N5cJnrt1vm7ulKkiSZU1ls6UqSJJmNTLqSJEnFSHO/lZZadVh6lsGjuIIp7Sw0qvC0TL2nlYU2LC0jS17Xh2Sp04Snpmd6FmRblc4yTMlML9PXWmgtwg0ZaQW6Xg+isbAK02eklenrqdZZhmelpz709bzvPV0hhHLrm6cfdt/SXcq9/S+KogghhBL796fmDueJ5dT3QxRFKdDDQSGE0nxR2Z604OCICgW+Xg8ihFBGrIssil09sRb1dnuk6ylvL0iSJBUjsyfds6EJ/HXs3i2Rj9c/3CQOU/4OoM/cQ8zafiXPuk/Wn8fv422svOu4EQnpVJq8mZtxqQ91THM6c/UWv+88ec/17//8cMMwTPrpH56evJDpf+3Os+7Zz5fTY8oi+nywmOgE4yzw3/yxi+7vLeK1WavJ0t8ZtO2fQ4E0em3mQ8VQEiQHnyXywL1nEw/64+Gqda/9+j5nv+rHjX++z7Mu7uwuznzZm8CZw8lKeXIrZaOvnuHSzj/vuf7wog8far8Hf3yXDe/25OSf3+VZ9++Hg9jwbk82fzqEzNQkAE78Pp0N7/Zi93evY9BnAbDh3Z78MtiH5KibDxXDwzB70q1TwZ5Bje7dP/rjXrUKvc+TIfGoVYK1o5txJjSeyMTcXXFHt6vEhz1q5nndov3X8fd2LPTxSoK6lcvxbPsG91z/+UvdCr3PE5dD0ahV/Dv1ZU5fvUVEXFKu9cveHcI/X4xgSAd/Vu45Q1hMIqev3WLjlyPwr1qBTf/d6Wa67mAAFVyf3OIjG+86uLW497hIvs98VOh9JgWdQqjV1Hl3DcnXz5ARn/tr+83NP1L7rT/waPscEXtWFHr/JYVL5bpUaz/4nuubjij8rbbISycRag09v9pA9NXTpMZG5Frf6vXp9PxqA77NenBl92pSYsKIvnqGnl+tx62aP8FHNgPQ4Z1FVGpRvAPyFWvSjUnOYMD8wwxbdJQXFh/j8LUYDlyJ5rttlwHoPHM/r/16kk4z9hFw0/jJPvjHI4U+zongOFpVdQageWVnTt/IPZGCm51FntfEp2YSk5yBl5NVoY9X3GISUuj1/s8M+nQZQ79cwcGA6+w7c41v/9wFQNsJ8xgx/S9aj5vL2aAwAPp+uKTQxzl68Qat61YCoKWfLycv524NaDVqANIzs6he0ZWQyDhqerkD4OfjzpHzIQDsOX2VZrW9UT0hNRqZSTGcmzaQwJnDOf/DiyRcOkL8+QPcWD8TgNOfdOXij6M59XFnkkMCAAj4tvCTMSddPYF9zZYA2NdoTvL106Z1+vQU1Ja2qHVWONRsSdK1E49+YsUgLSGGjVP6svmTZ9n6+XDCAg5x68x+TvwxHYC/x3Vg5zevsGZsO2KunQPg3w8GFPo4kRePUb5eawA867Qg6sqpXOvtPLwBEGo1CEFSxA2cvI1DkDj51iLi/FEArJ2K/5ngfXsvFLUVR0J4sYUPPet5MnThf3nWxyRlMOuN5pwMiWfl8Zt8WN4+zzbBMSmM++NMrmXNKzvzdtdqpt/j0zKpYWELgK2Fhvi0rAfGtvjAdZ5v7sXCvdcLe1rFbum2Y4zs3pTeLfwY+Gne2U+i4pPZ+vVIjl8O5Y9dp6jzQt4HrdfDY3n9+zW5lrWs48vkIR1Mv8cnp1Hb25hEba0siE9Oy7V9bGIKz3z+K0mp6fz14XB0GjVHL4ZgMBg4cO46CSnGbxjLtx/n+9f7sHb/uUc+9+IQsec3PDu8iEvjHgR+91ye9ZmJ0dSZsp6ka6eIPLgKG6/aebZJiwzmyuIJuZbZV2+GV9+3TL9npcRjVcGYCNQWNrluIWSlxKO2Mr6HVZY2ZKUmFsm5PW4Xti6nVo8RVGrRi80f5/0gSkuIptc3m4i6dILLu/6iSSW/PNskhl1nz/djci0rV6cFDYe+Y/o9IzkerY/x26rWypaM5Ly3X7LSU7iweRldP/odfWY6EReOoRgMhJ09RIYZb9cUa9INjkmlV71yAPjlk1Aru9mg06go52BBfGpmvvvwdrZm9WtN73scB0stSenGRJuUnoWvrfV9t0/N1HMxPIlxHasW5DTM7np4LH1bGN+sdSuVy7O+ankXdFoN5Z3tSbgrUd7m4+HEhi9euu9xHGwsTYkzKTUdX0/nXOud7KzZ8vVINh4OZN66g3z+Ujd6NK1N7w+WUMvHHTcHG05cDqVGRTcsddqHOVWzSI8KxuWpngBYe+dNCpYelVFpdOicPNHf44/X0s0bv0n3vgcMoLF2QJ+dTPXpyVjau961zng7x5CWjMbqyaiWTQoPplLL3gA4V6qTZ719+SqotTqsXcqRnpz/VH52nj70+HLtfY+js3EgM8V4fTJTk7Dy9M2zzf65b9NwyCR0NsZc49O8Oxvf74eTd02sHFzzbF9cijXpejtbce5mAj4u1gTeSqRTLbdc6wvy5bMgLV1/bwfWnLhFl9oeHLoWS+/6eRNTTiExqVyPTmHIgv84H5ZIfGomS15sVODzKm4+Hk6cDQrD19OZc0FhdGlUPdf6nEMt3KtLYEFauo2qVWTV3tM83aQmBwKu07fVnT8ig8GAQVHQqNXYW1tioTO+lV7u3oSXuzfh+zX7aFbLm8DrEew6dYWDAdc5cy2Mb//azVuD2j7yNXicLFy9SQ4JwNLNh5SQAJzqdcy1PvdQFvlf34K0dG0rNSDq8N84N+hCwoVDuDS+c29RbWGNPi0JQ2Ya8RcOYFvJ/9FPrBjYengTc+0c9p6+xFwPwOupzrnW57p293hvFqSl61a9IVf2rMa7SVfCzh6kcss+ubY/8/c8HL2qU8G/nWlZ7e4vUbv7S5xePRuP2vdvuD1OxZp0hzbxYuSyEyw7FIJaJdCqVegNhZuWrCAt3QZejvzxXyh95h6iXXVX3OwsOBuaQGBYIoMaVWDurqv8cTQUAUQlZTCqbSX+ebMFAGN/P8073ardd//m9nynRvxv2u8s2XwUlUqg1ajQ6ws3GFVBWroNq1VgxY4TPD15IR39q+LuaMuZq7c4dz2cHs1q8ezny1EJgYVWw5wx/QAY/tVvJKak07y2D01qetOkpjdDOxoTRt8Pl5T4hAvg3mYIF+e+Qvju5QihQqg1KPoH36LKqSAtXdtKDYjY/ydnv+qHo19bdA5uJAefJeXGedxaDKRcl1c4N20QGhtHqr0y+1FOqdjU6PwcO75+ifObfkGo1ajUWvT6wv2NF6Sl61bNn0vbf2PDuz2p4N8eKyd3oq+eIeZ6IJVb9eXo0s9xr9mYG8d3ULXdIGp0eY5tX75AZmoiHrWb4VHzKQB2TX+NW2f2khAWRMMhkyhfv81Dn3tBFWtxhN6goBLGT7vhPx9lxqC6+T7UKq2KqjhCrzegUgmEEDzz+XJ+eKMv7o62RRhpyfa4iyMUgx6ECiEEgbOep8oL09E5uD34hSVUcRZHGPR6hMp47bZ8OpTWb87Eysm9KA5dYjxqcUSxtnTjUjN5cclx9AYDbbNboFLhxSWnMmzqb+j1Bto3qFKmEm5xyEqO58Lsl1AMWThkt0ClgslIjmPbF89j0Oup4N+u1CXcolCsSdfFRse615sV5yFLJRd7GzZNfdncYZRaWjtn6kz+29xhPJEs7V3o+fU/5g6jRDN7cYQkSVJZUqwt3cIa/OMR/ny1SZHv93JEEhP+PINQCepXdODT3rXYcT6S2TuvAnA+LJE/X2mCh70Fry4/SZZBoUUVZ97tVp0fdlxh54Uo03YBn3Qq8vget74fLuHvT18o8v0qisKrM1cRGhWPg40VS94ejBCCV2asJDI+mS6NqzOmXys2Hg7k+7/3k5mlZ2gHf0Y8XfT/j80l4NtnqP3WH0W+3+TgswT9/jEAqeHXqDzsC5wbduPar++THHIOx7rtqdhjzP138oT594MBPP3Zqsey7w3v9iT66hkGzj2IjWt5oq+e4dDC9wFIuHmV5qO+xrdZ98dy7DLZ0nW20fHbyKdYO7oZMckZXIlMpkNNN1a/1pRVo5rg7WyNX3k7/j55i+eaebHu9WacDIknKS2LNztUYfVrTZk2wI921c3X168kOhsUjrVOyz9fjKBRtQrsOHmF9YcCaFLTiw1fvMTJKzeJik+mc6PqbJr6Mlu/HslvO56MSitzs/Gug9+klfhNWomVRyUcarV6YBmxdG93l/+6VK5Ljy/X0uPLtdiXr0z5eo+vF0ORtHTPhiYwadVZrHRqBjeqQF//cjy36BjpWQZqeNjyzcA6HLgSzZxd1xBAlkGhQ01X1p0Mo30NVyZ2qcbY30+j1ai4EJZIf//yvNjSx7T/i+FJTPk7gPRMPUOaVGRIEy/eXX2O82GJqFWCVaMK1+fO2UZn+lktRK7+wSdD4qlf0R4hBJVdrbkRl4bBYOzhodXc2XLzuQi6+j3eEsIzV28xft46rCy0DGnvz4A2dRn86TLSs/TU9HJn5uje7DtzjVlr9iGEsVdDp4bVWLP/LB39q/HOs+0ZPWs1Oo2agOAIBrWtx8jud67VhZBIJv20gbTMLIZ3asRznRoycf56AoMjUKsE6z+/f5eyu5VztkOffa0SUtJxsrMiMDichlWNY2vUqOjG8UuhdGls7FeckaWnagXzfHAlB5/l6tJ3UekscWsxCNemfQmcORwlKwOrCtWp8vw04s8f4OameYCxR4NT3fZEHVmHY532ePWZwOVF4xAaHSmh53Fr1h/PDi+Y9p9y8xLXfp2CkpWOe6shuLd+lqvLJpMSegGhUj2wO9m9ZMRHoNJZobayzbeMWHdXn+LiEH31DPvnvoXGwoqqHZ6hSpv+bP5kCIbMdBy9a9Lq9encOrOf06t/ACFQ9FlUbNiRq/vWUrFhBxoOeZs9M99ApdERG3yeKm0HULvHCNP+40IucuDHd9FnpFGj8zCqdx7G/nmTiL0eiEqtpvsXfxc65nuV/6bEhqOxtEZn/fgeThdJ0t17KYrxnarSubY7iqKgKLDspUZYatW8+dspgqKMI1DZWqj58Tl/3l55Fgu1ig1vNqf3nENM7GLsF9uxphvT+vvRb/5hhjfzMu1/xtbLzB1aH1dbHUMWHuWZxhW5FJHEmnz666Zm6hm28GiuZVXcbPhmYN7qmAthicSkZFDZzca0bFOOZFrfy4HPN17gxz3X6FWvHBbZYw0A7LwQyeIXGj7CVXuwXaev8NbgdnR7qkb2dVX444PnsNRpGTVzFdduxQBgZ2XBz28PZtzcdei0GrZ8/QrdJi/knWfbA9C5UXVmvNaLHlN+5sWujU37//qPnSyYOAg3BxsGfrKUoR0acPFGJP/k0383NT2TQZ8uy7WsagVXZo7ubfrdydaKhJQ0mr7xPa4ONnw0vBORcUnsPxdEm3qVORhwnSrljZMTz1m7n3nrD/FqT/N0Uo8P2EfFXmNxqt/ZWECiKNQatxSV1pJLC8eQFhEEgNrShuqj5nNl6SSERkfdKes5O7UvXn2MhQ9OdTtQefhXnJs2AI+2d0qGb6ybQbVX5qC1dyXwu2G4tRxM6q1L+L2T9+uyPiOV8zOH51pm6VmZKs/nnag69uRWnBoYCw7uV0ZcnG6e2kODZybi/VQX07Xs8uEKNDpLdn/3Ogm3rgHGct0Okxayb85E1FoLen/zLxve6UHDIW8DULFxJ1qO/pZ/3utNzW7/M+3/+O/f0H7ifCwd3dj88WCqdRxC/I2L9Ji6Lk8sWempbP7k2VzLHCpUpdXr0wt0LsFHtuD9VNeHvRQFUiRJd1DjCszYepm1p27xcitfqrrZMHHlGSIS07kRm0ZEYgYANTyMpYwe9hbU8DT+bKm5c4ejdjk7VCqBl5MVMcl3yoCvRCXz6vKTgHHQnNjUTJ5v5s0bv53G29mKSV2rmSpdrLTqBxZPACSmZfLe3wHMGVI/1/IDV6KZ2NlYDjx/9zWmdK9Bx5puvLz0BKFxqVRwtCIqKR0LrRo7y8db2jqkvT9f/7GTNfvOMqpXM6pWcGXs7LWExyUSEhFPeJyxhLRW9vgInk62pp8ttXf+1/r5eqBSqfB2dyI6IcW0/MrNaF76xjjkXnRiCrFJqbzY7SlenbkKH3cn3hva4c51tdA+sJhix8nLeHs48cs7zzJ33QFW7T1L/1Z12HHyCv0++gUXBxvcHIwfcK/3acnI7k3p+f5inu/UCAfb4h1oyK3lIELWzSDqyDrKdX4ZK48qXPnlLTLiI8mIvmH6qm6dndR0Dh5YVzDW+qu0d7o6WnvVQqhUWLhUJDMpxrQ8Lfwql358DYDMxBiykuPwaDecywvHYOHqhVfft03XVq2zKnDLN/bUVioP/wq4fxlxcara4RlO/PYNV/f+jV+vV3CoUJV9s8eRGhtBUuQN0whgTt7GEQOtnTxwyh43Qa2zNO3H2bc2QqXC1t2btIRo0/KE0Cvs+GYkYBxQJz0plprd/sfuGaOx8/Cm4bB3TddSY2H1wMKK+wn5bzMtXvv2oV9fEEWSdG0sNHzZz4+IhHTeWX2OAQ3LUbeCPW+0r8LrK06hZJdK5qwAzG/AqcBbiXg5WRESm4qzzZ2EVtXNhi/61sbZRkem3oBGJehWx50+DcoxadVZAm4lmsZyKEhLV1EUxv95hre7VMPT4c7/9OvRKZRzsESX/UFgUMDRSosQAjtLjWk8h60BEXSu9fj7btpY6vjmlZ6ExyYycf4GBrWtR70q5RjXfzCvfLfSVEWZ+7rmvbAB18PxdnckOCIWF/s741BUreDKNyN74GxvTWaWHo1aRY+mtejfqi7j563jbFA4dSsZB8spSEvXoCg4ZSdPJ1sr4pPTUKtVzBjVC0VReG3Wap6q6UV6ZhYWWg06rQYbSy06bfE/z1VZWFN52BdkxEdwddlk3Jr1w8a7LtW7v86lBW9yp7z3/m/alBvnsXDxIj36BlrbO2NTWHpWodKwz9HaOmPIykSoNTj7d8O1SR+uLn2HlBsB2HgZx3UoaEtXn55CVmoiOifj/5P7lREXJ62lDS1GfU1KbDgH5k2iStsBuFSpR/0BY9g1/TXT33+u+3j5XMvYoEDs3L1JigjG0t7FtNyhQlWav/oVlvZ3rqVPs+5Ubt2P/XPfIiboHC7Z4zw8Sks3My2ZjOQEbFyKZGajeyqSd/vq4zf561goKRl6xnSogr+3A7O2X+G/oLhC7Wf3xShm77xKP/9yaNR3WsATOldl9IpTZOkNONvomDO0PkMWHCXLoOBgpaFKjtsDBWnpHrgSw/7LMabW9Ee9alK/ogObzoXT1e9OZ+7/Nfdm3J+nMShQzd3W1FLfHBDBl33zjixV1P7ac5rfd54kJS2DCQPb0LBaRab/tZvDgcGF2s+Ok1eYuXovA1vXQ6O+c4vknWfaMXLGX2TqDbjYWfPThIEM+HgpWQYDDjaWVKtw541fkJZuhwZVWb7tOD2n/IxGrWLJpGcIiYzjtZmrUakEo3o1x9pCx48bDrH+YACZej2D2tTHyqL4B8OJOrSGyAMrMWSkUKH7m9hWbsCNDd+TeDnv6Hf3E3duN6EbZ+ParB9CfefPyav3BC799AaKPhOtrTNVR84m8LthKPos1Nb2WHlWMW1b0JZu3LndOPrdecCTXxmxOVzZvZJLO/4kKz2F+oPG4VatISf//I6IwMINyxp6cienVn1Plbb9UeW4lv5D3mbX9FcxZGVhae9Mu4nz2fTRYBR9FjobBxwq3BmoqqAt3fzKf0NP7Mo1VsPjUmLmSLs95kF5x5I/nu3DMsccaaNnrWbKsI5P9ADidyspc6RdXjQOr36TsHAu/1j2X1SehDnS9sx8g0bPvYeNa8m+liDnSJMkSXqilJjiiFnP1jN3CKXS3LH9zR1CqVV1xExzh1BqtBn3ZIyiVhSKvKX7MNPrFNQf/92g9Td7OHUjnsPXYujxwwF6zT7I/N3GLikHrkTT5Mtd9J93mM82nAfghx1X6D/vMP3nHab2R9sASMnQ8+Zvpxg4/zAztl665/Ge+ekIfeYe4rlFR0nOfog2avlJan64lcPX7jypvntZRpaBHj8cYOJfZ/Ld78N4mOl2CmrF9hM0feN703Q8izf/R58PFtNzys9kZunZd+Ya9V+ZQc8pP/PhEuPcUj9tOETnST/R5Z2f2HP66j33/dqs1XSbvJA+HyzmZvSdLk3hsYmUG/wpoVHGgaw/WLyJqs9/xR+77ky7Mvyr3x7red/Lw0y9U1AR+/7g5PttSQoyTs1zv4kpbwvdOJtz0wZybtpA/htjfPiWHhPKuWkDOTu1D8khxhk5ri6bzMkPOtxzP+byMNPxFNTF7b+xcnQLoi6fIvLicdZP6s76Sd05uvzLe77m1pn9rBzdIt+4tn7xvGlqofOblrLi+dpFPmlliWnpFtTYDlWoX9GBsPg01rzWDJ1GxYD5h3m+uXFOpCFNKjK+050b6292qMKbHapwOSKJGVuNc7H9tPcaL7TwppGP032P9e3AOng5W7PiSAirT9xkeDNvPu1di6WHbHJtd/cynUbF/GENmJnPTMQl1YSBbWlQtTwhkXEEXA9n7Wcv5lo/vFND3hrczvR7t6dq8ErPZiSkpPG/r3+nTb3K+e538rPt8fZwYvepKyzceJgPhxv7mP70z2EaVato2u7Nfq2o5Z27w/qyd4eYJek+bhV6jMHWt16uirILc0eSER+Z78OwCt3foEL3N0gNu0zIOuPMt6H/zsV3yCfoHD25uvRdary+gMrDpz7WD4ySqsGg8bhWrY8hK5Ne0zYCsPH9/mSkJKKzzjvjhkvluvSduYOtnw3LtTw2+AJZaXe6VNbs9jwR54u+EVnglu7Ev84QEmMM6IuNFzh1I55fD4fQf95hnv7+QJ7JH8f+fto0lfnt1u/F8CQG/XiE3rMP8tuRkEcK3DNH1y61EKiyb2uvPBZK37mH2HMxKtf2OSvITobE88fRUAbMP8zRoNh7HsPL2dq0/9tdsdzt8w5Hmd+ygho7Zy3B4cYYPl66hZOXb7J06zF6TvmZjm//yKkruT9lR89abWod3k5IF0Ii6fPBYrq+u4Dl244/dCwAO09eISUtk94fLOar33aYlv++6xTd31vErlPGDxJvD+MHllatvtcEALm2U2ePsQoQn5RKdEIK3u6Opu2KY3jKK0veJi3K+L67vvJLkoJOE75nBeemDeT05z1Iup77m8nlReNIjzFe/9vJLOXmJc59M5izU/sQsff3R4rnfhNT5ifmxBacG3QBIC0iCBsvP7R2LugzUu77usdl7+zxJIYbe9L898unRF0+xYUty/jnvT6sndglz2SRe2a+YWo13m5lxoVcZOP7/Vk/qTsXt/76SPGoNMZeMAa9HmtnTzQW+T+U19nYo8nRP/i2gA0LqfX0C48UQ0EUuKXbvY4H/54L55XWlTgTmsCU7jWo5m7LsKZeXI9O4Zstl5h9V6HB3fKrLFNlZ8uCTMOTn90Xo/BxscZSq6Z+RQd2vdWapPQshi48SquqLqb956wguxSexKttKlHDw5aRy07ct4w4JUPPr4dDWD6i8T23eRQ9mtbin8OBvNa7BWeu3uLj57tQ3cuV5zs3Iigshqm/7eTH8ff/epZfZZlKZfxAKsi0PDlFxiUhBKz77EVem7WaM1dv0aBqeQ5+/wZJqekM/HQZbepWMu3/279281yn+1fmGQwGZqzcw4zXjP1IF/x7hJe6PcX89QcLdI2KinPDbsSe2ES5ziNJDj6Lz8D3sCpXDY82Q0mLvE7I399SbeQP991HfpVmIvtaFGSKnpwKW1EWd3YnNd742fiLIcdMIYbCzRpSVHybdef6oY3U6TOK6KtneOp/H+LoVY0aXYaTEBbE8RXTaDdh7n33kV+12e3rWZBpe+52Zfcqjv82jYr+7XN1O3uQxIgQNJbWWNjd/9tvUShwVK2rufLLwWDaVneluoexVbI9MIJF+6+jEgKtOncPilxTIWX/N7/KMpfscRAKMg3P3W7GpfLDjiv8kj2fmY2F8XScrHVUcrUmJiUDV1uLPBVkjtZamlVyQgiBRnX/nh/vrDrLxC7VsLd6PH1J29WvzM+bjtDevyo1sqcv33L0Ij/9c9h4XTW5v4zkN/9ZfpVlLvbG2x0FmZYnJ3sbS1r6+QLQorYPl25GUbeycY45JztrKpdzITohBTdHW7Ydv0RYTCIfPHf/kda+WLGD/q3r4uvpTGp6JhdCIs0ybY9D7daE71qKg19brMsbP8zjTm/j1vbFxtkO7v4jzeda51dpprUzFkUUZIqenApTUZaZEIVKa3lngkpVjveFyjydkMrXb0vgv0uo4N8eRy/jh0fIf1s5t2EhQqVCpb7rbyaf65lftdntwoiCTNtztyptB1C5dT92fP0SMUEBOPsWrD/9uXU/4tf7VZLCC9cH/mEUOOnqNCrsrbQsPxRC97rGio0Fe4NYOaopoXGpvLMq9/TadpYawhPScbHVcTV77IW7K8u0OQogCtvSTc/SM+6PM3zV38+UbJPSsrC11JCRZSAoKgUna2NCv7uCzN/LkQvhSfi6WJOVPUBLVFI6TtY61DmS8Pzd16juYUvbxziamE6rwcHGil82H6Vnc+MbZN76g6z/7EVuRMUzYd76XNvbW1sQFpOIq4MNV7LHXri7skybY4yIwrZ0m9TwYuUe49fcgOvhDGhTl8TUdOysLMjIzOLarWic7ay5diuGH9bs4/f374w3EJuYgpWFNtfMv+sOnCM2KZUPslvDwRFxBIXFMPCTpQRcDycuOY0V7w192MtXKCqNDrWVPeG7l+PS0Dhs361tC6n99l+kR4dybdm7ubZXW9mREReO1s6FtAjjw9q7K81uf6WFwrd086so02ekomSmo7FxzLVt7KltuSbItHTzITkkAJ2jJ2rd/We7flzUWh06GwcubF6KT/MeAJxd9yPdv/ib5Mgb7J/3dq7tddb2pMSEYengQsJN48PXu6vNcl7PwrZ09ZnpqLUWCJUKrZUtap0FWemp6DPTsbB1vO+5JEfeYP+ciaQnxZGRHE9F//a4VX88Y6sU6kFa19rufLAukE96GWuoW1dzpd+8Q7So4pJn20GNKjD+zzPUKmeHe/a0PHdXlv00/M4Mp4Vt6a45cYuLEUlMyk72c4bWZ9eFSJYdMt6zG9na15RA764gG92uEuP+OENyRpbpodvH68/zSe9appZ3RpaBqf9eoJGPEzsvRDGwUXmGNvHi4/WB/Hs2nG2BkYxs7cOAhhXyXVYYTzepweSF//LFS90AaFe/Cj2m/EzLOr55tn22fQNe/2ENfj6eeDgZv3HcXVm2eNKdhymFbenWr1Ke33eepOeUn6la3oXG1b1Yvu04SzYbK7VG9WqOWq1i2p+7CI9NYtCny3B1sGHJpGeYs+4APZvWpkHVOx3cpyzeRDlne3pO+Zl2Darw1qC2bJ32CnCncAPg+zX7jMM8CkFUfBKv92lZqGtYUM4NunDtt4/wfeZjABxqtebc1/2xr9Eiz7ZuLQZx5efxWFeshdbB+C3k7kqz6q/9aNq+sC3d/CrK4gL2kBISQPmuo3JtG3NyC5WGfW76vcLTr3N50VgUfSaVhn5RmEtQpHyadOXQwik0HfGZMa4Gbflncm/K1c17Pat2GMzeWWNx8q1tmsbn7mqzDu8sMm1f2JZu8OFNBGz8GcVgwLNOcxzKVyH05G5irp2jbr/Rpu2ir53lyM8fEX31DJs+GkS3T/6i4+QlgLFnQ1jAoceWcKEEVaQVxIbTt5i98xpfD/CjfsWirbCa8ncAXxRRaW9GloGBPx6haSUnpnSvYVpujoq0glh74ByzVu9lxqjeuRJmYb27cCNTRzyd7/gPD2P4V79hZaHlp/EDcy0vKRVpDyP66AZC/51L5eFfYeubt2/6zS0/4ezfDUs370Lv++qyyaSFX80ziPqTUJH2sK7tX8fpVT/QcvS3uFbN+0zp7Nr5+DR9GjtPn3xefX/nNy3l3IYFPP3ZylxDQT5qRdoTlXSfdCU16T5pnuSkaw6lOemagywDliRJeoLIpCtJklSMZNKVJEkqRve9p2upVYelZxke70RgZYiFRhWelqn3tLLQhqVlZMnr+pAsdZrw1PTMAo00rdJZhimZ6WX6WgutRbghI61IRubWWFiF6TPSyvT1VOssw7PSUx/6et436UqSJElFS95ekCRJKkYy6UqSJBUjmXQlSZKKkUy6kiRJxUgmXUmSpGIkk64kSVIxkklXkiSpGMmkK0mSVIxk0pUkSSpGMulKkiQVI5l0JUmSipFMupIkScVIJl1JkqRiJJOuJElSMZJJV5IkqRjJpCtJklSMZNKVJEkqRjLpSpIkFSOZdCVJkoqRTLqSJEnFSCZdSZKkYiSTriRJUjGSSVeSJKkY/R9idGOStiJlnQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.tree import plot_tree\n",
    "\n",
    "plot_tree(pipe_dt[\"clf\"], filled=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "33dd961a",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = pipe_dt[\"vect\"].get_feature_names_out()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0638c6a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('残念', 'が', 'ます')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words[2159], words[198], words[518]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff5bd725",
   "metadata": {},
   "source": [
    "## ランダムフォレスト\n",
    "\n",
    "ランダムフォレストは決定木をバギングしたモデルです。\n",
    "`sklearn.ensemble.BaggingClassifier` を使うと次のように実装できます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "49aa38a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import BaggingClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1f3055cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "bagging = BaggingClassifier(\n",
    "    DecisionTreeClassifier(splitter=\"random\"),  # splitterはrandomに設定して、特徴量をランダムに探索する\n",
    "    n_estimators=1000,\n",
    "    random_state=0,\n",
    "    n_jobs=-1,  # 全てのCPUを使う\n",
    ")\n",
    "pipe_bagging = Pipeline([\n",
    "    (\"vect\", TfidfVectorizer(tokenizer=str.split)),\n",
    "    (\"clf\", bagging),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4101479e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                 TfidfVectorizer(tokenizer=&lt;method &#x27;split&#x27; of &#x27;str&#x27; objects&gt;)),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 BaggingClassifier(base_estimator=DecisionTreeClassifier(splitter=&#x27;random&#x27;),\n",
       "                                   n_estimators=1000, n_jobs=-1,\n",
       "                                   random_state=0))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                 TfidfVectorizer(tokenizer=&lt;method &#x27;split&#x27; of &#x27;str&#x27; objects&gt;)),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 BaggingClassifier(base_estimator=DecisionTreeClassifier(splitter=&#x27;random&#x27;),\n",
       "                                   n_estimators=1000, n_jobs=-1,\n",
       "                                   random_state=0))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(tokenizer=&lt;method &#x27;split&#x27; of &#x27;str&#x27; objects&gt;)</pre></div></div></div><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">clf: BaggingClassifier</label><div class=\"sk-toggleable__content\"><pre>BaggingClassifier(base_estimator=DecisionTreeClassifier(splitter=&#x27;random&#x27;),\n",
       "                  n_estimators=1000, n_jobs=-1, random_state=0)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">base_estimator: DecisionTreeClassifier</label><div class=\"sk-toggleable__content\"><pre>DecisionTreeClassifier(splitter=&#x27;random&#x27;)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-8\" type=\"checkbox\" ><label for=\"sk-estimator-id-8\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">DecisionTreeClassifier</label><div class=\"sk-toggleable__content\"><pre>DecisionTreeClassifier(splitter=&#x27;random&#x27;)</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('vect',\n",
       "                 TfidfVectorizer(tokenizer=<method 'split' of 'str' objects>)),\n",
       "                ('clf',\n",
       "                 BaggingClassifier(base_estimator=DecisionTreeClassifier(splitter='random'),\n",
       "                                   n_estimators=1000, n_jobs=-1,\n",
       "                                   random_state=0))])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_bagging.fit(train[\"tokens\"], train[\"label_num\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4bde2e58",
   "metadata": {},
   "outputs": [],
   "source": [
    "score_bagging = pipe_bagging.predict_proba(test[\"tokens\"])[:,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7022429d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.precision_recall_curve.PrecisionRecallDisplay at 0x7f0f7167f460>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAnUUlEQVR4nO3deXxU5dn/8c9FFAOKYAUpFTFosSAgUaKIqAURa2sBFxRcWlArj1Skj636wy4KqH1wty59FBWwLoC2WsFqbd2QTSWYsLqAihLlVwEVsex4PX+ck3QIk8zJcibMzPf9euWVOft1Jss193Lu29wdERHJXY0aOgAREWlYSgQiIjlOiUBEJMcpEYiI5DglAhGRHLdHQwdQUy1btvSCgoKGDkNEJKMsWLBgrbu3SrYt4xJBQUEBxcXFDR2GiEhGMbOPqtqmqiERkRynRCAikuOUCEREcpwSgYhIjlMiEBHJcbElAjObaGafmdmSKrabmd1lZivMbJGZHRVXLCIiUrU4SwSTgVOr2f5DoEP4NRz43xhjERGRKsT2HIG7v2ZmBdXsMhD4kwfjYL9uZi3MrI27r44jnrEzlrLs06/iOLWkwcDCAzmvR7uGDkMkKzVkG8GBwKqE5bJw3S7MbLiZFZtZ8Zo1a9ISnOw+lq3+imdKP2noMESyVkY8WezuE4AJAEVFRbWaSee6/p3rNSZJn8H3z2voEESyWkOWCD4BDkpYbhuuExGRNGrIRDAd+GnYe+hYYH1c7QMiIlK12KqGzGwK0BtoaWZlwHXAngDufh/wHPAjYAWwEbgwrlhERKRqcfYaOjfFdgcui+v6IiISjZ4sFhHJcUoEIiI5LiO6j0pue+PDz4HU3Uj10JlI7ahEIFlBD52J1J5KBLLbO/67LQF49Gc9qtxHD52J1J4Sgez2qksAIlJ3qhoSEclxSgQiIjlOiUBEJMcpEYiI5DglAhGRHKdEICKS45QIRERynBKBiEiOUyIQEclxSgQiIjlOiUBEJMdprCHJClUNVa2hqUVSU4lAspaGphaJRiUCyQrJhqrW0NQi0SgRSFbQUNUitaeqIRGRHKdEICKS41Q1JFkrWU8i9SIS2ZUSgeSMZau/AqiXRPD4Gx/v0iNJSUYylRKBZK3KPYlq24so2T/98tJGj/bfAuo3yYikmxKBZK3a9iSq/I+/8j/98teJJQB1VZVMpkQgOS3Kp/3K//RFsk2kRGBmBwC9gO8Am4AlQLG7fxNjbCL1KlnjcZRP+7U9d1WUVGR3U20iMLM+wGjgW0AJ8BmQD5wOHGpmfwZuc/evYo5TJBbp/rSvtgTZHaUqEfwIuMTdP668wcz2AH4M9AP+EkNsIvUq2TAU6T632hJkd2Tu3tAx1EhRUZEXFxc3dBgitVIw+m/AzlVRqiqSdDCzBe5elGxbrZ8sNrMLax+SiIBGSJXdQ116DY0FJtVXICK5oL6ebRCpT6kaixdVtQloXf/hiGQ3jZIqu6NUJYLWwA+ALyqtN2BuLBGJiEhapUoEzwL7uHtp5Q1m9mocAYmISHpVmwjc/eJqtp1X/+GISDIa5E7iFOsQE2Z2KvAHIA940N3HV9reDngYaBHuM9rdn4szJpHdnQa5k3SLLRGYWR5wL8EDZ2XAfDOb7u7LEnb7LfCEu/+vmR0OPAcUxBWTyO6mtsNeqLeR1Kc4SwTHACvc/QMAM5sKDAQSE4ED+4avmwOfxhiPSEaIMuxFVWMbqbpIaiPORHAgsCphuQyo3HduDPAPM7sc2Bs4OdmJzGw4MBygXTv9kkv2qM9hL1RdJLUVORGY2QR3H17Vci2dC0x299vMrCfwiJl1qTyqqbtPACZAMMREHa8pstuobQJIlkBUXSS1VZMSwf0pliv7BDgoYbltuC7RxcCpAO4+z8zygZYEo5yKSBWSJRDN0Sy1FXmsIXdfUN1yEvOBDmbW3swaA0OA6ZX2+RjoC2BmnQiGuF4TNSYRqZrGMZKoUg0xMYOgQTcpdx9QzbbtZjYSeIGga+hEd19qZuMIJrWZDvwKeMDMrgivM8wzbThUkd2ExjGS2kpVNXRrXU4ePhPwXKV11ya8XkYw85mI1FF9jWOU7DkGUDVTNkv1ZPHM8tdm1gRo5+7vxh6ViKRN5X/8yZ5jqNwjSckiu0Sds7g/QemgMdDezAqBcdVVDYnI7ifKU8vJnmMoGP033vjw84rqpijJQjJH1F5DYwgeEHsVwN1Lzax9TDGJSD2o7VPLUSQ7Rm0SmStqItjm7uvNLHGdGnVFMkxt/ulDvPM9S8OLmgiWmtl5QJ6ZdQBGofkIRHZr9fnPWwkgu0VNBJcDvwG2AFMIuoReH1dQIlJ3+uctUUVKBO6+EfiNmd0ULPqGeMMSEZF0ifRksZkdbWaLgUXAYjNbaGbd4w1NRETSIWrV0EPAz919FoCZHQ9MAo6IKzARySxRhsbW8we7p6iJYEd5EgBw99lmtj2mmEQkS7zx4ee88eHnFf/89fzB7inVWENHhS9nmtn9BA3FDgwmfKZARASS91IqGP23nfbR8we7p1QlgtsqLV+X8FrPEYhIhWS9lGrThVXVR+mXaqyhPukKRESyT5QEULltQdVH6VeTGcpOAzoTzBkAgLuPiyMoEcldqj5Kv6iDzt0HNAX6AA8Cg4A3Y4xLRHKEhq9oeFFLBMe5+xFmtsjdx5rZbcDzcQYmIrlBCaDhRZ2qclP4faOZfQfYBrSJJyQREUmnqCWCZ82sBXAL8BZBj6EH4wpKRETSJ+pYQ+UDzP3FzJ4F8t19fXxhiYhIuqR6oOzMarbh7k/Vf0giIpJOqUoE/avZ5oASgYhIhkv1QNmF6QpEREQaRtReQyIikqWUCEREclzkISZERDKNBrCLJuoQE02BXwHt3P2ScAL777n7s7FGJyISWrb6qxpPeqMB7KKJWiKYBCwAeobLnwBPAkoEIhK7gYUH7rIuyqQ3GsAumqiJ4FB3H2xm50Iwmb2ZWYxxiYhUOK9Hu10+wUeZ9CaZZFNq5npVUdREsNXMmhBORmNmhwJbYotKRCSF+hq1VFVF0RPBGODvwEFm9hjQCxgWU0wiIinVNgFUTiCqKoo+1tA/zGwBcCxgwC/cfW2skYmIxEDDXu8qaq+hGcDjwHR3/3e8IYmISDpFrRq6FRgMjDez+cBU4Fl33xxbZCIiDSRZV9RsblCOWjU0E5hpZnnAScAlwERg3xhjExGJXbJeRJW7omZ7g3JNJq9vQjAa6WDgKODhuIISEWlIlbuiZnuDctQ2gieAYwh6Dt0DzHT3b+IMTEQkHeqrG2omi1oieAg41913xBmMiEi6RUkAyaqPIHvaDVLNUHaSu78M7A0MrPwwsWYoE5FclU3tBqlKBN8HXib5TGUpZygzs1OBPwB5wIPuPj7JPucQPLDmwEJ3Py912CIi6ZOs+iib2g1SzVB2XfhynLt/mLjNzNpXd2zYw+heoB9QBsw3s+nuvixhnw7ANUAvd//CzA6oxT2IiMQq29sPok5M85ck6/6c4phjgBXu/oG7byV49mBgpX0uAe519y8A3P2ziPGIiEg9SdVG0BHoDDQ3szMTNu0L5Kc494HAqoTlMqByWj0svM4cguqjMe7+9yRxDAeGA7Rrl/n1cSKSHSrPkZBsKOxMaFBO1UbwPeDHQAt2bifYQPBpvj6u3wHoDbQFXjOzru7+ZeJO7j4BmABQVFTk9XBdEZE6STZHQmWZ0qCcqo3gGeAZM+vp7jVtGfkEOChhuW24LlEZ8Ia7bwM+NLP3CBLD/BpeS0QkrZLNkXDBg28AmTeyaaqqoavd/WbgvPJJaRK5+6hqDp8PdAgblT8BhgCVewT9FTgXmGRmLQmqij6IHr6IyO4jUxuVU1UNvR1+L67pid19u5mNBF4gqP+f6O5LzWwcUOzu08Ntp5jZMmAHcJW7r6vptUREpPZSVQ3NCL9XjCtkZo2Afdz9q1Qnd/fngOcqrbs24bUDvwy/RESkAUTqPmpmj5vZvma2N7AEWGZmV8UbmoiIpEPU5wgOD0sApwPPA+2Bn8QVlIiIpE/URLCnme1JkAimh7181I1TRCQLRE0E9wMrCQafe83MDgZSthGIiMjuL+oMZXcBdyWs+sjM+sQTkoiIpFPUxuLmZna7mRWHX7cRlA5ERCTDRa0amkgwrMQ54ddXwKS4ghIRkfSJOkPZoe5+VsLyWDMrjSEeERFJs6glgk1mdnz5gpn1AjbFE5KIiKRT1BLBpcCfzKx5uPwFMDSekEREJJ1SJgIzKwS+SzBo3CcAUYaXEBGRzFBt1ZCZXQs8AZwF/A0YrCQgIpJdUpUIBgOF7r7RzPYH/g48EH9YIiKSLqkai7e4+0aAcHjoqI3LIiKSIVKVCA4xs+nhawMOTVjG3QfEFpmIiKRFqkQwsNLyrXEFIiIiDSPVxDQz0xWIiIg0jFS9hmaYWf9wCOrK2w4xs3FmdlF84YmISNxSVQ1dQjCN5J1m9jmwBsgHCoD3gXvc/ZlYIxQRkVilqhr6/8DVwNVmVgC0IRha4r3y3kQiIpLZog4xgbuvJJicRkREsoieCxARyXFKBCIiOU6JQEQkx0VqIwjnHxgDHBweY4C7+yHxhSYiIukQtbH4IeAKYAGwI75wREQk3aImgvXu/nyskYiISIOImgheMbNbgKeALeUr3f2tWKISEZG0iZoIeoTfixLWOXBS/YYjIiLpFikRuHufuAMREZGGEan7qJk1N7Pbzaw4/LotYSJ7ERHJYFGfI5gIbADOCb++AibFFZSIiKRP1DaCQ939rITlsWZWGkM8IiKSZlFLBJvM7PjyhfABs03xhCQiIukUtUQwAng4bBcw4HNgWFxBiYhI+kTtNVQKdDOzfcPlr+IMSkRE0qfaRGBmF7j7o2b2y0rrAXD322OMTURE0iBViWDv8HuzuAMREZGGkWqqyvvD72Nrc3IzOxX4A5AHPOju46vY7yzgz8DR7l5cm2uJiEjtRH2g7GYz29fM9jSzl8xsjZldkOKYPOBe4IfA4cC5ZnZ4kv2aAb8A3qh5+CIiUldRu4+eEjYQ/5hg3uLvAlelOOYYYIW7f+DuW4GpwMAk+10P3ARsjhiLiIjUo6iJoLwK6TTgSXdfH+GYA4FVCctl4boKZnYUcJC7/626E5nZ8PLhLdasWRMxZBERiSJqInjWzN4BugMvmVkr6vgJ3swaAbcDv0q1r7tPcPcidy9q1apVXS4rIiKVREoE7j4aOA4ocvdtwL9JXs2T6BPgoITltuG6cs2ALsCrZrYSOBaYbmaJQ12LiEjMUj1HcJK7v2xmZyasS9zlqWoOnw90MLP2BAlgCHBe+caweqllwnlfBa5UryERkfRK9RzB94GXgf5JtjnVJAJ3325mI4EXCLqPTnT3pWY2Dih29+m1jFlEROpRqucIrgu/X1ibk7v7c8BzldZdW8W+vWtzDRERqZuozxH83sxaJCzvZ2Y3xBaViIikTdReQz909y/LF9z9C+BHsUQkIiJpFTUR5JnZXuULZtYE2Kua/UVEJENEnY/gMYLnB8qnp7wQeDiekEREJJ2izkdwk5ktBE4OV13v7i/EF5aIiKRL1BIBwNvAdnd/0cyamlkzd98QV2AiIpIeUXsNXUIwTPT94aoDgb/GFJOIiKRR1Mbiy4BewFcA7r4cOCCuoEREJH2iJoIt4VDSAJjZHgRPFouISIaLmghmmtmvgSZm1g94EpgRX1giIpIuURPB/wPWAIuB/yIYNuK3cQUlIiLpk7LXUDjl5FJ37wg8EH9IIiKSTilLBO6+A3jXzNqlIR4REUmzqM8R7AcsNbM3CSalAcDdB8QSlYiIpE3URPC7WKMQEZEGk2qGsnzgUuC7BA3FD7n79nQEJiIi6ZGqjeBhoIggCfwQuC32iEREJK1SVQ0d7u5dAczsIeDN+EMSEZF0SlUi2Fb+QlVCIiLZKVWJoJuZfRW+NoIni78KX7u77xtrdCIiErtUk9fnpSsQERFpGFGHmBARkSylRCAikuOUCEREcpwSgYhIjlMiEBHJcUoEIiI5TolARCTHKRGIiOQ4JQIRkRynRCAikuOUCEREcpwSgYhIjlMiEBHJcUoEIiI5TolARCTHKRGIiOQ4JQIRkRyXaqrKOjGzU4E/AHnAg+4+vtL2XwI/A7YDa4CL3P2jml5n27ZtlJWVsXnz5nqIWkSqkp+fT9u2bdlzzz0bOhSpR7ElAjPLA+4F+gFlwHwzm+7uyxJ2KwGK3H2jmY0AbgYG1/RaZWVlNGvWjIKCAsysPsIXkUrcnXXr1lFWVkb79u0bOhypR3FWDR0DrHD3D9x9KzAVGJi4g7u/4u4bw8XXgba1udDmzZvZf//9lQREYmRm7L///ip5Z6E4E8GBwKqE5bJwXVUuBp5PtsHMhptZsZkVr1mzJunBSgIi8dPfWXbaLRqLzewCoAi4Jdl2d5/g7kXuXtSqVav0BicikuXiTASfAAclLLcN1+3EzE4GfgMMcPctMcYTq7y8PAoLC+nSpQv9+/fnyy+/rJfzTp48mZEjR9bLuQoKCujatSuFhYUUFhYyd+7cejlvZaWlpTz33HNVbi8pKeHiiy8Ggvtr1aoVhYWFdO7cmUGDBrFx48Yqj62ta6+9lhdffLFWx27dupUTTzyR7du3J92+adMmvv/977Njx46KdXfeeSf5+fmsX7++Yt2rr75K8+bNKSwspFOnTowdO7ZW8ST6/PPP6devHx06dKBfv3588cUXSfe7+uqr6dy5M506dWLUqFG4Oxs3buS0006jY8eOdO7cmdGjR1fsf8899zBx4sQ6xyeZIc5EMB/oYGbtzawxMASYnriDmR0J3E+QBD6LMZbYNWnShNLSUpYsWcK3vvUt7r333oYOKalXXnmF0tJSSktLOe644yIdU9U/wKqkSgS///3vGTVqVMXy4MGDKS0tZenSpTRu3Jhp06bV6HpRjBs3jpNPPrlWxzZu3Ji+fftWGdfEiRM588wzycvLq1g3ZcoUjj76aJ566qmd9j3hhBMoLS2luLiYRx99lLfeeqtWMZUbP348ffv2Zfny5fTt25fx48fvss/cuXOZM2cOixYtYsmSJcyfP5+ZM2cCcOWVV/LOO+9QUlLCnDlzeP75oHb2oosu4u67765TbJI5Yus15O7bzWwk8AJB99GJ7r7UzMYBxe4+naAqaB/gybDu8WN3H1CX646dsZRln35Vx+h3dvh39uW6/p0j79+zZ08WLVoEwJtvvskvfvELNm/eTJMmTZg0aRLf+973mDx5MtOnT2fjxo28//77nHHGGdx8880ATJo0if/5n/+hRYsWdOvWjb322guAlStXctFFF7F27VpatWrFpEmTaNeuHcOGDaNJkyaUlJTw2WefMXHiRP70pz8xb948evToweTJk6uMtbpz5ufnU1JSQq9evbjsssu47LLLWLNmDU2bNuWBBx6gY8eOPPnkk4wdO5a8vDyaN2/Oiy++yLXXXsumTZuYPXs211xzDYMH/6cj2IYNG1i0aBHdunXbJZbt27fz73//m/322w+AGTNmcMMNN7B161b2339/HnvsMVq3bs2aNWs477zz+PTTT+nZsyf//Oc/WbBgAS1btuT666/n0UcfpVWrVhx00EF0796dK6+8kmHDhvHjH/+YQYMGUVBQwNChQ5kxYwbbtm3jySefpGPHjtWe9/TTT+eaa67h/PPP3yXuxx57jMcff7xi+f333+frr7/mj3/8IzfeeCMXXnjhLsfsvffedO/enRUrVnDUUUdF+K1K7plnnuHVV18FYOjQofTu3Zubbrppp33MjM2bN7N161bcnW3bttG6dWuaNm1Knz59gCDZHXXUUZSVlQHQtGlTCgoKePPNNznmmGNqHZ9khljbCNz9OXc/zN0Pdfcbw3XXhkkAdz/Z3Vu7e2H4VacksDvYsWMHL730EgMGBLfSsWNHZs2aRUlJCePGjePXv/51xb6lpaVMmzaNxYsXM23aNFatWsXq1au57rrrmDNnDrNnz2bZsv/0tr388ssZOnQoixYt4vzzz9/pU/UXX3zBvHnzuOOOOxgwYABXXHEFS5cuZfHixZSWllbs16dPHwoLC+nRo0fKc5aVlTF37lxuv/12hg8fzt13382CBQu49dZb+fnPfw4En7RfeOEFFi5cyPTp02ncuDHjxo2r+JSfmAQAiouL6dKly07rpk2bRmFhIQceeCCff/45/fv3B+D444/n9ddfp6SkhCFDhlQkyrFjx3LSSSexdOlSBg0axMcffwzA/Pnz+ctf/sLChQt5/vnnKS4urvLn1LJlS9566y1GjBjBrbfeWu15Abp06cL8+fN3Oc/WrVv54IMPKCgoqFg3depUhgwZwgknnMC7777Lv/71r12OW7duHa+//jqdO+/8AWPDhg0VVXeVvxJ/F8r961//ok2bNgB8+9vfTnqtnj170qdPH9q0aUObNm34wQ9+QKdOnXba58svv2TGjBn07du3Yl1RURGzZs1K9vZJlon1gbKGUJNP7vVp06ZNFBYW8sknn9CpUyf69esHwPr16xk6dCjLly/HzNi2bVvFMX379qV58+YAHH744Xz00UesXbuW3r17U94oPnjwYN577z0A5s2bV1HV8JOf/ISrr7664lz9+/fHzOjatSutW7ema9euAHTu3JmVK1dSWFgIBFVDLVu2rDiuunOeffbZ5OXl8fXXXzN37lzOPvvsim1btgTNOb169WLYsGGcc845nHnmmSnfp9WrV1O5wX/w4MHcc889uDuXXXYZt9xyC6NHj6asrIzBgwezevVqtm7dWtF3ffbs2Tz99NMAnHrqqRUliDlz5jBw4EDy8/PJz8+vSCjJlMfavXv3ivuv6rwQtAE1btyYDRs20KxZs4r1a9eupUWLFjude8qUKTz99NM0atSIs846iyeffLKinWfWrFkceeSRNGrUiNGjR++SCJo1a7ZT4q4JM0vaq2fFihW8/fbbFZ/2+/Xrx6xZszjhhBOAoCR27rnnMmrUKA455JCK4w444ADeeeedWsUimWW36DWUDcrbCD766CPcvaKN4He/+x19+vRhyZIlzJgxY6c+2OVVPhD8o6lpXXyi8nM1atRop/M2atSo1ufde++9Afjmm29o0aJFRdtCaWkpb7/9NgD33XcfN9xwA6tWraJ79+6sW7eu2nM2adKkyn7oZkb//v157bXXgKC0MnLkSBYvXsz9999fr/3Xy9+jmrzvW7ZsIT8/f6d1le9n8eLFLF++nH79+lFQUMDUqVOZMmVKxfYTTjiBkpISFixYwKWXXrrLNWpaImjdujWrV68GgiR7wAEH7LLP008/zbHHHss+++zDPvvsww9/+EPmzZtXsX348OF06NCB//7v/97puPLqTMl+SgT1rGnTptx1113cdtttbN++nfXr13PggcHjE9XV1Zfr0aMHM2fOZN26dRX11+WOO+44pk6dCgT10uWf6Ooiyjn33Xdf2rdvXxGLu7Nw4UIgqA/v0aMH48aNo1WrVqxatYpmzZqxYcOGpNfr1KkTK1asqDKe2bNnc+ihhwLs9N49/PDDFfv06tWLJ554AoB//OMfFT1levXqVZFsv/76a5599tkavRdVnReCqpyWLVvuMrTCfvvtx44dOyqSwZQpUxgzZgwrV65k5cqVfPrpp3z66ad89FG0kVPKSwTJvg4//PBd9h8wYEDFe/Pwww8zcODAXfZp164dM2fOZPv27Wzbto2ZM2dWVA399re/Zf369dx55527HPfee+/tUo0n2SnrqoZ2B0ceeSRHHHEEU6ZM4eqrr2bo0KHccMMNnHbaaSmPbdOmDWPGjKFnz560aNGiokoH4O677+bCCy/klltuqWjYrauo53zssccYMWIEN9xwA9u2bWPIkCF069aNq666iuXLl+Pu9O3bl27dutGuXTvGjx9PYWHhLo3FHTt2ZP369TtVsUybNo3Zs2fzzTff0LZt24qEOWbMGM4++2z2228/TjrpJD788EMArrvuOs4991weeeQRevbsybe//W2aNWvG0UcfzYABAzjiiCMqqsfKq96iqOq8EFSpVfXzO+WUU5g9ezYnn3wyU6dO3aXH1BlnnMHUqVMr2mXq0+jRoznnnHN46KGHOPjggysSWXFxMffddx8PPvgggwYN4uWXX6Zr166YGaeeeir9+/enrKyMG2+8kY4dO1Y0WI8cOZKf/exnQFDVNmbMmHqPOdcsW/0Vg++fl3rHCGracSUqc/d6P2mcioqKvHIj4Ntvv71L45fsvu644w6aNWtW8Q+nprZs2UJeXh577LEH8+bNY8SIERX16l9//TX77LMPGzdu5MQTT2TChAmRe+VUd94zzzyT8ePHc9hhh+1y3FtvvcUdd9zBI488Uqv72R2VlJRw++23J70n/b1F9/gbH/NM6S6PT9VaXRKBmS1w96Jk21QikLQbMWLETlVeNfXxxx9zzjnn8M0339C4cWMeeOCBim3Dhw9n2bJlbN68maFDh9aoa2ZV5926dSunn3560iQAcNRRR9GnTx927Nix07MEmWzt2rVcf/31DR1GxjuvRzvO69GuocNISSUCEakR/b1lpupKBFnTWJxpCU0kE+nvLDtlRSLIz89n3bp1+iUViVH5fASVu9BK5suKNoK2bdtSVlZGVUNUi0j9KJ+hTLJLViSCPffcUzMmiYjUUlZUDYmISO0pEYiI5DglAhGRHJdxzxGY2Rog2sAtu2oJrK3HcDKB7jk36J5zQ13u+WB3TzrXb8Ylgrows+KqHqjIVrrn3KB7zg1x3bOqhkREcpwSgYhIjsu1RDChoQNoALrn3KB7zg2x3HNOtRGIiMiucq1EICIilSgRiIjkuKxMBGZ2qpm9a2YrzGx0ku17mdm0cPsbZlbQAGHWqwj3/EszW2Zmi8zsJTM7uCHirE+p7jlhv7PMzM0s47saRrlnMzsn/FkvNbPH0x1jfYvwu93OzF4xs5Lw9/tHDRFnfTGziWb2mZktqWK7mdld4fuxyMyiz75UFXfPqi8gD3gfOARoDCwEDq+0z8+B+8LXQ4BpDR13Gu65D9A0fD0iF+453K8Z8BrwOlDU0HGn4efcASgB9guXD2jouNNwzxOAEeHrw4GVDR13He/5ROAoYEkV238EPA8YcCzwRl2vmY0lgmOAFe7+gbtvBaYCAyvtMxB4OHz9Z6CvmVkaY6xvKe/Z3V9x943h4utApo8lHOXnDHA9cBOwOZ3BxSTKPV8C3OvuXwC4+2dpjrG+RblnB/YNXzcHPk1jfPXO3V8DPq9ml4HAnzzwOtDCzNrU5ZrZmAgOBFYlLJeF65Lu4+7bgfXA/mmJLh5R7jnRxQSfKDJZynsOi8wHufvf0hlYjKL8nA8DDjOzOWb2upmdmrbo4hHlnscAF5hZGfAccHl6QmswNf17Tykr5iOQ6MzsAqAI+H5DxxInM2sE3A4Ma+BQ0m0Pguqh3gSlvtfMrKu7f9mQQcXsXGCyu99mZj2BR8ysi7t/09CBZYpsLBF8AhyUsNw2XJd0HzPbg6A4uS4t0cUjyj1jZicDvwEGuPuWNMUWl1T33AzoArxqZisJ6lKnZ3iDcZSfcxkw3d23ufuHwHsEiSFTRbnni4EnANx9HpBPMDhbtor0914T2ZgI5gMdzKy9mTUmaAyeXmmf6cDQ8PUg4GUPW2EyVMp7NrMjgfsJkkCm1xtDint29/Xu3tLdC9y9gKBdZIC7FzdMuPUiyu/2XwlKA5hZS4Kqog/SGGN9i3LPHwN9AcysE0EiyOZ5a6cDPw17Dx0LrHf31XU5YdZVDbn7djMbCbxA0ONgorsvNbNxQLG7TwceIig+riBolBnScBHXXcR7vgXYB3gybBf/2N0HNFjQdRTxnrNKxHt+ATjFzJYBO4Cr3D1jS7sR7/lXwANmdgVBw/GwTP5gZ2ZTCJJ5y7Dd4zpgTwB3v4+gHeRHwApgI3Bhna+Zwe+XiIjUg2ysGhIRkRpQIhARyXFKBCIiOU6JQEQkxykRiIjkOCUCiZ2Z7TCzUjNbYmYzzKxFPZ9/ZdhnHjP7uop9mpjZTDPLM7MCM9sUxrTMzO4Ln0SuyTWLzOyu8HVvMzsuYdulZvbTutxTeJ4xZnZlin0mm9mgGpyzoKpRLSvtd6OZrar8fprZSDO7KOr1JDMoEUg6bHL3QnfvQvDcxmUNEMNFwFPuviNcft/dC4EjCEasPL0mJ3P3YncfFS72Bo5L2Hafu/+prgE3sBkEA75VNpHsH8sn5ygRSLrNIxwgy8wONbO/m9kCM5tlZh3D9a3N7GkzWxh+HReu/2u471IzG17D654PPFN5ZTjo4Fzgu+Gn5ZftP3M2tAuve3ZYmlloZq+F63qb2bMWzGVxKXBFWMI4ofyTvJl1NLM3y68Vnn9x+Lp7WEJZYGYvWIrRI83sEjObH8bwFzNrmrD5ZDMrNrP3zOzH4f55ZnZLeMwiM/uvmrxZ7v56sqdVwxFsV5pZsiQhGUqJQNLGzPIIhgIof+p3AnC5u3cHrgT+GK6/C5jp7t0IxmVfGq6/KNy3CBhlZpFGjA2HJjjE3Vcm2dY0jGkxcDfwsLsfATwWxgFwLfCDMJ6dnsYOz3kfcEdY6pmVsO0doLGZtQ9XDQammdme4bUGhfczEbgxxW085e5HhzG8TTC+TrkCgk/vpwH3mVl+uH29ux8NHA1ckhBH+b1/x8yeS3HdZIqBE2pxnOymsm6ICdktNTGzUoKSwNvAP81sH4LqlPIhLwD2Cr+fBPwUIKzKWR+uH2VmZ4SvDyIYTC3K8AktgS8rrTs0jMmBZ9z9eTN7BDgz3P4IcHP4eg4w2cyeAJ6KcL1ETxAkgPHh98HA9wgGxPtneO95QKqxYrqY2Q1AC4KhQl5IvEY40uZyM/sA6AicAhyR0H7QnOD9eq/8IHf/lGCogpr6LLyGZAklAkmHTe5eGH76foGgjWAy8GVYT5+SmfUGTgZ6uvtGM3uVYHCxSNdPsu/7Ua/t7peaWQ+CT9wLzKx7xOsCTCNIdk8Fp/LlZtYVWOruPWtwnsnA6e6+0MyGEQ4sVx5i5ZAJZq+63N0TEwZWP9Oy5hO8p5IlVDUkaRPWL48iGCRsI/ChmZ0NFfOwdgt3fYlgOs3yuu7mBJ9ovwiTQEeCYaWjXvcLIC+sMqnOXP4zAOH5wKwwhkPd/Q13v5ZgVMuDKh23gWDY62TXfp9g8LffESQFgHeBVhaMnY+Z7WlmnVPE1gxYHVYrnV9p29lm1sjMDiWY0vFdgoQ7ItwfMzvMzPZOcY2oDgNS9jySzKFEIGnl7iXAIoLJRM4HLjazhQTtAOVTEP4C6BM2rC4g6NXzd2APM3uboJrl9Rpe+h/A8Sn2uRy40MwWAT8J4wC4xcwWh90u5xLMm5toBnBGeWNxkvNOAy7gP2PmbyUY/vym8N5LSeh1VIXfAW8QVFO9U2nbx8CbBLPOXerum4EHgWXAW2Hc91OpBqC6NgIzu9mCkS+bmlmZmY1J2NwL+GeKeCWDaPRRyQkWTFt5hbv/pKFjyWQWzGvxS72P2UUlAskJ7v4W8ErYc0lqryVB6USyiEoEIiI5TiUCEZEcp0QgIpLjlAhERHKcEoGISI5TIhARyXH/B7fjGVBlalWBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "PrecisionRecallDisplay.from_predictions(\n",
    "    y_true=test[\"label_num\"],\n",
    "    y_pred=score_bagging,\n",
    "    name=\"RandomForest (Bagging)\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6fc52c3",
   "metadata": {},
   "source": [
    "BuggingClassifier を使って実装しなくても、scikit-learn は\n",
    "[sklearn.ensembleRandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)\n",
    "を提供しています。\n",
    "ランダムフォレストを使う場合は、こちらを使う方がいいでしょう。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4838d9c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "84cde814",
   "metadata": {},
   "outputs": [],
   "source": [
    "random_forest = RandomForestClassifier(\n",
    "    n_estimators=1000,\n",
    "    random_state=0,\n",
    "    n_jobs=-1,\n",
    ")\n",
    "pipe_rf = Pipeline([\n",
    "    (\"vect\", TfidfVectorizer(tokenizer=str.split)),\n",
    "    (\"clf\", random_forest),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "2257ae6b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-3 {color: black;background-color: white;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                 TfidfVectorizer(tokenizer=&lt;method &#x27;split&#x27; of &#x27;str&#x27; objects&gt;)),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 RandomForestClassifier(n_estimators=1000, n_jobs=-1,\n",
       "                                        random_state=0))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-9\" type=\"checkbox\" ><label for=\"sk-estimator-id-9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                 TfidfVectorizer(tokenizer=&lt;method &#x27;split&#x27; of &#x27;str&#x27; objects&gt;)),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 RandomForestClassifier(n_estimators=1000, n_jobs=-1,\n",
       "                                        random_state=0))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-10\" type=\"checkbox\" ><label for=\"sk-estimator-id-10\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(tokenizer=&lt;method &#x27;split&#x27; of &#x27;str&#x27; objects&gt;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-11\" type=\"checkbox\" ><label for=\"sk-estimator-id-11\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(n_estimators=1000, n_jobs=-1, random_state=0)</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('vect',\n",
       "                 TfidfVectorizer(tokenizer=<method 'split' of 'str' objects>)),\n",
       "                ('clf',\n",
       "                 RandomForestClassifier(n_estimators=1000, n_jobs=-1,\n",
       "                                        random_state=0))])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_rf.fit(train[\"tokens\"], train[\"label_num\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "05596c23",
   "metadata": {},
   "outputs": [],
   "source": [
    "score_rf = pipe_rf.predict_proba(test[\"tokens\"])[:,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "842b14d2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.precision_recall_curve.PrecisionRecallDisplay at 0x7f0f71087400>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAmW0lEQVR4nO3de5xVZd338c+XAQWUQwr6EEiQmogiY5F47JbQUgsozTQrxWOaii8zfaxuT2TlIfPJwhsPAZqmqJlikdx5zBJJJkYTTEFFHeRORAS8RQX9PX+sNdNm2DN7DTN775nZ3/frNS/2Wuvaa/3WzDC/fV3Xuq5LEYGZmVWuLuUOwMzMysuJwMyswjkRmJlVOCcCM7MK50RgZlbhupY7gJbq169fDBkypNxhmJl1KDU1NW9ERP98xzpcIhgyZAjz588vdxhmZh2KpJebOuamITOzCudEYGZW4ZwIzMwqnBOBmVmFcyIwM6twRUsEkqZJel3SM00cl6RrJC2R9LSkTxYrFjMza1oxawQzgEOaOX4osHP6dQrwX0WMxczMmlC0cQQR8WdJQ5opMgG4OZJ5sJ+Q1FfSgIhYXox4LrlvIYteW1OMU1s7MaF6IMeMHlzuMMw6nHL2EQwEXs3Zrkv3bULSKZLmS5q/YsWKkgRnHcui5Wu4t3ZZucMw65A6xMjiiLgeuB5g1KhRm7WSzkXjdmvTmKx9Oeq6ueUOwazDKmeNYBmwQ872oHSfmZmVUDkTwSzg2PTpob2B1cXqHzAzs6YVrWlI0m3AgUA/SXXARUA3gIiYCswGDgOWAO8AxxcrFjMza1oxnxr6WoHjAZxerOubmVk2HllsZlbhnAjMzCqcE4GZWYVzIjAzq3AdYkCZ2eb4zbxXNnu0saersEriRGCdxqLlazYaYTzvpTcBGD10mxafB3AisIrhRGCdwoTqTaepGj10m836ZO/pKqzSOBFYp3DM6MH+BG+2mdxZbGZW4ZwIzMwqnBOBmVmFcyIwM6twTgRmZhXOicDMrMI5EZiZVTgnAjOzCudEYGZW4ZwIzMwqnBOBmVmFcyIwM6twTgRmZhXOs4+aFdDUAjdevMY6CycCs0bqF7SpX5cg3wI3XrzGOhMnArMC8i1w48VrrDMpaiKQdAjwc6AKuDEiLmt0/GPANKA/8CbwjYioK2ZMZoXsv1M/AG45aXSZIzErjaIlAklVwBTgYKAOeFLSrIhYlFPsp8DNEXGTpM8CPwG+WayYzLJwArBKU8ynhvYClkTEixHxPnA7MKFRmeHAQ+nrh/McNzOzIitm09BA4NWc7Tqg8Uetp4DDSZqPvgz0krRtRKzMLSTpFOAUgMGD3Tln7Y+fLLKOLFMikLQdsB/wUWAd8AwwPyI+bOX1vwv8UtJE4M/AMuCDxoUi4nrgeoBRo0ZFK69p1mp+ssg6k2YTgaQxwPnANsAC4HWgO/AlYEdJdwFXRcSaPG9fBuyQsz0o3dcgIl4jqREgaWvgiIh4a3NuxKyc/GSRdWSFagSHASdHxCuND0jqCnyRpDP4t3ne+ySws6ShJAngaOCYRufoB7yZ1iy+R/IEkVm75yeLrDNpNhFExLnNHNsA3NPccUlnAHNIHh+dFhELJU0maVaaBRwI/ERSkDQNnd7iOzArAycA60w2u7NY0vERMb25MhExG5jdaN+FOa/vAu7a3BjMzKz1WvP46CVtFoWZmZVNoc7ip5s6BGzf9uGYmVmpFWoa2h74PLCq0X4BjxclIjMzK6lCieD3wNYRUdv4gKRHihGQmZmVVqGnhk5s5tgxTR0zM7OOwyuUmZlVOCcCM7MK50RgZlbhvEKZWZE0npiunmcktfYmc41A0vXNbZtZYYuWr8k7XbVZObWkRnBdgW0zy5FvYjrPSGrtUeZEEBE1zW2b2cayTEzX1II2+bhJyYql0BQT9wFNLgQTEePbPCKzCnJv7TIWLV/D8AG9my3nRW6smArVCH5akijMKkTjDuT6JDDzW/s0+z43KVkxFRpZ/Gj9a0k9gMER8VzRozKrEMMH9GZC9cAWv89rJFtbyrpm8TiS2sEWwFBJ1cBkNw2ZtczmrmzmNZKtmLJ2Fl8M7AU8AhARtekSlGbWAm21spnXSLa2lDURrI+I1ZJy9zXZiWxmxVGoLyGXm48sq6yJYKGkY4AqSTsDk/B6BGYlU9+k1BL5nkhy85HlkzURnAn8AHgPuI1kQfofFisoM9tYlialLE8kufnI8smUCCLiHeAHki5PNmNtccMys9ba3CeSrPJkfWro08A0oFe6vRo4waOLzdqPLE8keSI8yydr09CvgG9HxGMAkvYHpgN7FCswM2uZzX0iyf0GljURfFCfBAAi4i+SNhR6k6RDgJ8DVcCNEXFZo+ODgZuAvmmZ8yNidsaYzGwzud/AchWaa+iT6ctHJV1H0lEcwFGkYwqaeW8VMAU4GKgDnpQ0KyIW5RT7T+COiPgvScOB2cCQzbgPM8tgc54+ss6vUI3gqkbbF+W8LjSOYC9gSUS8CCDpdmACkJsIAqh/tq0P8FqBc5pZK7TVgDbrXArNNTSmFeceCLyas10HNP4tvBj4b0lnAlsBB+U7kaRTgFMABg92O6aZWVvKvB6BpC8AuwHd6/dFxORWXv9rwIyIuErSPsCvJe0eER/mFoqI64HrAUaNGuURzWZmbSjTUpWSppL0C5wJCDgS+FiBty0DdsjZHpTuy3UicAdARMwlSTJuxDQzK6GsNYJ9I2IPSU9HxCWSrgL+WOA9TwI7p5PTLQOOBo5pVOYVYCwwQ9KuJIlgRfbwzay98NxGHVfWxevXpf++I+mjwHpgQHNviIgNwBkk01E8S/J00EJJkyXVT199DnCypKdInkiaGBFu+jHrgOrnNsq1aPmazEtxWvlkrRH8XlJf4Erg7yRP+9xY6E3pmIDZjfZdmPN6EbBf1mDNrH3I9+nfcxt1XFnnGqqfYO63kn4PdI+I1cULy8zak8Z/+PMtjOO5jTquQgPKDm/mGBFxd9uHZGbllO/TfuM//PkWxrGOq1CNYFwzxwJwIjDr4LIsg+k//J1boQFlx5cqEDNrH/xHv/JkHlBmZp1bS5bBtM7FicCswnkiOnMiMKtwnojOsk4x0VPSBZJuSLd3lvTF4oZmZmalkLVGMB2oAeobEZcBdwK/L0ZQZtY55Fsa0x3R7U/WKSZ2jIgrSKaWqF/MXkWLysw6JU850T5lrRG8L6kH6WI0knYE3itaVGbWKdR3RNf3Q3jKifYpayK4GLgf2EHSrSTzA00sUkxm1km4I7pjyDrX0H9LqgH2JmkSOisi3ihqZGZmVhKZEoGk+4DfALMi4n+LG5KZmZVS1s7inwIHAIsk3SXpK5K6F3qTmZm1f1mbhh4FHpVUBXwWOBmYBvQuYmxmZlYCLVm8vgfJbKRHAZ8EbipWUGZmVjpZ+wjuAPYieXLol8CjEfFhMQMzM7PSyFoj+BXwtYj4oJjBmJlZ6RVaoeyzEfEQsBUwQdp4MLFXKDMz6/gK1Qj+A3iI/CuVeYUyM7NOoNAKZRelLydHxEu5xyQNLVpUZmZWMln7CH5L8qRQrruAT7VtOGZW6X4z75W8E9N51tLiKdRHMAzYDegj6fCcQ72BggPKJB0C/ByoAm6MiMsaHb8aGJNu9gS2i4i+maM3sw4l37TUTZUZPXSbhn2Llq8BcCIokkI1gl2ALwJ92bifYC3JoLImpYPPpgAHA3XAk5JmRcSi+jIRcXZO+TOBPVsSvJl1PqOHbrPJp3/PWlpchfoI7gXulbRPRLT0J7EXsCQiXgSQdDswAVjURPmvARc1cczMOpGZ39qncCErmUJNQ+elC9IcI+lrjY9HxKRm3j4QeDVnuw7IOyetpI8BQ0meUDIz20hTTUruN2gbhZqGnk3/nV/kOI4G7mpqwJqkU4BTAAYP9g/drKOqX6imLbjfoO0Uahq6L/23YV4hSV2ArSNiTYFzLwN2yNkelO7L52jg9GbiuB64HmDUqFFR4Lpm1k61dqGa3CalIef/gXkvven1kNtApmmoJf1GUm9JWwHPkExHfW6Btz0J7CxpqKQtSP7Yz8pz7mHARwD3BplZXvvv1K9gbcLrIW++rOMIhkfEGklfB/4InA/UAFc29YaI2CDpDGAOyeOj0yJioaTJwPyIqE8KRwO3R4Q/6ZtZXvlqEl4Pue1kTQTdJHUDvgT8MiLWSyr4hzsiZgOzG+27sNH2xRljMDNr4PWQ207WFcquA5aSTD735/Qpn0J9BGZm1gFkXaHsGuCanF0vSxrTVHkzM+s4snYW95H0M0nz06+rSGoHZmbWwWVtGppGMq3EV9OvNcD0YgVlZmalk7WzeMeIOCJn+xJJtUWIx8yszXgm02yy1gjWSdq/fkPSfsC64oRkZtY27q1d1jACuZ7HG2wqa43gVOBmSX3S7VXAccUJycys5fLNR7Ro+RqGD+i90YhkjzfYVMFEIKka2Ilk4NcygAzTS5iZld3wAb2ZUD2w3GG0e4VmH70Q+AbJKOIrgJ9ExA2lCMzMrCUajzS27ArVCI4CqiPiHUnbAvcDTgRm1u44AWy+Qp3F70XEOwARsTJDeTMz62AK1Qg+Lql+cjgBO+ZsExHjixaZmZmVRKFEMKHR9k+LFYiZmZVHoYVpHi1VIGZmVh7NtvlLuk/SuHQK6sbHPi5psqQTiheemZkVW6GmoZOB7wD/T9KbwAqgOzAEeIFkbYJ7ixqhmVkbyjfwrNKnnCjUNPQ/wHnAeZKGAANIppZ4vv5pIjOzjqx+CgonggwiYinJ4jRmZh1e/bQTnnKiBYnAzKwzqB+BbP/mRGBmFcUjkDflRGBmFS1f5zFUVgdy1qUq95P0J0nPS3pR0kuSXix2cGZm5VBpaxZkrRH8CjibZBbSD4oXjplZeVTymgVZE8HqiPhjUSMxMysDdx5nTwQPS7oSuBt4r35nRPy9uTdJOgT4OVAF3BgRl+Up81XgYiCApyLimIwxmZm1mjuPsyeC+u/UqJx9AXy2qTdIqgKmAAcDdcCTkmZFxKKcMjsD3wP2i4hVkrZrSfBmZtZ6mRJBRIzZjHPvBSyJiBcBJN1OMpvpopwyJwNTImJVep3XN+M6ZmbWClmfGuoj6WeS5qdfV+UsZN+UgcCrOdt16b5cnwA+Iemvkp5Im5LyXf+U+muvWLEiS8hmZpZR1hXHpgFrga+mX2uA6W1w/a7AzsCBwNeAGyT1bVwoIq6PiFERMap///5tcFkzM6uXtY9gx4g4Imf7Ekm1Bd6zDNghZ3tQui9XHTAvItYDL0l6niQxPJkxLjMza6WsNYJ1kvav35C0H8kspM15EthZ0lBJWwBHA7MalbmHpDaApH4kTUUeqGZmVkJZawSnATel/QIC3gQmNveGiNgg6QxgDsnjo9MiYqGkycD8iJiVHvucpEUkA9XOjYiVm3crZma2ObI+NVQLjJTUO91ek/F9s4HZjfZdmPM6SBa++U7GeM3MrI01mwgkfSMibpH0nUb7AYiInxUxNjMzK4FCNYKt0n97FTsQMzMrj0JLVV6X/ntJacIxM7NSyzqg7ApJvSV1k/SgpBWSvlHs4MzMrPiyPj76ubSD+Isk6xbvBJxbrKDMzKx0siaC+iakLwB3RsTqIsVjZmYllnUcwe8l/ZNkENlpkvoD7xYvLDMzK5VMNYKIOB/YFxiVTgfxvyQziZqZWQdXaBzBZyPiIUmH5+zLLXJ3sQIzM7PSKNQ09B/AQ8C4PMcCJwIzsw6v0DiCi9J/jy9NOGZmVmpZxxH8OHedAEkfkXRp0aIyM7OSyfr46KER8Vb9Rrq05GFFicjMzEoqayKokrRl/YakHsCWzZQ3M7MOIus4gluBByXVL095PHBTcUIyMyuveS+9CcBR181tttyE6oEcM3pwKUIqqqzrEVwu6SngoHTXDyNiTvHCMjNr3xYtT5ZlqZhEkHoW2BARD0jqKalXRKwtVmBmZuWy/079ALjlpNFNlilUW+hIMiUCSScDpwDbADsCA4GpwNjihWZmVh7NJYDOKGtn8enAfsAagIhYDGxXrKDMzKx0siaC9yLi/foNSV1JRhabmVkHlzURPCrp+0APSQcDdwL3FS8sMzMrlayJ4P8CK4B/AN8CZgP/WaygzMysdAp2FkuqAhZGxDDghuKHZGZmpVSwRhARHwDPSWrxw7KSDpH0nKQlks7Pc3xiuv5xbfp1UkuvYWZmrZN1HMFHgIWS/kayKA0AETG+qTekNYkpwMFAHfCkpFkRsahR0ZkRcUbLwjYzs7aSNRFcsBnn3gtYEhEvAki6nWRVs8aJwMzMyqjQCmXdgVOBnUg6in8VERsynnsg8GrOdh2Qb5TGEZI+AzwPnB0RrzYuIOkUkgFtDB7c8Ydzm5m1J4X6CG4CRpEkgUOBq9r4+vcBQyJiD+BPNDGRXURcHxGjImJU//792zgEM7PKVqhpaHhEjACQ9Cvgby049zJgh5ztQem+BhGxMmfzRuCKFpzfzMzaQKEawfr6Fy1oEqr3JLCzpKGStgCOBmblFpA0IGdzPMnEdmZmVkKFagQjJa1JX4tkZPGa9HVERO+m3hgRGySdAcwBqoBpEbFQ0mRgfkTMAiZJGg9sAN4EJrbudszMrKUKLV5f1ZqTR8RsklHIufsuzHn9PeB7rbmGmZm1TtYpJszMrJNyIjAzq3BOBGZmFc6JwMyswjkRmJlVOCcCM7MKl3XSOTMza2TR8jUcdd3cjfZNqB7IMaM71pxoTgRmZpthQvXATfYtWp6Mv3UiMDOrAMeMHrzJH/zGtYOOwn0EZmYVzonAzKzCORGYmVU4JwIzswrnRGBmVuGcCMzMKpwTgZlZhXMiMDOrcJ1iQNn69eupq6vj3XffLXcoZhvp3r07gwYNolu3buUOxUok37QTjbW3aSg6RSKoq6ujV69eDBkyBEnlDscMgIhg5cqV1NXVMXTo0HKHYyWQb9qJxtrjNBSdIhG8++67TgLW7khi2223ZcWKFeUOxUok37QTjbXHaSg6RSIAnASsXfLvpTU276U3gY0TQrmbitxZbGZWRouWr+He2mVljcGJoI1UVVVRXV3N7rvvzrhx43jrrbfa5LwzZszgjDPOaJNzDRkyhBEjRlBdXU11dTWPP/54m5y3sdraWmbPnt3k8QULFnDiiSdutO9LX/oSe++990b7Lr74YgYOHNjwfZ01a1arY6upqWHEiBHstNNOTJo0iYjYpMzq1asZN24cI0eOZLfddmP69OkbHV+zZg2DBg3a6Ody0EEHsWrVqlbHZ53f/jv1Y/+d+jHzW/sw81v7MHxA73KH5ETQVnr06EFtbS3PPPMM22yzDVOmTCl3SHk9/PDD1NbWUltby7777pvpPRs2bGjRNQolgh//+MdMmjSpYfutt96ipqaG1atX8+KLL25U9uyzz6a2tpY777yTE044gQ8//LBFsTR22mmnccMNN7B48WIWL17M/fffv0mZKVOmMHz4cJ566ikeeeQRzjnnHN5///2G4xdccAGf+cxnNnrPN7/5Ta699tpWxWaV4ZaTRnPLSaPLHcZGitpHIOkQ4OdAFXBjRFzWRLkjgLuAT0fE/NZc85L7FrLotTWtOcUmhn+0NxeN2y1z+X322Yenn34agL/97W+cddZZvPvuu/To0YPp06ezyy67MGPGDGbNmsU777zDCy+8wJe//GWuuOIKAKZPn85PfvIT+vbty8iRI9lyyy0BWLp0KSeccAJvvPEG/fv3Z/r06QwePJiJEyfSo0cPFixYwOuvv860adO4+eabmTt3LqNHj2bGjBlNxtrcObt3786CBQvYb7/9OP300zn99NNZsWIFPXv25IYbbmDYsGHceeedXHLJJVRVVdGnTx8eeOABLrzwQtatW8df/vIXvve973HUUUc1XG/t2rU8/fTTjBw5smHf3Xffzbhx49h+++25/fbb+f73v79JnLvuuitdu3bljTfeYLvttsv8s8i1fPly1qxZ01DzOPbYY7nnnns49NBDNyonibVr1xIRvP3222yzzTZ07Zr8V6mpqeFf//oXhxxyCPPn//tXdfz48RxwwAH84Ac/2KzYzMqpaIlAUhUwBTgYqAOelDQrIhY1KtcLOAuYV6xYSumDDz7gwQcfbGj6GDZsGI899hhdu3blgQce4Pvf/z6//e1vgeST84IFC9hyyy3ZZZddOPPMM+natSsXXXQRNTU19OnThzFjxrDnnnsCcOaZZ3Lcccdx3HHHMW3aNCZNmsQ999wDwKpVq5g7dy6zZs1i/Pjx/PWvf+XGG2/k05/+NLW1tVRXVwMwZswYqqqq2HLLLZk3b16z56yrq+Pxxx+nqqqKsWPHMnXqVHbeeWfmzZvHt7/9bR566CEmT57MnDlzGDhwIG+99RZbbLEFkydPZv78+fzyl7/c5Pszf/58dt9994323XbbbVx44YVsv/32HHHEEXkTwbx58+jSpQv9+/ffaP/DDz/M2WefvUn5nj17btL0tWzZMgYNGtSwPWjQIJYt27Rt9owzzmD8+PF89KMfZe3atcycOZMuXbrw4Ycfcs4553DLLbfwwAMPbPSej3zkI7z33nusXLmSbbfddpNzmrVnxawR7AUsiYgXASTdDkwAFjUq90PgcuDctrhoSz65t6V169ZRXV3NsmXL2HXXXTn44IOBpL35uOOOY/HixUhi/fr1De8ZO3Ysffr0AWD48OG8/PLLvPHGGxx44IENf/COOuoonn/+eQDmzp3L3XffDSRNEeedd17DucaNG4ckRowYwfbbb8+IESMA2G233Vi6dGlDInj44Yfp169fw/uaO+eRRx5JVVUVb7/9No8//jhHHnlkw7H33nsPgP3224+JEyfy1a9+lcMPP7zg92n58uUb/TH/17/+xeLFi9l///2RRLdu3XjmmWcaksXVV1/NLbfcQq9evZg5c+YmT+GMGTOG2tragtdtiTlz5lBdXc1DDz3ECy+8wMEHH8wBBxzAzTffzGGHHbZRMsm13Xbb8dprrzkRWIdTzEQwEHg1Z7sO2KhhTNIngR0i4g+SmkwEkk4BTgEYPLj9DMLIVd9H8M477/D5z3+eKVOmMGnSJC644ALGjBnD7373O5YuXcqBBx7Y8J76Jh9IOptb2hafq/5cXbp02ei8Xbp02ezzbrXVVgB8+OGH9O3bN+8f3KlTpzJv3jz+8Ic/8KlPfYqamppmz9mjR4+NRoDfcccdrFq1qmHA1Zo1a7jtttv40Y9+BCR9BN/97nebPF9LagQDBw6krq6uYbuuro6BAzcdADR9+nTOP/98JLHTTjsxdOhQ/vnPfzJ37lwee+wxrr32Wt5++23ef/99tt56ay67LGnxrG/+M2upfKORS/lIadk6iyV1AX4GnFOobERcHxGjImJU46aB9qZnz55cc801XHXVVWzYsIHVq1c3/LFprq2+3ujRo3n00UdZuXIl69ev584772w4tu+++3L77bcDcOutt3LAAQe0Ot4s5+zduzdDhw5tiCUieOqppwB44YUXGD16NJMnT6Z///68+uqr9OrVi7Vr1+a93q677sqSJUsatm+77Tbuv/9+li5dytKlS6mpqWmIJ4v6GkHjr3xPRA0YMIDevXvzxBNPEBHcfPPNTJgwYZNygwcP5sEHHwSSGstzzz3Hxz/+cW699VZeeeUVli5dyk9/+lOOPfbYhiQQEfzP//wPQ4YMyRy7GSR/8Bs/OVTqR0qLWSNYBuyQsz0o3VevF7A78Eha3f8/wCxJ41vbYVxue+65J3vssQe33XYb5513HscddxyXXnopX/jCFwq+d8CAAVx88cXss88+9O3bt6FJB+AXv/gFxx9/PFdeeWVDx25rZT3nrbfeymmnncall17K+vXrOfrooxk5ciTnnnsuixcvJiIYO3YsI0eOZPDgwVx22WVUV1dv0lk8bNgwVq9ezdq1a1m5ciUvv/zyRo+NDh06lD59+jBvXnG6jK699lomTpzIunXrOPTQQxs6iqdOnQrAqaeeygUXXMDEiRMZMWIEEcHll1++UXNaPjU1Ney9994NncpmWeUbjXzUdXPz1hJa+uBKVsr3HHWbnFjqCjwPjCVJAE8Cx0TEwibKPwJ8t1ASGDVqVOQ+rQHw7LPPsuuuu7ZF2FYCV199Nb169eKkk04qdyht5qyzzmL8+PGMHTt2k2P+/bSW+s28V/LWCFqTCCTVRMSofMeK9vElIjZIOgOYQ/L46LSIWChpMjA/Ilo/Osg6pNNOO22jJq/OYPfdd8+bBMw2R5Y5i9pS0WoExeIagXU0/v209qC5GkGnGVnc0RKaVQb/XlpH0CkSQffu3Vm5cqX/01m7Ur8eQffu3csdilmzOsUjDoMGDaKurs7zvlu7U79CmVl71ikSQbdu3bwClJnZZuoUTUNmZrb5nAjMzCqcE4GZWYXrcOMIJK0AXt7Mt/cD3mjDcDoC33Nl8D1Xhtbc88ciIu9kbR0uEbSGpPlNDajorHzPlcH3XBmKdc9uGjIzq3BOBGZmFa7SEsH15Q6gDHzPlcH3XBmKcs8V1UdgZmabqrQagZmZNeJEYGZW4TplIpB0iKTnJC2RdH6e41tKmpkenydpSBnCbFMZ7vk7khZJelrSg5I+Vo4421Khe84pd4SkkNThHzXMcs+Svpr+rBdK+k2pY2xrGX63B0t6WNKC9Pf7sHLE2VYkTZP0uqRnmjguSdek34+nJX2y1ReNiE71RbIa2gvAx4EtgKeA4Y3KfBuYmr4+GphZ7rhLcM9jgJ7p69Mq4Z7Tcr2APwNPAKPKHXcJfs47AwuAj6Tb25U77hLc8/XAaenr4cDScsfdynv+DPBJ4Jkmjh8G/BEQsDcwr7XX7Iw1gr2AJRHxYkS8D9wOTGhUZgJwU/r6LmCsJJUwxrZW8J4j4uGIeCfdfALo6HMjZ/k5A/wQuBx4t5TBFUmWez4ZmBIRqwAi4vUSx9jWstxzAL3T132A10oYX5uLiD8DbzZTZAJwcySeAPpKGtCaa3bGRDAQeDVnuy7dl7dMRGwAVgPbliS64shyz7lOJPlE0ZEVvOe0yrxDRPyhlIEVUZaf8yeAT0j6q6QnJB1SsuiKI8s9Xwx8Q1IdMBs4szShlU1L/78X1CnWI7DsJH0DGAX8R7ljKSZJXYCfARPLHEqpdSVpHjqQpNb3Z0kjIuKtcgZVZF8DZkTEVZL2AX4tafeI+LDcgXUUnbFGsAzYIWd7ULovbxlJXUmqkytLEl1xZLlnJB0E/AAYHxHvlSi2Yil0z72A3YFHJC0laUud1cE7jLP8nOuAWRGxPiJeAp4nSQwdVZZ7PhG4AyAi5gLdSSZn66wy/X9vic6YCJ4EdpY0VNIWJJ3BsxqVmQUcl77+CvBQpL0wHVTBe5a0J3AdSRLo6O3GUOCeI2J1RPSLiCERMYSkX2R8RMwvT7htIsvv9j0ktQEk9SNpKnqxhDG2tSz3/AowFkDSriSJoDOvWzsLODZ9emhvYHVELG/NCTtd01BEbJB0BjCH5ImDaRGxUNJkYH5EzAJ+RVJ9XELSKXN0+SJuvYz3fCWwNXBn2i/+SkSML1vQrZTxnjuVjPc8B/icpEXAB8C5EdFha7sZ7/kc4AZJZ5N0HE/syB/sJN1Gksz7pf0eFwHdACJiKkk/yGHAEuAd4PhWX7MDf7/MzKwNdMamITMzawEnAjOzCudEYGZW4ZwIzMwqnBOBmVmFcyKwopP0gaRaSc9Iuk9S3zY+/9L0mXkkvd1EmR6SHpVUJWmIpHVpTIskTU1HIrfkmqMkXZO+PlDSvjnHTpV0bGvuKT3PxZK+W6DMDElfacE5hzQ1q2Wjcj+S9Grj76ekMySdkPV61jE4EVgprIuI6ojYnWTcxulliOEE4O6I+CDdfiEiqoE9SGas/FJLThYR8yNiUrp5ILBvzrGpEXFzawMus/tIJnxrbBqdfy6fiuNEYKU2l3SCLEk7SrpfUo2kxyQNS/dvL+l3kp5Kv/ZN99+Tll0o6ZQWXvfrwL2Nd6aTDj4O7JR+Wn5I/16zYXB63SPT2sxTkv6c7jtQ0u+VrGVxKnB2WsM4oP6TvKRhkv5Wf630/P9IX38qraHUSJqjArNHSjpZ0pNpDL+V1DPn8EGS5kt6XtIX0/JVkq5M3/O0pG+15JsVEU/kG62azmC7VFK+JGEdlBOBlYykKpKpAOpH/V4PnBkRnwK+C1yb7r8GeDQiRpLMy74w3X9CWnYUMElSphlj06kJPh4RS/Mc65nG9A/gF8BNEbEHcGsaB8CFwOfTeDYajZ2ecypwdVrreSzn2D+BLSQNTXcdBcyU1C291lfS+5kG/KjAbdwdEZ9OY3iWZH6dekNIPr1/AZgqqXt6fHVEfBr4NHByThz19/5RSbMLXDef+cABm/E+a6c63RQT1i71kFRLUhN4FviTpK1JmlPqp7wA2DL997PAsQBpU87qdP8kSV9OX+9AMplalukT+gFvNdq3YxpTAPdGxB8l/Ro4PD3+a+CK9PVfgRmS7gDuznC9XHeQJIDL0n+PAnYhmRDvT+m9VwGF5orZXdKlQF+SqULm5F4jnWlzsaQXgWHA54A9cvoP+pB8v56vf1NEvEYyVUFLvZ5ewzoJJwIrhXURUZ1++p5D0kcwA3grbacvSNKBwEHAPhHxjqRHSCYXy3T9PGVfyHrtiDhV0miST9w1kj6V8boAM0mS3d3JqWKxpBHAwojYpwXnmQF8KSKekjSRdGK5+hAbh0yyetWZEZGbMFDbLMvaneR7ap2Em4asZNL25Ukkk4S9A7wk6UhoWId1ZFr0QZLlNOvbuvuQfKJdlSaBYSTTSme97iqgKm0yac7j/HsCwq8Dj6Ux7BgR8yLiQpJZLXdo9L61JNNe57v2CySTv11AkhQAngP6K5k7H0ndJO1WILZewPK0WenrjY4dKamLpB1JlnR8jiThnpaWR9InJG1V4BpZfQIo+OSRdRxOBFZSEbEAeJpkMZGvAydKeoqkH6B+CcKzgDFpx2oNyVM99wNdJT1L0szyRAsv/d/A/gXKnAkcL+lp4JtpHABXSvpH+tjl4yTr5ua6D/hyfWdxnvPOBL7Bv+fMf59k+vPL03uvJeepoyZcAMwjaab6Z6NjrwB/I1l17tSIeBe4EVgE/D2N+zoatQA010cg6QolM1/2lFQn6eKcw/sBfyoQr3Ugnn3UKoKSZSvPjohvljuWjkzJuhbf8fexc3GNwCpCRPwdeDh9csk2Xz+S2ol1Iq4RmJlVONcIzMwqnBOBmVmFcyIwM6twTgRmZhXOicDMrML9f8Yj/VoiG1T7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "PrecisionRecallDisplay.from_predictions(\n",
    "    y_true=test[\"label_num\"],\n",
    "    y_pred=score_rf,\n",
    "    name=\"RandomForest\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cac39e81",
   "metadata": {},
   "source": [
    "## 特徴量の重要度\n",
    "\n",
    "ランダムフォレストでは、 `feature_importances_` 属性を見ることで、 素性の重要度を知ることができます。\n",
    "モデルの精度を改善していくステップで重要な情報となります。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "59d47fef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.62063358e-03, 3.31198460e-06, 1.33262805e-04, ...,\n",
       "       8.58042819e-05, 3.38860088e-05, 1.90558795e-04])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_rf[\"clf\"].feature_importances_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "1a17970c",
   "metadata": {},
   "outputs": [],
   "source": [
    "importances = pipe_rf[\"clf\"].feature_importances_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0b8a82e2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0.06262130422316252, '残念'),\n",
       " (0.03232310022880119, 'が'),\n",
       " (0.029163227525310448, '狭い'),\n",
       " (0.02537735713464425, 'ない'),\n",
       " (0.024742874383347428, '。'),\n",
       " (0.020948336676578652, 'ぬ'),\n",
       " (0.020610069080122854, '少し'),\n",
       " (0.0188537174145827, 'た'),\n",
       " (0.016496863123408128, 'です'),\n",
       " (0.015541715069967977, '悪い')]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(sorted(zip(importances, words), reverse=True))[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f364cc6b",
   "metadata": {},
   "source": [
    "この結果を見ると、「が」「です」といった分類に効果があるとは思えない単語が重要な素性として選ばれてしまっていることが分かります。\n",
    "そこで、例えば次の手としてストップワードを定義して素性から取り除く方針が思いつきます。\n",
    "\n",
    "このように、素性を選択するためにランダムフォレストをまずは適用してみるという方針も可能です。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97cfea08",
   "metadata": {},
   "source": [
    "## 勾配ブースティング\n",
    "\n",
    "最後に、勾配ブースティングの手法を見ておきます。\n",
    "\n",
    "勾配ブースティングを分類問題に適用するには\n",
    "[sklearn.ensemble.GradientBoostingClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html)\n",
    "を使います。\n",
    "\n",
    "並列化できず、したがって `n_jobs` パラメータは指定できないことに注意してください。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "59d68f3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "5853be20",
   "metadata": {},
   "outputs": [],
   "source": [
    "gb = GradientBoostingClassifier(\n",
    "    n_estimators=1000,\n",
    "    random_state=0,\n",
    "    learning_rate=0.1,\n",
    ")\n",
    "\n",
    "pipe_gb = Pipeline([\n",
    "    (\"vect\", TfidfVectorizer(tokenizer=str.split)),\n",
    "    (\"clf\", gb),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "4336c198",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-4 {color: black;background-color: white;}#sk-container-id-4 pre{padding: 0;}#sk-container-id-4 div.sk-toggleable {background-color: white;}#sk-container-id-4 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-4 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-4 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-4 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-4 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-4 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-4 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-4 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-4 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-4 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-4 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-4 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-4 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-4 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-4 div.sk-item {position: relative;z-index: 1;}#sk-container-id-4 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-4 div.sk-item::before, #sk-container-id-4 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-4 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-4 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-4 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-4 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-4 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-4 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-4 div.sk-label-container {text-align: center;}#sk-container-id-4 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-4 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-4\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                 TfidfVectorizer(tokenizer=&lt;method &#x27;split&#x27; of &#x27;str&#x27; objects&gt;)),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 GradientBoostingClassifier(n_estimators=1000,\n",
       "                                            random_state=0))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-12\" type=\"checkbox\" ><label for=\"sk-estimator-id-12\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                 TfidfVectorizer(tokenizer=&lt;method &#x27;split&#x27; of &#x27;str&#x27; objects&gt;)),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 GradientBoostingClassifier(n_estimators=1000,\n",
       "                                            random_state=0))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-13\" type=\"checkbox\" ><label for=\"sk-estimator-id-13\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(tokenizer=&lt;method &#x27;split&#x27; of &#x27;str&#x27; objects&gt;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-14\" type=\"checkbox\" ><label for=\"sk-estimator-id-14\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GradientBoostingClassifier</label><div class=\"sk-toggleable__content\"><pre>GradientBoostingClassifier(n_estimators=1000, random_state=0)</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('vect',\n",
       "                 TfidfVectorizer(tokenizer=<method 'split' of 'str' objects>)),\n",
       "                ('clf',\n",
       "                 GradientBoostingClassifier(n_estimators=1000,\n",
       "                                            random_state=0))])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_gb.fit(train[\"tokens\"], train[\"label_num\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "2ed8ce08",
   "metadata": {},
   "outputs": [],
   "source": [
    "score_gb = pipe_gb.predict_proba(test[\"tokens\"])[:,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "73680993",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.precision_recall_curve.PrecisionRecallDisplay at 0x7f0f7070b4c0>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAntElEQVR4nO3dfZxVZb338c+XQQUTMEW9DSRISEXQMSdR0Y6kllqAaYZaPuRTmYi3Wd5axyc6PWfdWXgjKpBpipYpFsnJh8wSyZnDYIKpqKiDnEREwOMj+rv/WGumzTDMXgOz9p7Z+/t+veY1e611rbV+aw/s317Xda3rUkRgZmbVq0e5AzAzs/JyIjAzq3JOBGZmVc6JwMysyjkRmJlVuZ7lDqCj+vfvH4MHDy53GGZm3UpDQ8PLEbFDW9u6XSIYPHgw9fX15Q7DzKxbkfTcxra5asjMrMo5EZiZVTknAjOzKudEYGZW5ZwIzMyqXG6JQNJ0SS9Jemwj2yXpKklLJD0q6SN5xWJmZhuX5x3BTOCIdrYfCQxLf84C/l+OsZiZ2Ubk9hxBRPxZ0uB2iowHbohkHOyHJW0raeeIWJ5HPFfctYjFL67J49BlMb52ACeOGlTuMMysApSzjWAA8ELBclO6bgOSzpJUL6l+xYoVJQmuK1u8fA13Ni4rdxhmViG6xZPFETENmAZQV1e3STPpXDZ2z06NqZwmXDOv3CGYWQUp5x3BMmCXguWB6TozMyuhciaC2cDJae+h/YHVebUPmJnZxuVWNSTpZuAQoL+kJuAyYAuAiJgKzAGOApYArwNfzCsWMzPbuDx7DZ1QZHsA5+R1fjMzy8ZPFpuZVTknAjOzKtctuo+aFfOr+c+3+WyFH7wzK86JwLqUjX2gFzP/2VcAGDVku5Z1i5cnT5I7EZi1z4nASibLh3xbH+hZjBqy3Qbf/v3gnVk2TgTWKTrrQ76tD3Qzy5cTgRXlD/nOk7Xqq9rfJystJwJbT1sfVP6QLy7rB3yW99JtG1ZqTgRVLOuHfrV9yG9Kg3XWto0s76XbNqzUnAiqSOsPuGr70O/Mb+2tVfL7ZpXPiaBCZfm2X+kfXs3X2/wNuzO/tZtVEieCClHt3/az6M7X31Zi767XYl2PE0GFuLNxGYuXr2H4zn2B7v2h11kOGtofgBvPGFXmSDpu8fI167UVtE7sblC2zuRE0A21rvIAWpLArC8dUK6wupzumAAg+abfWuvE7gZl60xOBBVi+M592/wAse7nxFGD/E3fSsqJoBvqzlUeZtb15JoIJB0B/BSoAa6LiO+12v5BYDqwA/AK8IWIaMozpkrgBGBmnSm3+Qgk1QBTgCOB4cAJkoa3KvYj4IaI2AuYDHw3r3jMzKxted4R7AcsiYhnACTdAowHFheUGQ58NX19P3BHjvGYWQYeD6n65JkIBgAvFCw3Aa3rNBYCx5BUH30G6CNp+4hYWVhI0lnAWQCDBvkfntmm6qwBBN19tbKUu7H4a8DPJZ0K/BlYBrzbulBETAOmAdTV1UUpAzTrLko5Sqy7r1aWPBPBMmCXguWB6boWEfEiyR0BkrYBjo2IV3OMyawitPUsSblHifV0od1XnongEWCYpCEkCeB44MTCApL6A69ExHvAxSQ9iMxsE5T6afJiTz83lwFXIXV1uSWCiFgnaSIwl6T76PSIWCRpMlAfEbOBQ4DvSgqSqqFz8orHrJKU+1mSLE8/g6uQugtFdK8q97q6uqivry93GGaWQXMi8NAn5SepISLq2tqW23MEZmbWPTgRmJlVOScCM7Mql6mxWNKOwGjgA8AbwGMkDb7v5RibmZmVQLuJQNIY4CJgO2AB8BLQCzga2FXSr4ErI2JNznGamVlOit0RHAWcGRHPt94gqSfwaeBw4Dc5xGZmZiXQbiKIiK+3s20dHiTOzKzb2+TGYklf7MxAzMysPDan19AVnRaFmZmVTbHG4kc3tgnYqfPDMTOzUivWWLwT8ElgVav1Ah7KJSIzq1geobRrKpYIfgdsExGNrTdI+lMeAZlZZSkcpdQjlHZNxXoNnd7OthM3ts3MDDYcpdQjlHZN5Z6hzMwq2ImjBvmbfjfgsYbMzKqcE4GZWZXLNRFIOkLSE5KWSLqoje2DJN0vaYGkRyUdlWc8Zma2ocyJQNK09pbbKF8DTAGOBIYDJ0ga3qrYvwO3RsQ+JHMaX501HjMz6xwduSO4pshya/sBSyLimYh4G7gFGN+qTAB909f9gBc7EI+ZmXWCzIkgIhraW27DAOCFguWmdF2hy4EvSGoC5gDntnUgSWdJqpdUv2LFiqwhm5lZBsWGmLiL5Ft7myJi3Gae/wRgZkRcKekA4JeSRrSe8CYipgHTIJm8fjPPaWZmBYo9R/CjzTj2MmCXguWB6bpCpwNHAETEPEm9gP4kE+CYmVkJFHuy+IHm15J6A4Mi4omMx34EGCZpCEkCOB5o/TTy88ChwExJe5DMfua6HzOzEsrURiBpLNAI3J0u10qa3d4+6cQ1E4G5wOMkvYMWSZosqblK6QLgTEkLgZuBUyPCVT9mZiWUdYiJy0l6Af0JICIa02/67YqIOSSNwIXrLi14vRgYnTEGMzPLQdZeQ+9ExOpW6/zN3cysAmS9I1gk6USgRtIwYBKej8DMrCJkvSM4F9gTeIukLn8N8L9zisnMzEoo0x1BRLwOfFPS95PFWJtvWGZmVipZew19VNLfgUeBv0taKGnffEMzM7NSyNpGcD3wlYh4EEDSQcAMYK+8AjOz6uR5jUsvayJ4tzkJAETEXyStyykmM6sizfMYe17j8ik21tBH0pcPSLqGpKE4gAmkzxSYmXUmz2tcesXuCK5stXxZwWs/R2Bmm+2gof0BuPGMUWWOpHoVG2toTKkCMbPq5ARQflnbCJD0KZJnCXo1r4uIyXkEZWZmpZMpEUiaCmwNjAGuAz4L/C3HuMzM2tVW7yL3LNo0We8IDoyIvSQ9GhFXSLoS+EOegZmZNWvds6hwXXPvovnPvsL8Z19xctgEWRPBG+nv1yV9AFgJ7JxPSGZmxbXuXTT4ot9vUMbdTrPJmgh+J2lb4IfAf5H0GLour6DMzApl6VnUVhl3O80m61hD30pf/kbS74BebQxLbWaWiyw9i9z7aNMVe6DsmHa2ERG3F9n/COCnQA1wXUR8r9X2n5A0QEPSGL1jRGybIW4zM+skxe4IxrazLYCNJgJJNcAU4HCgCXhE0ux0VrLkABHnF5Q/F9gnS9BmZtZ5ij1Q9sXNOPZ+wJKIeAZA0i3AeGDxRsqfwPpPLpuZWQlknZhmUwwAXihYbkrXbUDSB4EhwH0b2X6WpHpJ9StWrOj0QM3MqlmeiaAjjgd+HRHvtrUxIqZFRF1E1O2www4lDs3MrLJlHmJiEywDdilYHpiua8vxwDk5xmJmVWrx8jUbdCP1Q2bryzpD2daSLpF0bbo8TNKni+z2CDBM0hBJW5J82M9u49i7A+8H3OHXzDrV+NoBDN+573rrFi9f0+bEN9Us6x3BDKABOCBdXgbcBvxuYztExDpJE4G5JN1Hp0fEIkmTgfqIaE4KxwO3RISHtTazTnXiqEEbfPP3Q2YbypoIdo2ICZJOgGQye0kqtlNEzAHmtFp3aavlyzPGYGZmOcjaWPy2pN6kk9FI2hV4K7eozMysZLLeEVwO3A3sIukmYDRwak4xmZlZCWUda+g/JTUA+wMCzouIl3ONzMzMSiLrxDR3Ab8CZkfE/+QbkpmZlVLWNoIfAQcDiyX9WtJnJfUqtpOZmXV9WauGHgAeSAeS+zhwJjAd6NvujmZm1uV1ZPL63iSjkU4APgL8Iq+gzMysdLK2EdxKMpro3cDPgQci4r08AzMzs9LIekdwPXDCxgaFMzOz7qvYDGUfj4j7gPcB41s/TFxshjIzM+v6it0R/BvJHAFtzVTW7gxlZmbWPRSboax5xrDJEfFs4TZJQ3KLyszMSibrcwS/aWPdrzszEDMzK49ibQS7A3sC/SQdU7CpL+AHyszMKkCxNoLdgE8D27J+O8FakofKzMysmyvWRnAncKekAyLCszmYmVWgYlVDF0bED4ATmyelKRQRk4rsfwTwU5IZyq6LiO+1UeZzJMNcB7AwIk7MHr6ZmW2uYlVDj6e/6zt64HRcoinA4UAT8Iik2RGxuKDMMOBiYHRErJK0Y0fPY2Zmm6dY1dBd6e+WcYUk9QC2iYg1RY69H7AkIp5J97sFGA8sLihzJjAlIlal53mpw1dgZmabJVP3UUm/ktRX0vuAx0iGo/56kd0GAC8ULDel6wp9GPiwpL9KejitSmrr/GdJqpdUv2LFiiwhm5lZRlnHGhoeEWskfR74A3AR0AD8sBPOPww4BBgI/FnSyIh4tbBQREwDpgHU1dXFZp7TzKrY/GdfAWDCNf/q/zK+dgAnjhpUrpDKLusDZVtI2gI4mmSWsndIJ7JvxzJgl4Llgem6Qk3Nx0ufXH6SJDGYmZXE4uVruLOx9UdTdcl6R3ANsBRYSPKt/YNAsTaCR4Bh6VAUy4DjgdY9gu4ATgBmSOpPUlX0TMaYzMw67KCh/QG48YxRwPp3BtUq6wxlVwFXFax6TtKYIvuskzQRmEvSfXR6RCySNBmoj4jZ6bZPSFoMvAt8PSJWbsqFmJll0ZwA7F+yTkzTD7gM+Fi66gFgMrC6vf0iYg4wp9W6SwteB/DV9MfMzMoga9XQdJLeQp9Ll08CZgDHbHQPM7NuoK3GY6iuBuSsiWDXiDi2YPkKSY05xGNmVnaLlydNoE4E63tD0kER8RcASaOBN/ILy8ysNFo3HkP1NSBnTQRfBm5I2woAVgGn5BOSmVnpuPE4QyKQVAsMJen+uQwgw/ASZmbWTbT7QJmkS4FbgWOB3wMTnATMzCpLsTuCCUBtRLwuaXvgbuDa/MMyM7NSKTbExFsR8TpA+qBX1iEpzMysmyh2R/AhSbPT1wJ2LVgmIsblFpmZmZVEsUQwvtXyj/IKxMzMyqPYxDQPlCoQMzMrj2K9hu6SNDYdgrr1tg9JmizptPzCMzOzvBWrGjqTZEC4/yvpFWAF0AsYDDwN/Dwi7sw1QjMzy1WxqqH/Bi4ELpQ0GNiZZGiJJ5t7E5mZWfeWdYgJImIpyeQ0ZmYVrdqms8ycCMzMqtX8Z19h/rOvbDClZaUkh1wfEJN0hKQnJC2RdFEb20+VtEJSY/pzRp7xmJllcdDQ/hw0tD+zvnQAs750QJtlKmmu49zuCCTVAFOAw0kmqX9E0uyIWNyq6KyImJhXHGZmHdV6RNJKH6o661SVo4HLgQ+m+4hkpskPtbPbfsCSiHgmPcYtJA+otU4EZmZdWqUPVZ31juB64HyggWSS+SwGAC8ULDcBbb2bx0r6GPAkcH5EvNC6gKSzgLMABg3q/vVxZmZdSdZEsDoi/pDD+e8Cbo6ItyR9CfgF8PHWhSJiGjANoK6uLnKIw8ysQyppruOsjcX3S/qhpAMkfaT5p8g+y4BdCpYHputaRMTKiHgrXbwO2DdjPGZmXU53bUDOekfQXKVTV7AuaOPbe4FHgGGShpAkgOOBEwsLSNo5Ipani+OAxzPGY2ZWVpXUgJwpEUTEmI4eOCLWSZoIzAVqgOkRsUjSZKA+ImYDkySNA9YBrwCndvQ8ZmblUEkNyFl7DfUDLgM+lq56AJgcEavb2y8i5gBzWq27tOD1xcDFHQnYzMw6V9Y2gunAWuBz6c8aYEZeQZmZWelkbSPYNSKOLVi+QlJjDvGYmVmJZb0jeEPSQc0L6QNmb+QTkpmZlVLWO4KzgV+kbQXCDbtmZhUja6+hRmBvSX3T5TV5BmVmZqXTbiKQ9IWIuFHSV1utByAifpxjbGZmVgLF7gjel/7uk3cgZmZWHsWmqrwm/X1FacIxM7NSy9RrSNIPJPWVtIWke9PJZL6Qd3BmZpa/rN1HP5E2EH+aZN7iocDX8wrKzMxKJ2siaK5C+hRwW7GhJczMrPvI+hzB7yT9g+QhsrMl7QC8mV9YZmZWKpnuCCLiIuBAoC4i3gH+h2TaSTMz6+aKPUfw8Yi4T9IxBesKi9yeV2BmZlYaxaqG/g24DxjbxrbAicDMrNsr9hzBZenvL5YmHDMzK7WszxF8R9K2Bcvvl/QfGfY7QtITkpZIuqidcsdKCkl1GytjZmb5yNp99MiIeLV5ISJWAUe1t4OkGmAKcCQwHDhB0vA2yvUBzgPmZ4zFzMw6UdZEUCNpq+YFSb2BrdopD7AfsCQinomIt4FbaLun0beA7+PuqGZmZZE1EdwE3CvpdEmnA38EflFknwHACwXLTem6FpI+AuwSEb/PGIeZmXWyrPMRfF/SQuCwdNW3ImLu5pxYUg/gx2SY4EbSWcBZAIMGDdqc05qZWStZnywGeBxYFxH3SNpaUp+IWNtO+WXALgXLA9N1zfoAI4A/pc8m/C9gtqRxEVFfeKCImAZMA6irq4sOxGxmZkVk7TV0JvBr4Jp01QDgjiK7PQIMkzRE0pbA8cDs5o0RsToi+kfE4IgYDDwMbJAEzMwsX1nbCM4BRgNrACLiKWDH9naIiHXARGAuyd3ErRGxSNJkSeM2PWQzM+tMWauG3oqIt5uHl5DUk+TJ4nZFxBxgTqt1l26k7CEZYzEzs06U9Y7gAUnfAHpLOhy4Dbgrv7DMzKxUst4R/B/gDODvwJdIvuVfl1dQZmbd1eLla5hwzbyW5fG1AzhxVNfu7Vg0EaRPCC+KiN2Ba/MPycysexpfu96jUixevgag+yeCiHg3HS9oUEQ8X4qgzMy6oxNHDVrvQ7/wzqAry1o19H5gkaS/kUxKA0BEuPePmVk3lzURXJJrFGZmVjbFZijrBXwZGErSUHx9+nyAmZlViGLdR38B1JEkgSOBK3OPyMzMSqpY1dDwiBgJIOl64G/5h2RmZqVU7I7gneYXrhIyM6tMxe4I9pa0Jn0tkieL16SvIyL65hqdmZnlrtjk9TWlCsTMzMoj61hDZmZWoZwIzMyqnBOBmVmVcyIwM6tyuSYCSUekA9YtkXRRG9u/LOnvkhol/UXS8DzjMTOzDeWWCNLhq6eQPJE8HDihjQ/6X0XEyIioBX4A/DiveMzMrG153hHsByyJiGci4m3gFmB8YYGIWFOw+D4yTH9pZmadK+voo5tiAPBCwXITMKp1IUnnAF8FtgQ+nmM8ZmbWhrI3FkfElIjYlWQ6zH9vq4yksyTVS6pfsWJFaQM0M6tweSaCZcAuBcsD03UbcwtwdFsbImJaRNRFRN0OO+zQeRGamVmuieARYJikIZK2BI4HZhcWkDSsYPFTwFM5xmNmZm3IrY0gItZJmgjMBWqA6RGxSNJkoD4iZgMTJR1GMsrpKuCUvOIxM7O25dlYTETMAea0Wndpwevz8jy/mZkVV/bGYjMzKy8nAjOzKpdr1ZCZWbVbvHwNE66Z1+H9xtcO4MRRg3KIaENOBGZmORlfO2CT9lu8PBl0wYmgA9555x2ampp48803yx2K2Xp69erFwIED2WKLLcodipXBiaMGbdKH+abcQWyOikgETU1N9OnTh8GDByOp3OGYARARrFy5kqamJoYMGVLucMw2qiIai99880223357JwHrUiSx/fbb+07VuryKSASAk4B1Sf53ad1BxSQCMzPbNE4EnaSmpoba2lpGjBjB2LFjefXVVzvluDNnzmTixImdcqzBgwczcuRIamtrqa2t5aGHHuqU47bW2NjInDlzNrp9wYIFnH766eutO/roo9l///3XW3f55ZczYMCAlvd19uz1hqraJA0NDYwcOZKhQ4cyadIkIjacAmP16tWMHTuWvffemz333JMZM2ast33NmjUMHDhwvb/LYYcdxqpVqzY7PrNycCLoJL1796axsZHHHnuM7bbbjilTppQ7pDbdf//9NDY20tjYyIEHHphpn3Xr1nXoHMUSwXe+8x0mTZrUsvzqq6/S0NDA6tWreeaZZ9Yre/7559PY2Mhtt93GaaedxnvvvdehWFo7++yzufbaa3nqqad46qmnuPvuuzcoM2XKFIYPH87ChQv505/+xAUXXMDbb7/dsv2SSy7hYx/72Hr7nHTSSVx99dWbFZtZuVREr6FCV9y1iMUvrilesAOGf6Avl43dM3P5Aw44gEcffRSAv/3tb5x33nm8+eab9O7dmxkzZrDbbrsxc+ZMZs+ezeuvv87TTz/NZz7zGX7wgx8AMGPGDL773e+y7bbbsvfee7PVVlsBsHTpUk477TRefvlldthhB2bMmMGgQYM49dRT6d27NwsWLOCll15i+vTp3HDDDcybN49Ro0Yxc+bMjcba3jF79erFggULGD16NOeccw7nnHMOK1asYOutt+baa69l991357bbbuOKK66gpqaGfv36cc8993DppZfyxhtv8Je//IWLL76YCRMmtJxv7dq1PProo+y9994t626//XbGjh3LTjvtxC233MI3vvGNDeLcY4896NmzJy+//DI77rhj5r9FoeXLl7NmzZqWO4+TTz6ZO+64gyOPPHK9cpJYu3YtEcFrr73GdtttR8+eyX+VhoYG/vnPf3LEEUdQX1/fss+4ceM4+OCD+eY3v7lJsZmVU8UlgnJ79913uffee1uqPnbffXcefPBBevbsyT333MM3vvENfvOb3wDJN+cFCxaw1VZbsdtuu3HuuefSs2dPLrvsMhoaGujXrx9jxoxhn332AeDcc8/llFNO4ZRTTmH69OlMmjSJO+64A4BVq1Yxb948Zs+ezbhx4/jrX//Kddddx0c/+lEaGxupra0FYMyYMdTU1LDVVlsxf/78do/Z1NTEQw89RE1NDYceeihTp05l2LBhzJ8/n6985Svcd999TJ48mblz5zJgwABeffVVttxySyZPnkx9fT0///nPN3h/6uvrGTFixHrrbr75Zi699FJ22mknjj322DYTwfz58+nRowet56O4//77Of/88zcov/XWW29Q9bVs2TIGDhzYsjxw4ECWLdtwioyJEycybtw4PvCBD7B27VpmzZpFjx49eO+997jgggu48cYbueeee9bb5/3vfz9vvfUWK1euZPvtt9/gmGZdWcUlgo58c+9Mb7zxBrW1tSxbtow99tiDww8/HEjqm0855RSeeuopJPHOO++07HPooYfSr18/AIYPH85zzz3Hyy+/zCGHHNLygTdhwgSefPJJAObNm8ftt98OJFURF154Ycuxxo4diyRGjhzJTjvtxMiRIwHYc889Wbp0aUsiuP/+++nfv3/Lfu0d87jjjqOmpobXXnuNhx56iOOOO65l21tvvQXA6NGjOfXUU/nc5z7HMcccU/R9Wr58+Xof5v/85z956qmnOOigg5DEFltswWOPPdaSLH7yk59w44030qdPH2bNmrVBL5wxY8bQ2NhY9LwdMXfuXGpra7nvvvt4+umnOfzwwzn44IO54YYbOOqoo9ZLJoV23HFHXnzxRScC63YqLhGUS3Mbweuvv84nP/lJpkyZwqRJk7jkkksYM2YMv/3tb1m6dCmHHHJIyz7NVT6QNDZ3tC6+UPOxevTosd5xe/ToscnHfd/73gfAe++9x7bbbtvmB+7UqVOZP38+v//979l3331paGho95i9e/der1/9rbfeyqpVq1oeuFqzZg0333wz3/72t4GkjeBrX/vaRo/XkTuCAQMG0NTU1LLc1NTEgAEbDgEwY8YMLrroIiQxdOhQhgwZwj/+8Q/mzZvHgw8+yNVXX81rr73G22+/zTbbbMP3vvc9gJbqP7Puxo3FnWzrrbfmqquu4sorr2TdunWsXr265cOmvbr6ZqNGjeKBBx5g5cqVvPPOO9x2220t2w488EBuueUWAG666SYOPvjgzY43yzH79u3LkCFDWmKJCBYuXAjA008/zahRo5g8eTI77LADL7zwAn369GHt2rVtnm+PPfZgyZIlLcs333wzd999N0uXLmXp0qU0NDS0xJNF8x1B65+2ekTtvPPO9O3bl4cffpiI4IYbbmD8+PEblBs0aBD33nsvkNyxPPHEE3zoQx/ipptu4vnnn2fp0qX86Ec/4uSTT25JAhHBf//3fzN48ODMsZt1FbkmAklHSHpC0hJJF7Wx/auSFkt6VNK9kj6YZzylss8++7DXXntx8803c+GFF3LxxRezzz77ZPpmvvPOO3P55ZdzwAEHMHr0aPbYY4+WbT/72c+YMWMGe+21F7/85S/56U9/utmxZj3mTTfdxPXXX9/SpfLOO+8E4Otf/zojR45kxIgRHHjggey9996MGTOGxYsXU1tby6xZs9Y7zu67787q1atZu3YtS5cu5bnnnluv2+iQIUPo168f8+fP3+xra8vVV1/NGWecwdChQ9l1111bGoqnTp3K1KlTgaRX0EMPPcTIkSM59NBD+f73v79edVpbGhoa2H///Vsalc02V/OopYU/V9y1KJdzqa1+1J1yYKkGeBI4HGgimcP4hIhYXFBmDDA/Il6XdDZwSERMaPOAqbq6uijsrQHw+OOPr/eBaV3bT37yE/r06cMZZ5xR7lA6zXnnnce4ceM49NBDN9jmf5/WUb+a/zx3Nm7YkaGjPRgLSWqIiLq2tuX59WU/YElEPJMGcQswHmhJBBFxf0H5h4Ev5BiPdRFnn332elVelWDEiBFtJgGzTbGpo5ZuqjyrhgYALxQsN6XrNuZ04A85xmNdRK9evTjppJPKHUanOvPMM8sdgtkm6xIVmpK+ANQB/7aR7WcBZ0HSkNeWiPAAX9bl5FX1ataZ8rwjWAbsUrA8MF23HkmHAd8ExkXEW20dKCKmRURdRNS1fqAIkm+YK1eu9H8661Ka5yPo1atXuUMxa1eedwSPAMMkDSFJAMcDJxYWkLQPcA1wRES8tKknGjhwIE1NTaxYsWJz4jXrdM0zlJl1ZbklgohYJ2kiMBeoAaZHxCJJk4H6iJgN/BDYBrgtrdZ5PiLGdfRcW2yxhWeAMjPbRLm2EUTEHGBOq3WXFrw+LM/zm5lZcX6y2MysyjkRmJlVudyeLM6LpBXAc5u4e3/g5U4MpzvwNVcHX3N12Jxr/mBEbNjtkm6YCDaHpPqNPWJdqXzN1cHXXB3yumZXDZmZVTknAjOzKldtiWBauQMoA19zdfA1V4dcrrmq2gjMzGxD1XZHYGZmrTgRmJlVuYpMBBmmyNxK0qx0+3xJg8sQZqeqxmlBi11zQbljJYWkbt/VMMs1S/pc+rdeJOlXpY6xs2X4tz1I0v2SFqT/vo8qR5ydRdJ0SS9Jemwj2yXpqvT9eFTSRzb7pBFRUT8kA9w9DXwI2BJYCAxvVeYrwNT09fHArHLHXYJrHgNsnb4+uxquOS3XB/gzyQx4deWOuwR/52HAAuD96fKO5Y67BNc8DTg7fT0cWFruuDfzmj8GfAR4bCPbjyKZxEvA/iTT/W7WOSvxjqBlisyIeBtoniKz0HjgF+nrXwOHqnvPalP0miPi/oh4PV18mGR+iO4sy98Z4FvA94E3SxlcTrJc85nAlIhYBRCbMbx7F5HlmgPom77uB7xYwvg6XUT8GXilnSLjgRsi8TCwraSdN+eclZgIskyR2VImItYBq4HtSxJdPqpxWtCi15zeMu8SEb8vZWA5yvJ3/jDwYUl/lfSwpCNKFl0+slzz5cAXJDWRjHZ8bmlCK5uO/n8vqktMVWmlU2xa0EohqQfwY+DUModSaj1JqocOIbnr+7OkkRHxajmDytkJwMyIuFLSAcAvJY2IiPfKHVh3UYl3BFmmyGwpI6knye3kypJEl49Omxa0Gyl2zX2AEcCfJC0lqUud3c0bjLP8nZuA2RHxTkQ8CzxJkhi6qyzXfDpwK0BEzAN6kQzOVqky/X/viEpMBC1TZErakqQxeHarMrOBU9LXnwXui7QVppsqes0F04KOq4B6YyhyzRGxOiL6R8TgiBhM0i4yLiLqyxNup8jyb/sOkrsBJPUnqSp6poQxdrYs1/w8cCiApD1IEkElz1s7Gzg57T20P7A6IpZvzgErrmoosk2ReT3J7eMSkkaZ48sX8ebLeM2dMi1oV5HxmitKxmueC3xC0mLgXeDrEdFt73YzXvMFwLWSzidpOD61O3+xk3QzSTLvn7Z7XAZsARARU0naQY4ClgCvA1/c7HN24/fLzMw6QSVWDZmZWQc4EZiZVTknAjOzKudEYGZW5ZwIzMyqnBOB5U7Su5IaJT0m6S5J23by8ZemfeaR9NpGyvSW9ICkGkmDJb2RxrRY0tT0SeSOnLNO0lXp60MkHViw7cuSTt6ca0qPc7mkrxUpM1PSZztwzMEbG9WyVblvS3qh9fspaaKk07Kez7oHJwIrhTciojYiRpA8t3FOGWI4Dbg9It5Nl5+OiFpgL5IRK4/uyMEioj4iJqWLhwAHFmybGhE3bG7AZXYXyYBvrU2n8sfyqTpOBFZq80gHyJK0q6S7JTVIelDS7un6nST9VtLC9OfAdP0dadlFks7q4Hk/D9zZemU66OBDwND02/J9+tecDYPS8x6X3s0slPTndN0hkn6nZC6LLwPnp3cYBzd/k5e0u6S/NZ8rPf7f09f7pncoDZLmqsjokZLOlPRIGsNvJG1dsPkwSfWSnpT06bR8jaQfpvs8KulLHXmzIuLhtp5WTUewXSqprSRh3ZQTgZWMpBqSoQCan/qdBpwbEfsCXwOuTtdfBTwQEXuTjMu+KF1/Wlq2DpgkKdOIsenQBB+KiKVtbNs6jenvwM+AX0TEXsBNaRwAlwKfTONZ72ns9JhTgZ+kdz0PFmz7B7ClpCHpqgnALElbpOf6bHo904FvF7mM2yPio2kMj5OMr9NsMMm3908BUyX1SrevjoiPAh8FziyIo/naPyBpTpHztqUeOHgT9rMuquKGmLAuqbekRpI7gceBP0rahqQ6pXnIC4Ct0t8fB04GSKtyVqfrJ0n6TPp6F5LB1LIMn9AfeLXVul3TmAK4MyL+IOmXwDHp9l8CP0hf/xWYKelW4PYM5yt0K0kC+F76ewKwG8mAeH9Mr70GKDZWzAhJ/wFsSzJUyNzCc6QjbT4l6Rlgd+ATwF4F7Qf9SN6vJ5t3iogXSYYq6KiX0nNYhXAisFJ4IyJq02/fc0naCGYCr6b19EVJOgQ4DDggIl6X9CeSwcUynb+Nsk9nPXdEfFnSKJJv3A2S9s14XoBZJMnu9uRQ8ZSkkcCiiDigA8eZCRwdEQslnUo6sFxziK1DJpm96tyIKEwYqHOmZe1F8p5ahXDVkJVMWr88iWSQsNeBZyUdBy3zsO6dFr2XZDrN5rrufiTfaFelSWB3kmGls553FVCTVpm05yH+NQDh54EH0xh2jYj5EXEpyaiWu7Taby3JsNdtnftpksHfLiFJCgBPADsoGTsfSVtI2rNIbH2A5Wm10udbbTtOUg9Ju5JM6fgEScI9Oy2PpA9Lel+Rc2T1YaBozyPrPpwIrKQiYgHwKMlkIp8HTpe0kKQdoHkKwvOAMWnDagNJr567gZ6SHiepZnm4g6f+T+CgImXOBb4o6VHgpDQOgB9K+nva7fIhknlzC90FfKa5sbiN484CvsC/xsx/m2T48++n195IQa+jjbgEmE9STfWPVtueB/5GMuvclyPiTeA6YDHwX2nc19CqBqC9NgJJP1Ay8uXWkpokXV6weTTwxyLxWjfi0UetKiiZtvL8iDip3LF0Z0rmtfiq38fK4jsCqwoR8V/A/WnPJdt0/UnuTqyC+I7AzKzK+Y7AzKzKORGYmVU5JwIzsyrnRGBmVuWcCMzMqtz/B34ouZ+JFtTBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "PrecisionRecallDisplay.from_predictions(\n",
    "    y_true=test[\"label_num\"],\n",
    "    y_pred=score_gb,\n",
    "    name=\"RandomForest\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8cd036d",
   "metadata": {},
   "source": [
    "```{note}\n",
    "正則化としてearly stoppingを使う場合には `n_iter_no_change` と `validation_fraction` を設定します。\n",
    "\n",
    "    gb = GradientBoostingClassifier(\n",
    "        n_estimators=1000,\n",
    "        random_state=0,\n",
    "        learning_rate=0.1,\n",
    "        # early stoppingのための設定\n",
    "        validation_fraction=0.1,\n",
    "        n_iter_no_change=3,\n",
    "    )\n",
    "\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}