{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ガウス過程\n",
    "\n",
    "PHYSBOではガウス過程回帰を実行しながらベイズ最適化を行なっています。\n",
    "\n",
    "そのため、学習データが与えられた際にガウス過程回帰を実行することもでき、学習済みモデルを利用したテストデータの予測も行うことができます。\n",
    "\n",
    "ここでは、その手順について紹介します。\n",
    "\n",
    "\n",
    "## 探索候補データの準備\n",
    "\n",
    "これまでのマニュアルでも利用していたs5-210.csvを用いてガウス過程を実行します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import physbo\n",
    "\n",
    "import os\n",
    "import urllib2\n",
    "import ssl\n",
    "import numpy as np\n",
    "\n",
    "ssl._create_default_https_context = ssl._create_unverified_context\n",
    "        \n",
    "def load_data():\n",
    "    A =  np.asarray(np.loadtxt('data/s5-210.csv',skiprows=1, delimiter=',') )\n",
    "    X = A[:,0:3]\n",
    "    t  = -A[:,3]\n",
    "    return X, t\n",
    "\n",
    "X, t = load_data()\n",
    "X = physbo.misc.centering( X )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 学習データの定義\n",
    "\n",
    "対象データのうち、ランダムに選んだ1割をトレーニングデータとして利用し、別のランダムに選んだ1割をテストデータとして利用します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ntrain = 1798\n",
      "Ntest = 1798\n"
     ]
    }
   ],
   "source": [
    "N = len(t)\n",
    "Ntrain = int(N*0.1)\n",
    "Ntest = min(int(N*0.1), N-Ntrain)\n",
    "\n",
    "id_all   = np.random.choice(N, N, replace=False)\n",
    "id_train  = id_all[0:Ntrain]\n",
    "id_test = id_all[Ntrain:Ntrain+Ntest]\n",
    "\n",
    "X_train = X[id_train]\n",
    "X_test = X[id_test]\n",
    "\n",
    "t_train = t[id_train]\n",
    "t_test = t[id_test]\n",
    "\n",
    "print \"Ntrain =\", Ntrain\n",
    "print \"Ntest =\", Ntest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ガウス過程の学習と予測\n",
    "\n",
    "以下のプロセスでガウス過程を学習し、テストデータの予測を行います。\n",
    "\n",
    "1. ガウス過程のモデルを生成します。\n",
    "\n",
    "2. X_train（学習データのパラメータ）, t_train（学習データの目的関数値）を用いてモデルを学習します。\n",
    "\n",
    "3. 学習されたモデルを用いてテストデータ(X_test)に対する予測を実行します。\n",
    "\n",
    "共分散の定義(ガウシアン)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "cov = physbo.gp.cov.gauss( X_train.shape[1],ard = False )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "平均の定義"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean = physbo.gp.mean.const()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "尤度関数の定義(ガウシアン)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "lik = physbo.gp.lik.gauss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ガウス過程モデルの生成"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "gp = physbo.gp.model(lik=lik,mean=mean,cov=cov)\n",
    "config = physbo.misc.set_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ガウス過程モデルを学習"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start the initial hyper parameter searching ...\n",
      "Done\n",
      "\n",
      "Start the hyper parameter learning ...\n",
      "0 -th epoch marginal likelihood 16805.225990633342\n",
      "50 -th epoch marginal likelihood 5353.525281891333\n",
      "100 -th epoch marginal likelihood 3005.100784225642\n",
      "150 -th epoch marginal likelihood 1590.9517857305727\n",
      "200 -th epoch marginal likelihood 246.9563975152605\n",
      "250 -th epoch marginal likelihood -566.216332288157\n",
      "300 -th epoch marginal likelihood -1173.7954898625985\n",
      "350 -th epoch marginal likelihood -1548.6217728092386\n",
      "400 -th epoch marginal likelihood -1736.39439620689\n",
      "450 -th epoch marginal likelihood -1829.1302788508322\n",
      "500 -th epoch marginal likelihood -1866.3124105898187\n",
      "Done\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gp.fit(X_train, t_train, config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "学習されたガウス過程におけるパラメタを出力"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "likelihood parameter =   [-2.93775312]\n",
      "mean parameter in GP prior:  [-1.06002832]\n",
      "covariance parameter in GP prior:  [-0.86850454 -2.48529293]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gp.print_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "テストデータの平均値（予測値）および分散を計算"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "gp.prepare(X_train, t_train)\n",
    "fmean = gp.get_post_fmean(X_train, X_test)\n",
    "fcov = gp.get_post_fcov(X_train, X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "予測の結果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-1.05105624, -1.01027414, -1.00525062, ..., -0.99285067,\n",
       "       -0.97720778, -0.99763675])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fmean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "分散の結果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.00052229, 0.00039968, 0.00052004, ..., 0.00039218, 0.00044258,\n",
       "       0.00033381])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fcov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "予測値の平均二乗誤差の出力"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.004274437799555673"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean((fmean-t_test)**2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 訓練済みモデルによる予測\n",
    "\n",
    "学習済みモデルのパラメタをgp_paramsとして読み出し、これを用いた予測を行います。\n",
    "\n",
    "gp_paramsおよび学習データ(X_train, t_train)を記憶しておくことで、訓練済みモデルによる予測が可能となります。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "学習されたパラメタを準備(学習の直後に行う必要あり)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-2.93775312, -1.06002832, -0.86850454, -2.48529293])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#学習したパラメタを１次元配列として準備\n",
    "gp_params =  np.append(np.append(gp.lik.params, gp.prior.mean.params), gp.prior.cov.params)\n",
    "\n",
    "gp_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "学習に利用したモデルと同様のモデルをgpとして準備"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#共分散の定義 (ガウシアン)\n",
    "cov = physbo.gp.cov.gauss( X_train.shape[1],ard = False )\n",
    "\n",
    "#平均の定義\n",
    "mean = physbo.gp.mean.const()\n",
    "\n",
    "#尤度関数の定義 (ガウシアン)\n",
    "lik = physbo.gp.lik.gauss()\n",
    "\n",
    "#ガウス過程モデルの生成\n",
    "gp = physbo.gp.model(lik=lik,mean=mean,cov=cov)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "学習済みのパラメタをモデルに入力し予測を実行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#学習済みのパラメタをガウス過程に入力\n",
    "gp.set_params(gp_params)\n",
    "\n",
    "\n",
    "#テストデータの平均値（予測値）および分散を計算\n",
    "gp.prepare(X_train, t_train)\n",
    "fmean = gp.get_post_fmean(X_train, X_test)\n",
    "fcov = gp.get_post_fcov(X_train, X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "予測の結果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-1.05105624, -1.01027414, -1.00525062, ..., -0.99285067,\n",
       "       -0.97720778, -0.99763675])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fmean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "分散の結果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.00052229, 0.00039968, 0.00052004, ..., 0.00039218, 0.00044258,\n",
       "       0.00033381])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fcov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "予測値の平均二乗誤差の出力"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.004274437799555673"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean((fmean-t_test)**2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}