{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lab 5: logistic regression. ROC. AUC. One-hot encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1\n",
    "Consider the data from a health insurance company. Based on the characteristics of patients `Members.csv` (age, gender) and data on receiving medical care in the previous year `Claims_Y1.csv` (medical institution, doctor, type of problem, number of days of hospitalization, date, etc.), you need to predict the fact of hospitalization for at least 1 day in the next year `DaysInHospital_Y2.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "from sklearn import *\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the data. Use `MemberID` as the value for parameter `index_col`. \n",
    "\n",
    "* data from the 'DaysInHospital_Y2.csv' table assign to the `days2` variable\n",
    "* data from the 'Members.csv' table assign to the `m` variable\n",
    "* data from the 'Claims_Y1.csv' table assign to the `claims` variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# place for code\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2\n",
    "To anonymize data, the organizer provided approximate information about patients, for example, the age column shows age groups: '0-9', '10-19', '20-29', '30-39', '40-49', '50-59', '60-69', '70-79', '80+'. Convert string attributes to quantitative ones and replace the missing values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = pd.notnull(m.AgeAtFirstClaim)\n",
    "m.loc[i,'AgeAtFirstClaim'] = m.loc[i,'AgeAtFirstClaim'].apply(lambda s: s.split('-')[0] if s!='80+' else '80')\n",
    "m.loc[i,'AgeAtFirstClaim'] = m.loc[i,'AgeAtFirstClaim'].apply(lambda s: int(s))\n",
    "\n",
    "m.AgeAtFirstClaim = m.AgeAtFirstClaim.fillna(value=-1)\n",
    "\n",
    "m.Sex = m.Sex.fillna(value='N')\n",
    "\n",
    "claims.CharlsonIndex = claims.CharlsonIndex.map({'0':0, '1-2':1, '3-4':3, '5+':5})\n",
    "claims.LengthOfStay = claims.LengthOfStay.fillna(value=0)\n",
    "claims.LengthOfStay = claims.LengthOfStay.map({0:0, '1 day':1, '2 days':2, '3 days':3, '4 days':4,\\\n",
    "    '5 days':5, '6 days':6, '1- 2 weeks':10, '2- 4 weeks':21, '4- 8 weeks':42, '26+ weeks':182})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3\n",
    "Construct features using `claims` dta: \n",
    "* `f_Charlson` — the maximum index of Calson comorbidity for a patient (`CharlsonIndex` in a table `claims`)\n",
    "* `f_LengthOfStay` — the total number of days of hospitalization last year (`LengthOfStay` in a table `claims`) \n",
    "\n",
    "*Functions that can be useful: `.groupby(['MemberID']), .max ().sum()`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f_Charlson = # place for code\n",
    "f_LengthOfStay = # place for code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4\n",
    "\n",
    "Let's create a matrix of features with columns: `f_Charlson`, `f_LengthOfStay`, patient's age, `ClaimsTruncated` (whether there were too many cases of medical care):\n",
    "\n",
    "*Functions that can be useful: `.join()`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = days2\n",
    "data = data.join(f_Charlson)\n",
    "\n",
    "# place for code\n",
    "data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5\n",
    "Create a function that will divide the sample into two parts `dataTrain` and `dataTest`, train logistic regression on `dataTrain`, apply it to `dataTest`, build an error curve and calculate the area under it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calcAUC(data):\n",
    "    dataTrain, dataTest = model_selection.train_test_split(data, test_size=0.5, random_state=1)\n",
    "    model = linear_model.LogisticRegression()\n",
    "    model.fit( dataTrain.loc[:, dataTrain.columns != 'DaysInHospital'], dataTrain.DaysInHospital )\n",
    "    predictionProb = model.predict_proba( dataTest.loc[:, dataTest.columns != 'DaysInHospital'] )\n",
    "    fpr, tpr, _ = metrics.roc_curve(dataTest['DaysInHospital'], predictionProb[:,1])\n",
    "    plt.figure()\n",
    "    plt.plot(fpr, tpr, color='darkorange', lw=2)\n",
    "    plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\n",
    "    plt.show()\n",
    "    print( metrics.roc_auc_score(dataTest['DaysInHospital'], predictionProb[:,1]) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6\n",
    "Apply this function to `data`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7\n",
    "Logistic regression accepts only quantitative features as input.\n",
    "\n",
    "Add the patient's gender to our data using one hot encoding:\n",
    "\n",
    "*Functions that can be useful: `pd.get_dummies(m.Sex, prefix='pol')`*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8\n",
    "Try applying one hot encoding to the existing features in `data2` or creating new features using the `claims` array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Who built the feature matrix for which the logistic regression works with the best quality: +5 bonus points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
