{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# We will demonstrate skpref on discrete choice data\n", "\n", "We wil use the swissmetro dataset available to download on https://transp-or.epfl.ch/pythonbiogeme/examples_swissmetro.html\n", "This dataset tracks 470 respondents on which transportation alternative they have taken. There are 3 options in general: train, car and swissmetro.\n", "More details on the original use of the dataset can be found here: http://strc.ch/2001/bierlaire1.pdf\n", "\n", "Since at the moment of writing skpref still didn't have a discrete choice model interfaced, we will reduce the discrete choices to pairwise comparisons.\n", "\n", "In this notebook we will:\n", "\n", "* Start by tranforming the original swissmetro dataset into something that skpref can handle.\n", "* Fit a logistic regression using the data and the `ClassificationReducer()` method.\n", "* Fit a Bradley-Terry model using reduction to pairwise comparisons.\n", "* Show how two different aggregation methods for going from a pairwise comparison model to a discrete choice model work.\n", "* Show an example using `GridSearchCV()` and how to specify aggregation methods in `GridSearchCV()`\n", "* Show some of the evaluation methods that can be applied using skpref" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "pd.options.display.max_columns = 999\n", "import numpy as np\n", "import sys\n", "sys.path.insert(0, \"../..\")\n", "from skpref.base import ClassificationReducer\n", "from skpref.random_utility import BradleyTerry\n", "from skpref.task import ChoiceTask\n", "from skpref.metrics import f1_score, log_loss, log_loss_compare_with_t_test\n", "from skpref.utils import nice_print_results\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.preprocessing import OneHotEncoder\n", "from copy import deepcopy\n", "from sklearn.linear_model import LogisticRegression\n", "from skpref.model_selection import GridSearchCV" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
GROUPSURVEYSPIDPURPOSEFIRSTTICKETWHOLUGGAGEAGEMALEINCOMEGAORIGINDESTTRAIN_AVCAR_AVSM_AVTRAIN_TTTRAIN_COTRAIN_HESM_TTSM_COSM_HESM_SEATSCAR_TTCAR_COCHOICE
0201110110302021111112481206352200117652
120111011030202111110348306049100117842
220111011030202111113048606758300117522
32011101103020211111034030635220072522
42011101103020211111303660634220090842
\n", "
" ], "text/plain": [ " GROUP SURVEY SP ID PURPOSE FIRST TICKET WHO LUGGAGE AGE MALE \\\n", "0 2 0 1 1 1 0 1 1 0 3 0 \n", "1 2 0 1 1 1 0 1 1 0 3 0 \n", "2 2 0 1 1 1 0 1 1 0 3 0 \n", "3 2 0 1 1 1 0 1 1 0 3 0 \n", "4 2 0 1 1 1 0 1 1 0 3 0 \n", "\n", " INCOME GA ORIGIN DEST TRAIN_AV CAR_AV SM_AV TRAIN_TT TRAIN_CO \\\n", "0 2 0 2 1 1 1 1 112 48 \n", "1 2 0 2 1 1 1 1 103 48 \n", "2 2 0 2 1 1 1 1 130 48 \n", "3 2 0 2 1 1 1 1 103 40 \n", "4 2 0 2 1 1 1 1 130 36 \n", "\n", " TRAIN_HE SM_TT SM_CO SM_HE SM_SEATS CAR_TT CAR_CO CHOICE \n", "0 120 63 52 20 0 117 65 2 \n", "1 30 60 49 10 0 117 84 2 \n", "2 60 67 58 30 0 117 52 2 \n", "3 30 63 52 20 0 72 52 2 \n", "4 60 63 42 20 0 90 84 2 " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "swissmetro = pd.read_csv(\"data/swissmetro.dat\", sep='\\t')\n", "swissmetro.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looking at the data we can see that each row represents a choice.\n", "The full explanations of variables can be found here: https://transp-or.epfl.ch/pythonbiogeme/examples/swissmetro/swissmetro.pdf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Changing the format of the tables for skpref\n", "\n", "\n", "1) In this table the availability of alternatives is marked by `TRAIN_AV`, `CAR_AV`, `SM_AV` which indicate with 1 if the alternative is available and 0 otherwise. We need to convert these to a column that contains a list of alternatives for each row.\n", "\n", "2) The choices are indicated by the `CHOICE` column, which contains 0 for unknown (we will dropping these), 1 for Train, 2 for Swissmetro and 3 for a Car usage. We need to name these explicitly. We could just create a list of alternatives called 1,2,3 in step 1 which would bypass step 2, however, we prefer the clarity of having the alternatives named explicitly.\n", "\n", "# Using only a subset of the available features\n", "Under normal circumstances we would use one-hot encoding on a lot of the binary features before training a serious model, however, to make the demo simple, we will only use the travel time and cost features. Users can of course do whatever feature transformations they like before fitting a model.\n", "To build a classifier based on travel time and costs, we first need to split some columns from the swissmetro dataset into a secondary table." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
merge_indexTravel TimeCostalternative
0011248Train
006352Swiss Metro
0011765Car
1110348Train
116049Swiss Metro
\n", "
" ], "text/plain": [ " merge_index Travel Time Cost alternative\n", "0 0 112 48 Train\n", "0 0 63 52 Swiss Metro\n", "0 0 117 65 Car\n", "1 1 103 48 Train\n", "1 1 60 49 Swiss Metro" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_vals = swissmetro[['TRAIN_TT', 'TRAIN_CO']].copy()\n", "train_vals.columns = ['Travel Time', 'Cost']\n", "train_vals.reset_index(inplace=True)\n", "train_vals['alternative'] = 'Train'\n", "swissmetro_vals = swissmetro[['SM_TT', 'SM_CO']].copy()\n", "swissmetro_vals.columns = ['Travel Time', 'Cost']\n", "swissmetro_vals.reset_index(inplace=True)\n", "swissmetro_vals['alternative'] = 'Swiss Metro'\n", "car_vals = swissmetro[['CAR_TT', 'CAR_CO']].copy()\n", "car_vals.columns = ['Travel Time', 'Cost']\n", "car_vals.reset_index(inplace=True)\n", "car_vals['alternative'] = 'Car'\n", "dummy_secondary_table = (\n", " train_vals.append(swissmetro_vals.append(car_vals))\n", ").sort_values('index')\n", "dummy_secondary_table.rename(columns={'index': 'merge_index'}, inplace=True)\n", "dummy_secondary_table.head()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
alternativeschosenmerge_index
0[Train, Car, Swiss Metro]Swiss Metro0
1[Train, Car, Swiss Metro]Swiss Metro1
2[Train, Car, Swiss Metro]Swiss Metro2
3[Train, Car, Swiss Metro]Swiss Metro3
4[Train, Car, Swiss Metro]Swiss Metro4
\n", "
" ], "text/plain": [ " alternatives chosen merge_index\n", "0 [Train, Car, Swiss Metro] Swiss Metro 0\n", "1 [Train, Car, Swiss Metro] Swiss Metro 1\n", "2 [Train, Car, Swiss Metro] Swiss Metro 2\n", "3 [Train, Car, Swiss Metro] Swiss Metro 3\n", "4 [Train, Car, Swiss Metro] Swiss Metro 4" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "binary_concats = (swissmetro.TRAIN_AV.astype(str) + \n", " swissmetro.CAR_AV.astype(str) + \n", " swissmetro.SM_AV.astype(str)\n", " )\n", "\n", "alts = []\n", "for i in binary_concats.values:\n", " if i == '111':\n", " alts.append(['Train', 'Car', 'Swiss Metro'])\n", " elif i == '100':\n", " alts.append(['Train'])\n", " elif i == '000':\n", " alts.append(['None'])\n", " elif i == '010':\n", " alts.append(['Car'])\n", " elif i == '001':\n", " alts.append(['Swiss Metro'])\n", " elif i == '101':\n", " alts.append(['Train', 'Swiss Metro'])\n", " elif i == '110':\n", " alts.append(['Train', 'Car'])\n", " elif i == '011':\n", " alts.append(['Car', 'Swiss Metro'])\n", " \n", "swissmetro['alternatives'] = alts\n", " \n", "swissmetro['chosen'] = np.where(swissmetro.CHOICE.values==1, 'Train',\n", " np.where(swissmetro.CHOICE.values==2, 'Swiss Metro',\n", " np.where(swissmetro.CHOICE.values==3, 'Car',\n", " 'unknown')))\n", "swissmetro = swissmetro[swissmetro.CHOICE != 0].copy()\n", "swissmetro = swissmetro.reset_index()[['alternatives', 'chosen', 'index']]\n", "swissmetro.rename(columns={'index': 'merge_index'}, inplace=True)\n", "swissmetro.head()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([list(['Train', 'Car', 'Swiss Metro']),\n", " list(['Train', 'Car', 'Swiss Metro']),\n", " list(['Train', 'Car', 'Swiss Metro']), ...,\n", " list(['Train', 'Car', 'Swiss Metro']),\n", " list(['Train', 'Car', 'Swiss Metro']),\n", " list(['Train', 'Car', 'Swiss Metro'])], dtype=object)" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "swissmetro.alternatives.values" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Fit a logistic regression\n", "This will fit a logistic regression that uses only travel time and cost as covariates, with the following formulation for observation $i$ and $a \\in \\{\\text{Car, Train, Swiss Metro}\\}$:\n", "$$P(Y_i = a) = logit(\\lambda_{\\text{a}} + \\beta_1 (\\text{Travel Time})_a + \\beta_2(\\text{Cost})_a)$$ \n", "\n", "Below we showcase how the ChoiceTask wrapper can deal with creating this reduction" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "train, test = train_test_split(swissmetro, random_state=1, test_size=0.1)\n", "\n", "swiss_metro_train = ChoiceTask(train, 'alternatives', 'chosen',\n", " features_to_use=['Travel Time', 'Cost'],\n", " secondary_table=dummy_secondary_table,\n", " secondary_to_primary_link={\n", " 'merge_index': 'merge_index',\n", " 'alternative': 'alternatives'\n", " }\n", " )\n", "\n", "swiss_metro_test = ChoiceTask(test, 'alternatives', 'chosen',\n", " features_to_use=['Travel Time', 'Cost'],\n", " secondary_table=dummy_secondary_table,\n", " secondary_to_primary_link={\n", " 'merge_index': 'merge_index',\n", " 'alternative': 'alternatives'\n", " }\n", " )" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "my_log_red = ClassificationReducer(LogisticRegression(solver='lbfgs'))\n", "my_log_red.fit_task(swiss_metro_train)\n", "log_reg_preds = my_log_red.predict_proba_task(swiss_metro_test,\n", " ['Swiss Metro','Train', 'Car'])" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Swiss Metro [0.49 0.56 0.4 ... 0.47 0.35 0.51]\n", "Train [0.3 0.45 0.17 ... 0.36 0.29 0.26]\n", "Car [0.33 0. 0.2 ... 0.45 0.36 0.25]\n" ] } ], "source": [ "nice_print_results(log_reg_preds)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['Swiss Metro' 'Swiss Metro' 'Swiss Metro' 'Swiss Metro' 'Swiss Metro']\n", "[array(['Car', 'Train'], dtype='\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
merge_indexTravel TimeCost
alternative
Car5363.5123.79520978.742077
Swiss Metro5363.587.466350670.340697
Train5363.5166.626025514.335477
\n", "" ], "text/plain": [ " merge_index Travel Time Cost\n", "alternative \n", "Car 5363.5 123.795209 78.742077\n", "Swiss Metro 5363.5 87.466350 670.340697\n", "Train 5363.5 166.626025 514.335477" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dummy_secondary_table.groupby('alternative').mean()" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "
Multinomial Logit Model Regression Results
Dep. Variable: CHOICE No. Observations: 8,878
Model: Multinomial Logit Model Df Residuals: 8,873
Method: MLE Df Model: 5
Date: Wed, 01 Mar 2023 Pseudo R-squ.: 0.226
Time: 16:11:21 Pseudo R-bar-squ.: 0.225
AIC: 9,534.703 Log-Likelihood: -4,762.351
BIC: 9,570.159 LL-Null: -6,153.761
\n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "
coef std err z P>|z| [0.025 0.975]
Cost 0.0003 3.11e-05 8.763 0.000 0.000 0.000
Travel Time -0.0116 0.001 -22.819 0.000 -0.013 -0.011
Car 0.3031 0.046 6.635 0.000 0.214 0.393
Swiss Metro 0.1378 0.048 2.870 0.004 0.044 0.232
Train -0.4409 0.048 -9.159 0.000 -0.535 -0.347
" ], "text/plain": [ "\n", "\"\"\"\n", " Multinomial Logit Model Regression Results \n", "===================================================================================\n", "Dep. Variable: CHOICE No. Observations: 8,878\n", "Model: Multinomial Logit Model Df Residuals: 8,873\n", "Method: MLE Df Model: 5\n", "Date: Wed, 01 Mar 2023 Pseudo R-squ.: 0.226\n", "Time: 16:11:21 Pseudo R-bar-squ.: 0.225\n", "AIC: 9,534.703 Log-Likelihood: -4,762.351\n", "BIC: 9,570.159 LL-Null: -6,153.761\n", "===============================================================================\n", " coef std err z P>|z| [0.025 0.975]\n", "-------------------------------------------------------------------------------\n", "Cost 0.0003 3.11e-05 8.763 0.000 0.000 0.000\n", "Travel Time -0.0116 0.001 -22.819 0.000 -0.013 -0.011\n", "Car 0.3031 0.046 6.635 0.000 0.214 0.393\n", "Swiss Metro 0.1378 0.048 2.870 0.004 0.044 0.232\n", "Train -0.4409 0.048 -9.159 0.000 -0.535 -0.347\n", "===============================================================================\n", "\"\"\"" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "my_BT_red_feats.bt_with_feats.get_statsmodels_summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fit Bradley-Terry model with GridSearch\n", "The models we have fitted above also have hyperparameters, such as the method of gradient descent or regularisation. To optimise the hyperparameter selection, we can use `GridSearchCV()`. `GridSearchCV()` tries out a series of hyperparameter combinations and runs a k-fold cross-validation on an accuracy metric determined by the user to check which ones have performed best.\n", "\n", "In this section we will show how aggregation works with GridSearch, it is possible to just add `aggregation_method` in the `predict_proba_task` and the ouptuts work as expected. Note in this example we have chosen a very different alpha to the models above so that there is some slight difference in the ouptuts to two decimal places, so that we can see that different parameters were learned." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The model with the best parameters was:\n", "BradleyTerry(alpha=100, method='BFGS')\n", "With a score of -0.5345453019698206\n", "All the trials results summarised in descending score\n", " alpha method mean_test_score\n", "0 100 BFGS -0.534545\n", "1 1000 BFGS -0.546477\n" ] } ], "source": [ "to_tune = {'alpha': [100,1000], 'method': ['BFGS']}\n", "gs_bt = GridSearchCV(BradleyTerry(), to_tune, cv=3, scoring='neg_log_loss')\n", "gs_bt.fit_task(swiss_metro_train_BT_feats)\n", "gs_bt.inspect_results()" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Swiss Metro [0.63 0.75 0.74 ... 0.44 0.4 0.77]\n", "Train [0.06 0.25 0.04 ... 0.07 0.09 0.05]\n", "Car [0.31 0. 0.22 ... 0.49 0.51 0.18]\n" ] } ], "source": [ "nice_print_results(gs_bt.predict_proba_task(\n", " swiss_metro_test,['Swiss Metro','Train', 'Car'],\n", " aggregation_method='independent transitive'))" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "
Multinomial Logit Model Regression Results
Dep. Variable: CHOICE No. Observations: 8,878
Model: Multinomial Logit Model Df Residuals: 8,873
Method: MLE Df Model: 5
Date: Wed, 01 Mar 2023 Pseudo R-squ.: 0.226
Time: 16:11:38 Pseudo R-bar-squ.: 0.225
AIC: 9,534.703 Log-Likelihood: -4,762.351
BIC: 9,570.159 LL-Null: -6,153.761
\n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "\n", " \n", "\n", "
coef std err z P>|z| [0.025 0.975]
Cost 0.0003 3.11e-05 8.763 0.000 0.000 0.000
Travel Time -0.0116 0.001 -22.819 0.000 -0.013 -0.011
Car 0.3031 0.046 6.635 0.000 0.214 0.393
Swiss Metro 0.1378 0.048 2.870 0.004 0.044 0.232
Train -0.4409 0.048 -9.159 0.000 -0.535 -0.347
" ], "text/plain": [ "\n", "\"\"\"\n", " Multinomial Logit Model Regression Results \n", "===================================================================================\n", "Dep. Variable: CHOICE No. Observations: 8,878\n", "Model: Multinomial Logit Model Df Residuals: 8,873\n", "Method: MLE Df Model: 5\n", "Date: Wed, 01 Mar 2023 Pseudo R-squ.: 0.226\n", "Time: 16:11:38 Pseudo R-bar-squ.: 0.225\n", "AIC: 9,534.703 Log-Likelihood: -4,762.351\n", "BIC: 9,570.159 LL-Null: -6,153.761\n", "===============================================================================\n", " coef std err z P>|z| [0.025 0.975]\n", "-------------------------------------------------------------------------------\n", "Cost 0.0003 3.11e-05 8.763 0.000 0.000 0.000\n", "Travel Time -0.0116 0.001 -22.819 0.000 -0.013 -0.011\n", "Car 0.3031 0.046 6.635 0.000 0.214 0.393\n", "Swiss Metro 0.1378 0.048 2.870 0.004 0.044 0.232\n", "Train -0.4409 0.048 -9.159 0.000 -0.535 -0.347\n", "===============================================================================\n", "\"\"\"" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "gs_bt.best_estimator_.bt_with_feats.get_statsmodels_summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Evaluation methods\n", "In this section we show some of the evaluation methods available in skpref, specifically how to use `log_loss` and `log_loss_compare_with_t_test`. Please see the documentation for more details on how these work." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The F1 score of the Luce aggregation was 0.598\n", "The F1 score of the generic aggregation was 0.598\n" ] } ], "source": [ "print(f\"The F1 score of the Luce aggregation was \\\n", "{f1_score(swiss_metro_test.subset_vec, preds_outcome_Luce): .3}\")\n", "print(f\"The F1 score of the generic aggregation was \\\n", "{f1_score(swiss_metro_test.subset_vec, preds_outcome_ind_trans):.3}\")" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The log loss for each alternative in the Logistic Regression reduction was \n", " {'Swiss Metro_log_loss': 0.73, 'Train_log_loss': 0.44, 'Car_log_loss': 0.57}\n", "The log loss for each alternative in the Luce aggregation was \n", " {'Swiss Metro_log_loss': 0.7, 'Train_log_loss': 0.38, 'Car_log_loss': 0.52}\n", "The log loss for each alternative in the generic aggregation was \n", " {'Swiss Metro_log_loss': 0.71, 'Train_log_loss': 0.38, 'Car_log_loss': 0.52}\n", "The log loss for each alternative assigning random probability was \n", " {'Swiss Metro_log_loss': 0.8, 'Train_log_loss': 0.5, 'Car_log_loss': 0.61}\n" ] } ], "source": [ "random_probs = {\n", " 'Swiss Metro': np.ones(len(test)) * (1/3),\n", " 'Train': np.ones(len(test)) * (1/3),\n", " 'Car': np.ones(len(test)) * (1/3)\n", "}\n", "log_reg_preds\n", "print(f\"The log loss for each alternative in the Logistic Regression reduction was \\n \\\n", "{log_loss(swiss_metro_test.subset_vec, log_reg_preds)}\")\n", "print(f\"The log loss for each alternative in the Luce aggregation was \\n \\\n", "{log_loss(swiss_metro_test.subset_vec, preds)}\")\n", "print(f\"The log loss for each alternative in the generic aggregation was \\n \\\n", "{log_loss(swiss_metro_test.subset_vec, ind_trans_preds)}\")\n", "print(f\"The log loss for each alternative assigning random probability was \\n \\\n", "{log_loss(swiss_metro_test.subset_vec, random_probs)}\")" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The t-test for H0: Luce aggregation = Generic aggregation {'Swiss Metro': 0.02, 'Train': 0.74, 'Car': 0.27}\n", "The t-test for H0: Generic aggregation = random probability {'Swiss Metro': 0.0, 'Train': 0.0, 'Car': 0.0}\n", "The t-test for H0: Generic aggregation = Logistic Regression {'Swiss Metro': 0.18, 'Train': 0.0, 'Car': 0.0}\n", "The t-test for H0: Generic aggregation = random probability {'Swiss Metro': 0.0, 'Train': 0.0, 'Car': 0.0}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "../..\\skpref\\metrics\\_classification.py:254: RuntimeWarning: divide by zero encountered in log\n", " logged = np.log(predicted1[_alternative])\n", "../..\\skpref\\metrics\\_classification.py:258: RuntimeWarning: invalid value encountered in multiply\n", " np.nan_to_num(logged * binarized_outcome) +\n", "../..\\skpref\\metrics\\_classification.py:261: RuntimeWarning: divide by zero encountered in log\n", " logged2 = np.log(predicted2[_alternative])\n", "../..\\skpref\\metrics\\_classification.py:265: RuntimeWarning: invalid value encountered in multiply\n", " np.nan_to_num(logged2 * binarized_outcome) +\n" ] } ], "source": [ "print(f\"The t-test for H0: Luce aggregation = Generic aggregation \\\n", "{log_loss_compare_with_t_test(swiss_metro_test.subset_vec, preds, ind_trans_preds)}\")\n", "print(f\"The t-test for H0: Generic aggregation = random probability \\\n", "{log_loss_compare_with_t_test(swiss_metro_test.subset_vec, ind_trans_preds, random_probs)}\")\n", "print(f\"The t-test for H0: Generic aggregation = Logistic Regression \\\n", "{log_loss_compare_with_t_test(swiss_metro_test.subset_vec, ind_trans_preds, log_reg_preds)}\")\n", "print(f\"The t-test for H0: Generic aggregation = random probability \\\n", "{log_loss_compare_with_t_test(swiss_metro_test.subset_vec, ind_trans_preds, random_probs)}\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.11" }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }