We will demonstrate skpref on discrete choice data¶
We wil use the swissmetro dataset available to download on https://transp-or.epfl.ch/pythonbiogeme/examples_swissmetro.html This dataset tracks 470 respondents on which transportation alternative they have taken. There are 3 options in general: train, car and swissmetro. More details on the original use of the dataset can be found here: http://strc.ch/2001/bierlaire1.pdf
Since at the moment of writing skpref still didn’t have a discrete choice model interfaced, we will reduce the discrete choices to pairwise comparisons.
In this notebook we will:
Start by tranforming the original swissmetro dataset into something that skpref can handle.
Fit a logistic regression using the data and the
ClassificationReducer()method.Fit a Bradley-Terry model using reduction to pairwise comparisons.
Show how two different aggregation methods for going from a pairwise comparison model to a discrete choice model work.
Show an example using
GridSearchCV()and how to specify aggregation methods inGridSearchCV()Show some of the evaluation methods that can be applied using skpref
[1]:
import pandas as pd
pd.options.display.max_columns = 999
import numpy as np
import sys
sys.path.insert(0, "../..")
from skpref.base import ClassificationReducer
from skpref.random_utility import BradleyTerry
from skpref.task import ChoiceTask
from skpref.metrics import f1_score, log_loss, log_loss_compare_with_t_test
from skpref.utils import nice_print_results
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from copy import deepcopy
from sklearn.linear_model import LogisticRegression
from skpref.model_selection import GridSearchCV
[2]:
swissmetro = pd.read_csv("data/swissmetro.dat", sep='\t')
swissmetro.head()
[2]:
| GROUP | SURVEY | SP | ID | PURPOSE | FIRST | TICKET | WHO | LUGGAGE | AGE | MALE | INCOME | GA | ORIGIN | DEST | TRAIN_AV | CAR_AV | SM_AV | TRAIN_TT | TRAIN_CO | TRAIN_HE | SM_TT | SM_CO | SM_HE | SM_SEATS | CAR_TT | CAR_CO | CHOICE | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 3 | 0 | 2 | 0 | 2 | 1 | 1 | 1 | 1 | 112 | 48 | 120 | 63 | 52 | 20 | 0 | 117 | 65 | 2 |
| 1 | 2 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 3 | 0 | 2 | 0 | 2 | 1 | 1 | 1 | 1 | 103 | 48 | 30 | 60 | 49 | 10 | 0 | 117 | 84 | 2 |
| 2 | 2 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 3 | 0 | 2 | 0 | 2 | 1 | 1 | 1 | 1 | 130 | 48 | 60 | 67 | 58 | 30 | 0 | 117 | 52 | 2 |
| 3 | 2 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 3 | 0 | 2 | 0 | 2 | 1 | 1 | 1 | 1 | 103 | 40 | 30 | 63 | 52 | 20 | 0 | 72 | 52 | 2 |
| 4 | 2 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 3 | 0 | 2 | 0 | 2 | 1 | 1 | 1 | 1 | 130 | 36 | 60 | 63 | 42 | 20 | 0 | 90 | 84 | 2 |
Looking at the data we can see that each row represents a choice. The full explanations of variables can be found here: https://transp-or.epfl.ch/pythonbiogeme/examples/swissmetro/swissmetro.pdf
Changing the format of the tables for skpref¶
In this table the availability of alternatives is marked by
TRAIN_AV,CAR_AV,SM_AVwhich indicate with 1 if the alternative is available and 0 otherwise. We need to convert these to a column that contains a list of alternatives for each row.The choices are indicated by the
CHOICEcolumn, which contains 0 for unknown (we will dropping these), 1 for Train, 2 for Swissmetro and 3 for a Car usage. We need to name these explicitly. We could just create a list of alternatives called 1,2,3 in step 1 which would bypass step 2, however, we prefer the clarity of having the alternatives named explicitly.
Using only a subset of the available features¶
Under normal circumstances we would use one-hot encoding on a lot of the binary features before training a serious model, however, to make the demo simple, we will only use the travel time and cost features. Users can of course do whatever feature transformations they like before fitting a model. To build a classifier based on travel time and costs, we first need to split some columns from the swissmetro dataset into a secondary table.
[3]:
train_vals = swissmetro[['TRAIN_TT', 'TRAIN_CO']].copy()
train_vals.columns = ['Travel Time', 'Cost']
train_vals.reset_index(inplace=True)
train_vals['alternative'] = 'Train'
swissmetro_vals = swissmetro[['SM_TT', 'SM_CO']].copy()
swissmetro_vals.columns = ['Travel Time', 'Cost']
swissmetro_vals.reset_index(inplace=True)
swissmetro_vals['alternative'] = 'Swiss Metro'
car_vals = swissmetro[['CAR_TT', 'CAR_CO']].copy()
car_vals.columns = ['Travel Time', 'Cost']
car_vals.reset_index(inplace=True)
car_vals['alternative'] = 'Car'
dummy_secondary_table = (
train_vals.append(swissmetro_vals.append(car_vals))
).sort_values('index')
dummy_secondary_table.rename(columns={'index': 'merge_index'}, inplace=True)
dummy_secondary_table.head()
[3]:
| merge_index | Travel Time | Cost | alternative | |
|---|---|---|---|---|
| 0 | 0 | 112 | 48 | Train |
| 0 | 0 | 63 | 52 | Swiss Metro |
| 0 | 0 | 117 | 65 | Car |
| 1 | 1 | 103 | 48 | Train |
| 1 | 1 | 60 | 49 | Swiss Metro |
[4]:
binary_concats = (swissmetro.TRAIN_AV.astype(str) +
swissmetro.CAR_AV.astype(str) +
swissmetro.SM_AV.astype(str)
)
alts = []
for i in binary_concats.values:
if i == '111':
alts.append(['Train', 'Car', 'Swiss Metro'])
elif i == '100':
alts.append(['Train'])
elif i == '000':
alts.append(['None'])
elif i == '010':
alts.append(['Car'])
elif i == '001':
alts.append(['Swiss Metro'])
elif i == '101':
alts.append(['Train', 'Swiss Metro'])
elif i == '110':
alts.append(['Train', 'Car'])
elif i == '011':
alts.append(['Car', 'Swiss Metro'])
swissmetro['alternatives'] = alts
swissmetro['chosen'] = np.where(swissmetro.CHOICE.values==1, 'Train',
np.where(swissmetro.CHOICE.values==2, 'Swiss Metro',
np.where(swissmetro.CHOICE.values==3, 'Car',
'unknown')))
swissmetro = swissmetro[swissmetro.CHOICE != 0].copy()
swissmetro = swissmetro.reset_index()[['alternatives', 'chosen', 'index']]
swissmetro.rename(columns={'index': 'merge_index'}, inplace=True)
swissmetro.head()
[4]:
| alternatives | chosen | merge_index | |
|---|---|---|---|
| 0 | [Train, Car, Swiss Metro] | Swiss Metro | 0 |
| 1 | [Train, Car, Swiss Metro] | Swiss Metro | 1 |
| 2 | [Train, Car, Swiss Metro] | Swiss Metro | 2 |
| 3 | [Train, Car, Swiss Metro] | Swiss Metro | 3 |
| 4 | [Train, Car, Swiss Metro] | Swiss Metro | 4 |
[5]:
swissmetro.alternatives.values
[5]:
array([list(['Train', 'Car', 'Swiss Metro']),
list(['Train', 'Car', 'Swiss Metro']),
list(['Train', 'Car', 'Swiss Metro']), ...,
list(['Train', 'Car', 'Swiss Metro']),
list(['Train', 'Car', 'Swiss Metro']),
list(['Train', 'Car', 'Swiss Metro'])], dtype=object)
Fit a logistic regression¶
This will fit a logistic regression that uses only travel time and cost as covariates, with the following formulation for observation \(i\) and \(a \in \{\text{Car, Train, Swiss Metro}\}\):
Below we showcase how the ChoiceTask wrapper can deal with creating this reduction
[6]:
train, test = train_test_split(swissmetro, random_state=1, test_size=0.1)
swiss_metro_train = ChoiceTask(train, 'alternatives', 'chosen',
features_to_use=['Travel Time', 'Cost'],
secondary_table=dummy_secondary_table,
secondary_to_primary_link={
'merge_index': 'merge_index',
'alternative': 'alternatives'
}
)
swiss_metro_test = ChoiceTask(test, 'alternatives', 'chosen',
features_to_use=['Travel Time', 'Cost'],
secondary_table=dummy_secondary_table,
secondary_to_primary_link={
'merge_index': 'merge_index',
'alternative': 'alternatives'
}
)
[7]:
my_log_red = ClassificationReducer(LogisticRegression(solver='lbfgs'))
my_log_red.fit_task(swiss_metro_train)
log_reg_preds = my_log_red.predict_proba_task(swiss_metro_test,
['Swiss Metro','Train', 'Car'])
[8]:
nice_print_results(log_reg_preds)
Swiss Metro [0.49 0.56 0.4 ... 0.47 0.35 0.51]
Train [0.3 0.45 0.17 ... 0.36 0.29 0.26]
Car [0.33 0. 0.2 ... 0.45 0.36 0.25]
[9]:
outocme_preds = my_log_red.predict_task(swiss_metro_test)
print(outocme_preds.top_input_data[:5])
print(outocme_preds.boot_input_data[:5])
['Swiss Metro' 'Swiss Metro' 'Swiss Metro' 'Swiss Metro' 'Swiss Metro']
[array(['Car', 'Train'], dtype='<U11') array(['Train'], dtype='<U11')
array(['Car', 'Train'], dtype='<U11')
array(['Car', 'Train'], dtype='<U11')
array(['Car', 'Train'], dtype='<U11')]
Fit a Bradley-Terry model without covariates¶
Here we reduce the discrete choice problem to pairwise comparisons and fit a Bradley-Terry model.
The way these observations are broken down are such that when alternative \(a\) is chosen from the set \(A\) then an observation is expressed in a way that we say the chosen alternative was preferred to all not-chosen alternatives, \(a \succ j \forall j \in A \setminus a\). This transforms a table that looks like this:
Decision |
Alternatives |
Choice |
|---|---|---|
1 |
{a, b, c} |
a |
2 |
{b, c} |
c |
into a table that looks like this:
Decision |
Alternatives |
Choice |
|---|---|---|
1 |
{a, b} |
a |
1 |
{a, c} |
a |
2 |
{b, c} |
c |
A pariwise comparison model such as Bradley-Terry can be trained on the second table. Perhaps to make it similar to the pairwise comparison example we can also express the above table in the more familiar format:
Decision |
Alternative 1 |
Alternative 2 |
Alternative 1 is chosen |
|---|---|---|---|
1 |
a |
b |
1 |
1 |
a |
c |
1 |
2 |
b |
c |
0 |
and fit the Bradley-Terry model to learn the latent strength parameters for each alternative (e.g. \(\lambda_a\) for alternative \(a\)):
We can see that the Bradley-Terry probabilities begin to take into account the other alternatives that are offered to the decision makers, in contrast with logistic regression, which assumes that the probability of taking Swissmetro is the same whether a decision maker has a car as an alternative, a train or both. Once the Bradley-Terry model is trained, predictions have to be aggregated to discrete choice. In this section we showcase two aggregation methods one we call the Luce method the other the independent transitive method.
The Luce method (the default setting when aggregating a Bradley-Terry model)¶
For alternatives \(\{a, b, c\}\) when we fit the Bradley-Terry model we learn the function \(f(a), f(b), f(c)\) which include their strength parameters and potentially some covariates, in the econometrics literature this would be known as finding out the utility of each alternative. In the simplest case the utility equations only contain the strength parameters of the alternatives (e.g. \(\lambda_a\) for alternative \(a\)). The Luce aggregation method would predict the probability of choosing \(a\) from \(\{a, b, c\}\) as:
.
The independent transitive aggregation method¶
Let’s denote the probability of chosing \(a\) from \(\{a, b, c\}\) as \(P(a\succ \{a,b,c\})\). Suppose that we have a probabilistic pairwise comparison predictor (such as Bradley-Terry) that can provide us with the probability of preferring one over any two alternatives \(P(i \succ \{i, j\}) \forall i,j \in \{a, b, c\}\).
The independent transitive method stems from the following logic:
For \(a\) to be chosen from \(\{a, b, c\}\), \(a\) would have to be preferred to \(b\) and \(c\), that is \(a\succ\{a,b\} \cap a\succ\{a,c\}\)
Assuming that \(a\) being preferred to \(c\) is independent from \(a\) being preferred to \(b\), the probability of \(a\) being preferred to \(b\) and \(c\) is: \(P(a\succ\{a,b\} \cap a\succ\{a,c\}) = P(a\succ\{a,b\})P(a\succ\{a,c\})\)
In this three-alternative aggregation example, only 2 other things can happen in addition to \(a\) being chosen, \(b\) can be chosen or \(c\) can be chosen. Each of which can be expressed as we have expressed the probability of \(a\) being chosen in bullet 2.
By dividing the probability of \(a\) being chosen by all the three different possible outcomes (\(a\) is chosen, \(b\) is chosen or \(c\) is chosen), we arrive to the final equation of the probability that \(a\) is chosen from \(\{a, b, c\}\):
[10]:
# Fit a Bradley Terry model with no features
swiss_metro_train_BT = ChoiceTask(train.drop('merge_index', axis=1),
'alternatives',
'chosen', features_to_use=None)
swiss_metro_test_BT = ChoiceTask(test.drop('merge_index', axis=1),
'alternatives',
'chosen', features_to_use=None)
[11]:
my_BT_red = BradleyTerry(method='BFGS', alpha=1e-5)
my_BT_red.fit_task(swiss_metro_train_BT)
preds = my_BT_red.predict_proba_task(swiss_metro_test_BT,
['Swiss Metro','Train', 'Car'])
[12]:
nice_print_results(preds)
Swiss Metro [0.55 0.83 0.55 ... 0.55 0.55 0.55]
Train [0.11 0.17 0.11 ... 0.11 0.11 0.11]
Car [0.35 0. 0.35 ... 0.35 0.35 0.35]
[13]:
preds
[13]:
{'Swiss Metro': array([0.54530381, 0.8344241 , 0.54530381, ..., 0.54530381, 0.54530381,
0.54530381]),
'Train': array([0.10820537, 0.1655759 , 0.10820537, ..., 0.10820537, 0.10820537,
0.10820537]),
'Car': array([0.34649083, 0. , 0.34649083, ..., 0.34649083, 0.34649083,
0.34649083])}
[14]:
choice_preds = my_BT_red.predict_task(swiss_metro_test_BT)
print(choice_preds.top_input_data[:5])
print(choice_preds.boot_input_data[:5])
['Swiss Metro' 'Swiss Metro' 'Swiss Metro' 'Swiss Metro' 'Swiss Metro']
[array(['Car', 'Train'], dtype='<U11') array(['Train'], dtype='<U11')
array(['Car', 'Train'], dtype='<U11')
array(['Car', 'Train'], dtype='<U11')
array(['Car', 'Train'], dtype='<U11')]
Fitting a Bradley-Terry model with covariates¶
We now fit a Bradley-Terry model using the Travel Time and Cost covariates so that the equation above becomes:
[15]:
# Fit a Bradley Terry model with features
# Reducing training set because on my PC this gives a memory error
swiss_metro_train_BT_feats = ChoiceTask(
train.sample(frac=0.5), 'alternatives', 'chosen',
secondary_table=dummy_secondary_table,
secondary_to_primary_link={
'merge_index': 'merge_index',
'alternative': 'alternatives'
},
features_to_use=['Travel Time', 'Cost'])
swiss_metro_test = ChoiceTask(test, 'alternatives', 'chosen',
features_to_use=['Travel Time', 'Cost'],
secondary_table=dummy_secondary_table,
secondary_to_primary_link={
'merge_index': 'merge_index',
'alternative': 'alternatives'
}
)
my_BT_red_feats = BradleyTerry(method='BFGS', alpha=100, max_iter=100000)
my_BT_red_feats.fit_task(swiss_metro_train_BT_feats)
[16]:
swiss_metro_test = ChoiceTask(test, 'alternatives', 'chosen',
features_to_use=['Travel Time', 'Cost'],
secondary_table=dummy_secondary_table,
secondary_to_primary_link={
'merge_index': 'merge_index',
'alternative': 'alternatives'
}
)
preds = my_BT_red_feats.predict_proba_task(swiss_metro_test,
['Swiss Metro','Train', 'Car'])
[17]:
my_BT_red_feats.predict_task(swiss_metro_test).top_input_data
[17]:
array(['Swiss Metro', 'Swiss Metro', 'Swiss Metro', ..., 'Car', 'Car',
'Swiss Metro'], dtype=object)
[18]:
nice_print_results(preds)
Swiss Metro [0.56 0.75 0.67 ... 0.41 0.38 0.69]
Train [0.13 0.25 0.09 ... 0.14 0.15 0.11]
Car [0.31 0. 0.24 ... 0.45 0.47 0.21]
[19]:
ind_trans_preds = my_BT_red_feats.predict_proba_task(swiss_metro_test,
['Swiss Metro','Train', 'Car'],
aggregation_method='independent transitive')
nice_print_results(ind_trans_preds)
Swiss Metro [0.63 0.75 0.74 ... 0.44 0.4 0.77]
Train [0.06 0.25 0.04 ... 0.07 0.09 0.05]
Car [0.31 0. 0.22 ... 0.49 0.51 0.18]
[20]:
preds_outcome_ind_trans = my_BT_red_feats.predict_task(swiss_metro_test,
aggregation_method='independent transitive')
preds_outcome_Luce = my_BT_red_feats.predict_task(swiss_metro_test)
[21]:
dummy_secondary_table.groupby('alternative').mean()
[21]:
| merge_index | Travel Time | Cost | |
|---|---|---|---|
| alternative | |||
| Car | 5363.5 | 123.795209 | 78.742077 |
| Swiss Metro | 5363.5 | 87.466350 | 670.340697 |
| Train | 5363.5 | 166.626025 | 514.335477 |
[22]:
my_BT_red_feats.bt_with_feats.get_statsmodels_summary()
[22]:
| Dep. Variable: | CHOICE | No. Observations: | 8,878 |
|---|---|---|---|
| Model: | Multinomial Logit Model | Df Residuals: | 8,873 |
| Method: | MLE | Df Model: | 5 |
| Date: | Wed, 01 Mar 2023 | Pseudo R-squ.: | 0.226 |
| Time: | 16:11:21 | Pseudo R-bar-squ.: | 0.225 |
| AIC: | 9,534.703 | Log-Likelihood: | -4,762.351 |
| BIC: | 9,570.159 | LL-Null: | -6,153.761 |
| coef | std err | z | P>|z| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| Cost | 0.0003 | 3.11e-05 | 8.763 | 0.000 | 0.000 | 0.000 |
| Travel Time | -0.0116 | 0.001 | -22.819 | 0.000 | -0.013 | -0.011 |
| Car | 0.3031 | 0.046 | 6.635 | 0.000 | 0.214 | 0.393 |
| Swiss Metro | 0.1378 | 0.048 | 2.870 | 0.004 | 0.044 | 0.232 |
| Train | -0.4409 | 0.048 | -9.159 | 0.000 | -0.535 | -0.347 |
Fit Bradley-Terry model with GridSearch¶
The models we have fitted above also have hyperparameters, such as the method of gradient descent or regularisation. To optimise the hyperparameter selection, we can use GridSearchCV(). GridSearchCV() tries out a series of hyperparameter combinations and runs a k-fold cross-validation on an accuracy metric determined by the user to check which ones have performed best.
In this section we will show how aggregation works with GridSearch, it is possible to just add aggregation_method in the predict_proba_task and the ouptuts work as expected. Note in this example we have chosen a very different alpha to the models above so that there is some slight difference in the ouptuts to two decimal places, so that we can see that different parameters were learned.
[23]:
to_tune = {'alpha': [100,1000], 'method': ['BFGS']}
gs_bt = GridSearchCV(BradleyTerry(), to_tune, cv=3, scoring='neg_log_loss')
gs_bt.fit_task(swiss_metro_train_BT_feats)
gs_bt.inspect_results()
The model with the best parameters was:
BradleyTerry(alpha=100, method='BFGS')
With a score of -0.5345453019698206
All the trials results summarised in descending score
alpha method mean_test_score
0 100 BFGS -0.534545
1 1000 BFGS -0.546477
[24]:
nice_print_results(gs_bt.predict_proba_task(
swiss_metro_test,['Swiss Metro','Train', 'Car'],
aggregation_method='independent transitive'))
Swiss Metro [0.63 0.75 0.74 ... 0.44 0.4 0.77]
Train [0.06 0.25 0.04 ... 0.07 0.09 0.05]
Car [0.31 0. 0.22 ... 0.49 0.51 0.18]
[25]:
gs_bt.best_estimator_.bt_with_feats.get_statsmodels_summary()
[25]:
| Dep. Variable: | CHOICE | No. Observations: | 8,878 |
|---|---|---|---|
| Model: | Multinomial Logit Model | Df Residuals: | 8,873 |
| Method: | MLE | Df Model: | 5 |
| Date: | Wed, 01 Mar 2023 | Pseudo R-squ.: | 0.226 |
| Time: | 16:11:38 | Pseudo R-bar-squ.: | 0.225 |
| AIC: | 9,534.703 | Log-Likelihood: | -4,762.351 |
| BIC: | 9,570.159 | LL-Null: | -6,153.761 |
| coef | std err | z | P>|z| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| Cost | 0.0003 | 3.11e-05 | 8.763 | 0.000 | 0.000 | 0.000 |
| Travel Time | -0.0116 | 0.001 | -22.819 | 0.000 | -0.013 | -0.011 |
| Car | 0.3031 | 0.046 | 6.635 | 0.000 | 0.214 | 0.393 |
| Swiss Metro | 0.1378 | 0.048 | 2.870 | 0.004 | 0.044 | 0.232 |
| Train | -0.4409 | 0.048 | -9.159 | 0.000 | -0.535 | -0.347 |
Evaluation methods¶
In this section we show some of the evaluation methods available in skpref, specifically how to use log_loss and log_loss_compare_with_t_test. Please see the documentation for more details on how these work.
[26]:
print(f"The F1 score of the Luce aggregation was \
{f1_score(swiss_metro_test.subset_vec, preds_outcome_Luce): .3}")
print(f"The F1 score of the generic aggregation was \
{f1_score(swiss_metro_test.subset_vec, preds_outcome_ind_trans):.3}")
The F1 score of the Luce aggregation was 0.598
The F1 score of the generic aggregation was 0.598
[27]:
random_probs = {
'Swiss Metro': np.ones(len(test)) * (1/3),
'Train': np.ones(len(test)) * (1/3),
'Car': np.ones(len(test)) * (1/3)
}
log_reg_preds
print(f"The log loss for each alternative in the Logistic Regression reduction was \n \
{log_loss(swiss_metro_test.subset_vec, log_reg_preds)}")
print(f"The log loss for each alternative in the Luce aggregation was \n \
{log_loss(swiss_metro_test.subset_vec, preds)}")
print(f"The log loss for each alternative in the generic aggregation was \n \
{log_loss(swiss_metro_test.subset_vec, ind_trans_preds)}")
print(f"The log loss for each alternative assigning random probability was \n \
{log_loss(swiss_metro_test.subset_vec, random_probs)}")
The log loss for each alternative in the Logistic Regression reduction was
{'Swiss Metro_log_loss': 0.73, 'Train_log_loss': 0.44, 'Car_log_loss': 0.57}
The log loss for each alternative in the Luce aggregation was
{'Swiss Metro_log_loss': 0.7, 'Train_log_loss': 0.38, 'Car_log_loss': 0.52}
The log loss for each alternative in the generic aggregation was
{'Swiss Metro_log_loss': 0.71, 'Train_log_loss': 0.38, 'Car_log_loss': 0.52}
The log loss for each alternative assigning random probability was
{'Swiss Metro_log_loss': 0.8, 'Train_log_loss': 0.5, 'Car_log_loss': 0.61}
[28]:
print(f"The t-test for H0: Luce aggregation = Generic aggregation \
{log_loss_compare_with_t_test(swiss_metro_test.subset_vec, preds, ind_trans_preds)}")
print(f"The t-test for H0: Generic aggregation = random probability \
{log_loss_compare_with_t_test(swiss_metro_test.subset_vec, ind_trans_preds, random_probs)}")
print(f"The t-test for H0: Generic aggregation = Logistic Regression \
{log_loss_compare_with_t_test(swiss_metro_test.subset_vec, ind_trans_preds, log_reg_preds)}")
print(f"The t-test for H0: Generic aggregation = random probability \
{log_loss_compare_with_t_test(swiss_metro_test.subset_vec, ind_trans_preds, random_probs)}")
The t-test for H0: Luce aggregation = Generic aggregation {'Swiss Metro': 0.02, 'Train': 0.74, 'Car': 0.27}
The t-test for H0: Generic aggregation = random probability {'Swiss Metro': 0.0, 'Train': 0.0, 'Car': 0.0}
The t-test for H0: Generic aggregation = Logistic Regression {'Swiss Metro': 0.18, 'Train': 0.0, 'Car': 0.0}
The t-test for H0: Generic aggregation = random probability {'Swiss Metro': 0.0, 'Train': 0.0, 'Car': 0.0}
../..\skpref\metrics\_classification.py:254: RuntimeWarning: divide by zero encountered in log
logged = np.log(predicted1[_alternative])
../..\skpref\metrics\_classification.py:258: RuntimeWarning: invalid value encountered in multiply
np.nan_to_num(logged * binarized_outcome) +
../..\skpref\metrics\_classification.py:261: RuntimeWarning: divide by zero encountered in log
logged2 = np.log(predicted2[_alternative])
../..\skpref\metrics\_classification.py:265: RuntimeWarning: invalid value encountered in multiply
np.nan_to_num(logged2 * binarized_outcome) +