Module documentation¶

Tasks¶

`skpref.task.ChoiceTask`	Task for discrete and subset choice models
`skpref.task.PairwiseComparisonTask`	Task for choice based models

Models¶

`skpref.base.Model`	Base Class for all models
`skpref.random_utility.BradleyTerry`	Bradley Terry model
`skpref.base.ClassificationReducer`	Allows users to fit scikit-learn classifiers as predictors

Model Selection¶

skpref.model_selection.GridSearchCV

An adaption of scikit-learn's GridSearchCV into a choice model interface.

Metrics¶

`skpref.metrics.true_positives`	True positive count
`skpref.metrics.true_negatives`	True negative count
`skpref.metrics.false_positives`	False positive count
`skpref.metrics.false_negatives`	False negative count
`skpref.metrics.accuracy`	Accuracy
`skpref.metrics.recall`	Recall
`skpref.metrics.f1_score`	F1 Score
`skpref.metrics.log_loss`	Log Loss
`skpref.metrics.log_loss_compare_with_t_test`	Compares the log loss from two predictions with a paired t-test

Tasks¶

ChoiceTask¶

class skpref.task.ChoiceTask(primary_table: pandas.core.frame.DataFrame, primary_table_alternatives_names: Union[List[str], str], primary_table_target_name: Optional[str] = None, secondary_table: Optional[pandas.core.frame.DataFrame] = None, secondary_to_primary_link: Optional[dict] = None, entity_slot_type_kwargs: Optional[dict] = None, target_type_kwargs: Optional[dict] = None, features_to_use: Optional[Union[List[str], str]] = 'all')¶

Task for discrete and subset choice models

primary_table: DataFrame: The primary table is the one that contains the target variable and decision level data (covariates that may be different across each decision).
primary_table_alternatives_names: str: The column or attribute which corresponds to the alternatives in the primary table.
primary_table_target_name: str, default=None: The column name or attribute which corresponds to the ground truth
secondary_table: DataFrame, default=None: The secondary table that usually contains information about the alternatives in the primary table. For example the cleanliness perception of public transportation.
secondary_to_primary_link: dict, default:None: How to link the primary and secondary tables together. The key in the dictionary will correspond to the field in the secondary table and the value for each key will be the field in the primary table
entity_slot_type_kwargs: dict of PosetType args: arguments to tell about the PosetType of the entity slot type
target_type_kwargs: dict of PosetType args: arguments to tell about the PosetType of the entity slot type
features_to_use: list of strings, default = ‘all’: Column names of the features to use, by default the task will try to use every column as features. If the user wants to use a model that doesn’t use any features then it should be set to None

PairwiseComparisonTask¶

class skpref.task.PairwiseComparisonTask(primary_table: pandas.core.frame.DataFrame, primary_table_alternatives_names: List[str], primary_table_target_name: Optional[str] = None, target_column_correspondence: Optional[str] = None, secondary_table: Optional[pandas.core.frame.DataFrame] = None, secondary_to_primary_link: Optional[dict] = None, target_type_kwargs: Optional[dict] = None, features_to_use: Optional[Union[str, List[str]]] = 'all')¶

Task for choice based models

primary_table: str, DataFrame, scipy.io.arff: The primary table is the one that contains the target variable and covariates that vary on the decision level. For example the available methods of transportation for an individual and weather it rained or not at the time of the journey. If str it will be the directory where the primary table sits. Otherwise will also read in pandas DataFrames and scipy.io.arff
primary_table_alternatives_names: str: The column or attribute which corresponds to the alternatives in the primary table.
primary_table_target_name: str: The column name or attribute which corresponds to the ground truth
secondary_table: str, DataFrame, scipy.io.arff, default=None: The secondary table that usually contains information about the alternatives in the primary table. For example the cleanliness perception of public transportation. If str it will be the directory where the primary table sits. Otherwise will also read in pandas DataFrames and scipy.io.arff
secondary_to_primary_link: dict, default:None: How to link the primary and secondary tables together. The key in the dictionary will correspond to the field in the secondary table and the value for each key will be the field in the primary table
target_type_kwargs: dict of PosetType args: arguments to tell about the PosetType of the entity slot type
features_to_use: list of strings, default = ‘all’: Column names of the features to use, by default the task will try to use every column as features. If the user wants to use a model that doesn’t use any features then it should be set to None
target_column_correspondence: str, default=None: If the choice is a pairwise comparison and the target is not the name of an entity but a {1,0} variable that corresponds to one of the entities being chosen in one of the columns, then this should be the name of the column for which when the target variable is 1 then that column’s entity has been chosen. i.e. there is a column with home team another one with away team and target is 1 when home team wins.

Models¶

All models inherit from base.models the fit_task and predict_task methods. This helps maintain consistency within the package for how reduction and aggregation is treated.

class skpref.base.Model¶

Base Class for all models

Every model will have a fit and predict method. This is defined on the level in the model object e.g. BradleyTerry. Sometimes models will be fit on different tasks than their original design, for example a pairwise comparison model might be fit on a choice data that isn’t pairwise comparison. Pairwise comparison models will assume a different data structure to choice models. For example a pairwise comparison dataset might have the following format:

Table 1: pairwise comparison table

option 1	option 2	choice
Alt A	Alt B	Alt B

A choice dataset might have this format:

Table 2: choice table

options_presented	options_chosen
[A, B, C]	[A, C]

So pairwise comparison models and choice models would deal with different input data. But the user might want to run the choice task as if it were a pairwise comparison task which would have to look like table 1.

The task_unpackers will be methods whose job it would be to translate any dataset type into the format that is required by the model, for example all pairwise comparison models will have a task_unpacker that leaves the data as is if it looks like table 1 and changes the data into something like table 1 when it looks like table 2.

The task_packers will be used to predict the data on the right level. For example if a pairwise comparison model is used to predict a choice then the task_packer will be what aggregates the data up to something like table 2.

The fit_task function will be simply a wrapper that returns model.fit(task_unpacker(data)) and the predict_task function will be one that returns task_packer(model.preidct(data)) for every model.

This architecture is defined at the highest level and is fixed.

fit_task(task: skpref.task.PrefTask) → None¶

Fits the model using the details given in the task

Parameters: task (PrefTask) – The task that has been set up by the user

predict_task(task: skpref.task.PrefTask) → skpref.data_processing.PosetVector¶

Predicts outcomes using a task

Parameters: task (PrefTask) – The task that has been set up by the user
Returns
Return type: PosetVector of the predicted preferences

BradleyTerry¶

class skpref.random_utility.BradleyTerry(alpha=1e-06, method='Newton-CG', initial_params=None, max_iter=None, tol=1e-05)¶

Bradley Terry model

Class which fits a Bradley Terry Model based on the choix package hyperparamters can be recognised from the opt_pairwise function in choix

If alpha > 0, the function returns the maximum a-posteriori (MAP) estimate under an isotropic Gaussian prior with variance 1 / alpha.

When covariates are used then model is fit via pylogit package.

Parameters

alpha (float) – Regularization strength
method (str) – Optimization method. Either “BFGS” or “Newton-CG”
initial_params (array_like) – Parameters used to initialize the iterative procedure
max_iter (int) – Maximum number of iterations allowed
tol (float) – Tolerance for termination (method-specific)

Example

>>> import sys
>>> sys.path.insert(0, "..")
>>> from skpref.random_utility import BradleyTerry
>>> from skpref.task import PairwiseComparisonTask
>>> import pandas as pd
>>> # Using product choice data
>>> # using basketball match data
>>> NBA_file_loc = 'skpref/examples/data/NBA_matches.csv'
>>> NBA_results = pd.read_csv(NBA_file_loc)
>>> season_split = 2016
>>> train_data = NBA_results[NBA_results.season_start == season_split].copy()
>>> NBA_results_task_train = PairwiseComparisonTask(
... primary_table=train_data,
... primary_table_alternatives_names=['team1', 'team2'],
... primary_table_target_name ='team1_wins',
... target_column_correspondence='team1', features_to_use=None)
>>> mybt = BradleyTerry(method='BFGS', alpha=1e-5)
>>> mybt.fit_task(NBA_results_task_train)

predict_choice_task(task)¶

Predicts the probability that the corresponding entity will win in the task.

task: ChoiceTask type

predict_choice

predict_proba_task(task: skpref.task.PrefTask, outcome: Optional[Union[str, skpref.data_processing.PosetVector, List[str], List[skpref.data_processing.PosetVector]]] = None, column: Optional[str] = None, aggregation_method: str = 'Luce') → dict¶

Predicts the probability of specified outcomes for a specific task

Parameters

task (PrefTask) – The task for which predictions should be made
outcome (List) – The outcome for which predictions should be made, for example if the alternatives are ‘Car’, ‘Train’, ‘Bicycle’ then the user can ask for probabilities of [‘Car’, ‘Train] if they’re only interested in the probability of choosing ‘Car’ or ‘Train’
column (str) – Can also take a column name for which predictions should be made, probably more useful in pairwise comparison set ups, where team1 is in one column and team2 in another.
aggregation_method (str, default is 'Luce') –
This can be set to ‘Luce’ or ‘independent transitive’. When the method is set to ‘Luce’ then the code pretends that the parameters learned with the Bradldey-Terry method were learned with the Luce method and they used with the Luce formulation to create a prediction. For example, if the alternatives were {A, B, C} and for each of these alternatives we learn the function f(A), f(B), f(C) which include their strength parameters and potentially some covariates, the Luce prediction would say the probability of choosing A from {A, B, C} is \(\frac{e^{f(A)}}{e^{f(A)}+ e^{f(B)} + e^{f(C)}}\)

When set to ‘indeptendent transitive’ the aggregation is the following the probability of choosing A from {A, B, C} (denoted as \(P(A\succ \{A,B,C\})\) for simplicity) is \(\frac{P(A\succ\{A,B\})P(A\succ\{A,C\})}{P(A\succ\{A,B\})P(A\succ\{A,C\}) + P(B\succ\{A,B\})P(B\succ\{B,C\}) + P(C\succ\{A,C\})P(C\succ\{B,C\})}\)

Returns

Return type

A dictionary with the alternatives being the keys and for each key there’s a numpy array of floats which reflects the probability with which that alternative will be selected. When the alternative is not in the list of choices for a specific row the value will be 0. When a column is given instead of an outcome then the keys are the column name.

rank_entities(ascending=True)¶

Outputs the ranked order of entities.

Parameters: ascending (Boolean, default=True) – When True the weakest entity will be first in the list, when False the strongest entity will be first in the list.
Returns: rank – The ranks of the entities.
Return type: ndarray, shape (n_ents)

ClassificationReducer¶

class skpref.base.ClassificationReducer(model, take_feature_diff_for_pairwise_comparison: bool = False)¶

Allows users to fit scikit-learn classifiers as predictors

This is an object that allows users to model tasks using models that follow the scikit-learn structure of objects that have fit and predict methods.

Parameters

model (scikit-learn type model that user would like to fit) –

take_feature_diff_for_pairwise_comparison (bool, default = False) –

Assume we have a pairwise comparison with an alternative level table:

Assume an alternative level table of the format¶
alternative 1	alternative 2	alt1_chosen

Assume an alternative level table of the format¶
alternative	feature

There are two options for users to use these features:

Option 1: default setting, keeps the features separate for both entities and learns separate parameters on them¶
feauture alt 1	feature alt 2	alt1_chosen

Option 2: when take_feature_diff_for_pairwise_comparison is set to True then it creates one covariate which is the difference between the values of for the two alternatives in the pairwise comparison¶
feauture alt 1 - feature alt 2	alt1_chosen

Example

>>> import sys
>>> sys.path.insert(0, "..")
>>> from skpref.base import ClassificationReducer
>>> from skpref.task import PairwiseComparisonTask
>>> from sklearn.linear_model import LogisticRegression
>>> import pandas as pd
>>> # Using product choice data
>>> # using basketball match data
>>> NBA_file_loc = 'skpref/examples/data/NBA_matches.csv'
>>> NBA_results = pd.read_csv(NBA_file_loc)
>>> season_split = 2016
>>> train_data = NBA_results[NBA_results.season_start == season_split].copy()
>>> NBA_results_task_train = PairwiseComparisonTask(
... primary_table=train_data,
... primary_table_alternatives_names=['team1', 'team2'],
... primary_table_target_name ='team1_wins',
... target_column_correspondence='team1', features_to_use=['team_1_home'])
>>> my_log_red = ClassificationReducer(LogisticRegression(solver='lbfgs'))
>>> my_log_red.fit_task(NBA_results_task_train)

__init__(model, take_feature_diff_for_pairwise_comparison: bool = False)¶

Model Selection¶

GridSearchCV¶

class skpref.model_selection.GridSearchCV(estimator, param_grid, scoring=None, **kwargs)¶

An adaption of scikit-learn’s GridSearchCV into a choice model interface.

Parameters

estimator (estimator object.) – This is assumed to implement the scikit-learn estimator interface. Either estimator needs to provide a score function, or scoring must be passed.
param_grid (dict or list of dictionaries) – Dictionary with parameters names (string) as keys and lists of parameter settings to try as values, or a list of such dictionaries, in which case the grids spanned by each dictionary in the list are explored. This enables searching over any sequence of parameter settings.
kwargs (args) – all arguments that GridSearchCV takes from scikit-learn

Example

>>> import pickle
>>> import sys
>>> sys.path.insert(0, "..")
>>> from skpref.random_utility import BradleyTerry
>>> from skpref.task import ChoiceTask, PairwiseComparisonTask
>>> import pandas as pd
>>> # Using product choice data
>>> # using basketball match data
>>> NBA_file_loc = 'skpref/examples/data/'
>>> NBA_results = pd.read_csv(NBA_file_loc + 'NBA_matches.csv')
>>> NBA_team_salary_budget = pd.read_csv(NBA_file_loc
...     + 'team_salary_budgets.csv')
>>> season_split = 2016
>>> train_data = NBA_results[NBA_results.season_start == season_split].copy()
>>> NBA_results_task_train = PairwiseComparisonTask(
...     primary_table=train_data,
...     primary_table_alternatives_names=['team1', 'team2'],
...     primary_table_target_name ='team1_wins',
...     target_column_correspondence='team1',
...     features_to_use=['salary'],
...     secondary_table=NBA_team_salary_budget,
...     secondary_to_primary_link={'team': ['team1', 'team2'],
...                                'season_start': 'season_start'})
>>> to_tune = {'alpha': [1, 2, 4], 'method': ['BFGS']}
>>> gs_bt = GridSearchCV(BradleyTerry(), to_tune,  cv=3)
>>> gs_bt.fit_task(NBA_results_task_train)

fit_task(task: skpref.task.PrefTask)¶

Fits the grid search based on a task

Parameters: task (PrefTask) – The task for which the GridSearch should be run

inspect_results()¶: Returns the results of the grid-search in a user-friendly way.

predict_choice_task(task)¶

Predicts the probability that the corresponding entity will win in the task.

Parameters: task (ChoiceTask type) –
Returns
Return type: predict_choice

predict_proba_task(task: skpref.task.PrefTask, outcome: Optional[Union[str, skpref.data_processing.PosetVector, List[str], List[skpref.data_processing.PosetVector]]] = None, column: Optional[str] = None, *args, **kwargs)¶

Predicts the probability of specified outcomes for a specific task

Parameters

task (PrefTask) – The task for which predictions should be made
outcome (List) – The outcome for which predictions should be made, for example if the alternatives are ‘Car’, ‘Train’, ‘Bicycle’ then the user can ask for probabilities of [‘Car’, ‘Train] if they’re only interested in the probability of choosing ‘Car’ or ‘Train’
column (str) – Can also take a column name for which predictions should be made, probably more useful in pairwise comparison set ups, where team1 is in one column and team2 in another.

Returns

Return type

predict_task(task, *args, **kwargs)¶

Creates predictions for a task

Parameters: task (PrefTask) – The task for which predictions should be made

rank_entities(ascending=True)¶

Outputs the ranked order of entities.

Parameters: ascending (Boolean, default=True) – When True the weakest entity will be first in the list, when False the strongest entity will be first in the list.
Returns: rank – The ranks of the entities.
Return type: ndarray, shape (n_ents)

Metrics¶

skpref.metrics.true_positives(actuals: skpref.data_processing.SubsetPosetVec, predicted: skpref.data_processing.SubsetPosetVec) → int¶

True positive count

The number of alternatives predicted to be chosen and were actually chosen (tp)

Parameters

actuals (SubsetPosetVector) – the true values
predicted (SubsetPosetVector) – the predicted values

skpref.metrics.true_negatives(actuals: skpref.data_processing.SubsetPosetVec, predicted: skpref.data_processing.SubsetPosetVec) → int¶

True negative count

The number of alternatives predicted to be bot chosen and were actually not chosen (tn)

Parameters

actuals (SubsetPosetVector) – the true values
predicted (SubsetPosetVector) – the predicted values

skpref.metrics.false_positives(actuals: skpref.data_processing.SubsetPosetVec, predicted: skpref.data_processing.SubsetPosetVec) → int¶

False positive count

The number of alternatives predicted to be chosen and were actually not chosen (fp)

Parameters

actuals (SubsetPosetVector) – the true values
predicted (SubsetPosetVector) – the predicted values

skpref.metrics.false_negatives(actuals: skpref.data_processing.SubsetPosetVec, predicted: skpref.data_processing.SubsetPosetVec) → int¶

False negative count

The number of alternatives predicted to be not chosen and were actually chosen (fn)

Parameters

actuals (SubsetPosetVector) – the true values
predicted (SubsetPosetVector) – the predicted values

skpref.metrics.accuracy(actuals: skpref.data_processing.SubsetPosetVec, predicted: skpref.data_processing.SubsetPosetVec) → float¶

Accuracy

The percentage of alternatives correctly identified: (tp + tn) / (tp+fp+tn+fn)

Parameters

actuals (SubsetPosetVector) – the true values
predicted (SubsetPosetVector) – the predicted values

skpref.metrics.recall(actuals: skpref.data_processing.SubsetPosetVec, predicted: skpref.data_processing.SubsetPosetVec) → float¶

Recall

The percentage of chosen alternatives identified correctly by the predictions tp / (tp + fn)

Parameters

actuals (SubsetPosetVector) – the true values
predicted (SubsetPosetVector) – the predicted values

skpref.metrics.f1_score(actuals: skpref.data_processing.SubsetPosetVec, predicted: skpref.data_processing.SubsetPosetVec) → float¶

F1 Score

The harmonic mean between precision and recall (2 x precision x recall) / (precision + recall)

Parameters

actuals (SubsetPosetVector) – the true values
predicted (SubsetPosetVector) – the predicted values

skpref.metrics.log_loss(actuals: skpref.data_processing.SubsetPosetVec, predicted: dict, dp: int = 2, **kwargs) → dict¶

Log Loss

Calculates the mean log loss for each alternative, can provide arguments as in scikit-learn log loss

Parameters

actuals (SubsetPosetVector) – the true values
predicted (dict) – the dictionary of predicted probability values where key is the predicted outcome and values are the probabilities for each row
dp (int, default=2) – The number of decimal places to return

skpref.metrics.log_loss_compare_with_t_test(actuals: skpref.data_processing.SubsetPosetVec, predicted1: dict, predicted2: dict, dp: int = 2) → dict¶

Compares the log loss from two predictions with a paired t-test

For two different probabilistic predictions calculates whether they are significantly different with a paired t-test.

Parameters

actuals (SubsetPosetVec) – the true values
predicted1 (dict) – the dictionary of predicted probability values where key is the predicted outcome and values are the probabilities for each row for predictions from method 1
predicted2 (dict) – the dictionary of predicted probability values where key is the predicted outcome and values are the probabilities for each row for predictions from method 2
dp (int, default=2) – The number of decimal places to return

Returns

Return type

p values of the paired t-tests