Logistic regression, compared with a simple baseline
Author
Arsy Nas (group coursework, UBC Sauder COMM 414)
Published
2022
NoteSummary
Problem. COMM 414 Assignment 4, a group assignment from 2022. Predict a win or a loss for each Atlanta Braves home game from values that are known before the game.
Method. Logistic regression for four groups of variables, then a final model. A random 70/30 split with a fixed seed.
Results.
The final model is correct for 56.1% of the test games.
The team won 51.1% of the test games. A model that predicts a win for each game gets that score, so the model adds about five points.
Team performance is the best group of variables. Payroll alone, the schedule and the weather add almost nothing.
Limit. This is coursework. The model is a small step above the baseline and is not a tool to predict games.
1 Setup
Libraries, data and new variables
import numpy as npimport pandas as pdimport statsmodels.formula.api as smfimport plotly.graph_objects as goimport plotly.express as pxfrom _plot import showfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import confusion_matrix, accuracy_scorebaseball = pd.read_csv('../baseball.csv')weather = pd.read_csv('../weather.csv')df = baseball.join(weather.set_index('date'), on='event_date', how='left')df['weather'] = df['weather'].astype('category')df['is_night'] = df['is_night'].astype('category')df['weekday'] = df['weekday'].astype('category')df['opponent_rank'] = df['opponent_rank'].astype('int64')df['braves_rank'] = df['braves_rank'].astype('int64')df['is_weekend'] = df['weekday'].apply(lambda d: 1if d in ['Friday', 'Saturday'] else0)df['payroll_ratio'] = df['braves_payroll'] / (df['braves_payroll'] + df['opponent_payroll'])print(f"Games: {len(df):,} | share of wins: {df['win'].mean():.1%}")
Games: 461 | share of wins: 54.9%
2 Split
Show code
train, test = train_test_split(df, test_size=0.3, shuffle=True, random_state=0)print(f"Training set: {len(train)} games | test set: {len(test)} games")print(f"Share of wins: training {train['win'].mean():.1%} | test {test['win'].mean():.1%}")
Training set: 322 games | test set: 139 games
Share of wins: training 56.5% | test 51.1%
ms = {'1. Payroll': model_1, '2. Team performance': model_2, '3. Schedule': model_3, '4. Weather': model_4}comparison = pd.DataFrame({'Model': list(ms),'Pseudo R²': [m.prsquared for m in ms.values()],'Log-likelihood': [m.llf for m in ms.values()],'AIC': [m.aic for m in ms.values()],})comparison.style.format({'Pseudo R²': '{:.4f}', 'Log-likelihood': '{:,.2f}', 'AIC': '{:,.2f}'}).hide(axis='index')
Table 1: The four models on the training games
Model
Pseudo R²
Log-likelihood
AIC
1. Payroll
0.0042
-219.53
445.05
2. Team performance
0.0195
-216.14
442.29
3. Schedule
0.0070
-218.90
443.80
4. Weather
0.0042
-219.51
447.02
Team performance has the highest pseudo R² and the lowest AIC. All the values are small: no group of variables explains much of the result of one game.
4 The final model
The final model uses the team-performance variables and adds a payroll ratio.
where is the sum of each coefficient times its variable. The win percentage of the Braves has the largest coefficient. Read it as a direction only: this is a coursework model, and the coefficient is not a causal estimate.
Table 2: The model against two simple rules on the test games
Rule
Accuracy
Games correct
Wins predicted
Final model, threshold 0.5
56.1%
78
102
Always predict a win (baseline)
51.1%
71
139
First draft: final model, threshold 0.9
48.9%
68
0
WarningRead the accuracy against the baseline
The team won 51.1% of the test games. A model that predicts a win for each game is correct that often. The correct comparison is 56.1% against 51.1%, not against a coin flip.
Our first draft used a threshold of 0.9. No game had a win probability that high, so the draft predicted a loss for each game.