Predicting Smartphone Addiction

What the competition asks

Kaggle's Playground Series runs a fresh tabular problem every month. Season 6, Episode 8 gives you twelve columns describing someone's phone habits — hours of screen time, hours on social media, notifications per day, age, a couple of survey answers — and asks for one binary label, addicted_label.

The metric shapes everything downstream. ROC AUC only reads the ordering of predictions, not their absolute values. A model that says 0.9 for every positive and 0.1 for every negative scores exactly the same as one that says 0.51 and 0.49. This means calibration is irrelevant, thresholds are irrelevant, and the class imbalance mostly takes care of itself — there is no need to reweight classes or tune a cutoff. It also determines how models should be combined, which turns out to be where the interesting mistake lives.

Getting the data out

The first obstacle had nothing to do with machine learning. The standard kaggle CLI authenticates from a ~/.kaggle/kaggle.json holding a username and an API key. What was actually on this machine was a newer KGAT_… access token, which the CLI does not read — it fails with Could not find kaggle.json regardless.

Rather than go hunting for a second credential, I tested the token against the REST API directly under four different auth schemes to see whether any of them worked:

Bearer works. So the download step is nine lines of curl against /api/v1/competitions/data/download/<comp>/<file> rather than the CLI, and the same call also returns the competition metadata that tells you the metric is ROC AUC and the deadline is 31 August.

One gate is not solvable from a terminal: until you accept the competition rules in a browser, every download returns

{"code":403,"message":"You must accept this competition's
 rules before you'll be able to download files."}

The API exposes no endpoint for accepting rules, which is deliberate — it is a legal agreement. Worth knowing before you spend twenty minutes assuming your token is broken.

What the data looks like

Nine numeric columns, three categorical. Train and test are drawn from the same distribution to an almost boring degree — every numeric mean matches between the two within 0.11%, so there is no drift to correct and no need for adversarial validation.

Measuring each feature on its own, by the same metric the competition uses, sorts them immediately:

Three screen-time columns score 0.86–0.89 alone. Two more are weakly useful. The bottom four are, individually, noise: age at 0.502 and notifications_per_day at 0.492 are coin flips.

The categorical columns are worse than weak — they are flat:

Against a base rate of 70.9%, nothing here moves. stress_level being identical across High and Low is a good reminder that this data is generated, not observed — in a real survey, self-reported stress would almost certainly track phone use. I kept the columns anyway, since a GBDT will simply decline to split on them, but I stopped expecting anything from them.

The missingness problem

This is the part of the dataset that actually requires a decision. Every column has holes, and they are large:

Individually these look survivable. Together they are not: because the holes are spread across columns, only 44.3% of rows have every field present. Any approach that needs complete rows throws away more than half the data.

There are three honest options, and the choice matters more than any hyperparameter:

1. Drop incomplete rows. Discards 55.7% of 691k rows. Not seriously on the table, but worth naming to see how bad the alternative has to be before it wins.
2. Impute. Fill the holes with medians or a model, then train on a complete matrix. This invents data, and if a value is missing because of what it would have been, imputation actively erases the signal.
3. Leave the holes as holes. LightGBM, XGBoost and CatBoost all handle NaN natively: at each split, missing rows are sent down whichever branch reduces the loss more. The model learns what a hole is worth.

I went with the third, plus an explicit hedge: a binary indicator for every column that has any missing values, and a per-row count of how many fields are absent. If missingness is random these cost almost nothing, since a tree just never splits on them. If it isn't random, they hand the model the pattern directly instead of hoping it reconstructs it. That is 37 features from an original 12.

Is there a rule to steal?

Because Playground data is generated, it is always worth asking whether the label is a deterministic function of the features. If it is, the competition stops being a modelling problem and becomes a puzzle — find the rule, score 1.00, go home. This is common enough that checking first is just good hygiene.

The test: fit a plain logistic regression on the 306,175 complete rows, then look at how the true positive rate behaves across the model's own score. A hard threshold produces a step function — rates pinned at 0 and 1 with a cliff between. A probabilistic label produces a ramp.

It's a ramp. The label is genuinely probabilistic, there is a noise ceiling above every model, and no amount of cleverness reaches 1.00. That is a useful negative result: it says the remaining work is ordinary modelling, and time spent hunting for a magic threshold is time wasted.

The same probe returned something more useful, though. That logistic regression — nine raw features, no interactions, no tuning — scores 0.928 in-sample. The relationship is close to smooth and monotone. Hold that thought; it comes back in §9.

Finding the original

Since the synthetic data was generated from something, the something is usually still on Kaggle. Searching the datasets API for "smartphone addiction" returned several files at a suspiciously identical 0.18 MB. One of them matched exactly:

It carries one extra column the competition doesn't ship, addiction_level, and cross-tabulating it against the label is unambiguous:

So addicted_label is exactly addiction_level != "Mild" — a binary cut through a latent three-level severity. That explains the smooth ramp in §6: the underlying quantity is ordinal and the label throws away its resolution, so cases near the Mild/Moderate boundary are inherently ambiguous.

Knowing this is more satisfying than useful. The severity column has no counterpart in the test set, so it cannot become a feature. What remains is 7,500 extra genuine rows, available behind a --extra flag that appends them to every training fold but never scores them, so out-of-fold numbers stay comparable to runs without them. Against 691k rows they are about 1% more data and I expect them to do almost nothing — but the plumbing is there to measure it rather than guess.

The suppression effect

The most interesting structure in this dataset is not visible in any single correlation. Compare how two features look alone versus how they look once screen time is accounted for:

Gaming and work/study hours correlate positively with addiction on their own, and negatively once the model can see total screen time. Both signs are real and they are not in conflict. These columns are components of a single time budget: people with more gaming hours have more screen time overall, which is why the raw correlation is positive. But hold total screen time fixed, and an hour spent gaming or working is an hour not spent on whatever actually drives the label. Conditioned on the total, they become protective.

This is why the engineered features are mostly differences and ratios within that budget:

parts_sum     = social_media + gaming + work_study
unaccounted   = daily_screen_time - parts_sum
parts_ratio   = parts_sum / daily_screen_time
leisure_ratio = (social_media + gaming) / daily_screen_time
weekend_gap   = weekend_screen_time - daily_screen_time
mins_per_open = daily_screen_time * 60 / app_opens_per_day
notif_per_open = notifications_per_day / app_opens_per_day

A gradient-boosted tree splits on one feature at a time, at an axis-aligned threshold. It can represent "screen time > 8 AND gaming < 1", but it cannot represent "gaming ÷ screen time < 0.15" without an enormous staircase of splits approximating the boundary. Handing it the ratio directly converts an expensive approximation into one cheap split.

The trained model does reach for them: notif_per_open, parts_ratio and mins_per_open all land in the top fifteen features by split count, and notifications_per_day — a coin flip on its own — becomes the single most-split feature in the model once it can be combined with app opens.

Why more than one model

LightGBM is the default choice here and needs little defending: gradient boosting on trees is the strongest general method for tabular data of this size, it consumes NaN natively, and it handles categorical columns without one-hot expansion. It trains a fold of 553k rows in about 70 seconds on a laptop.

XGBoost and CatBoost are in the pipeline for the ordinary reason — they are the same family with different split-finding and regularisation, so their mistakes are correlated but not identical, and averaging trims variance.

The fourth model is the one worth explaining. It is a logistic regression on spline-expanded features, and it exists because of the probe result in §6: a raw linear model already reached 0.928, so the true relationship is close to smooth and monotone. That is precisely the shape a decision tree is bad at. Trees approximate a smooth slope with a staircase, and every step is a small error. A spline basis plus a linear model fits the smooth part exactly.

The point of adding it is not that it is strong. It is that it is wrong differently. A blend of models that make the same mistakes gains nothing; a blend of models that make different mistakes gains a lot.

That argument is reasonable and, as §12 shows, it was wrong — the finished blend gives this model zero weight. It is kept in the pipeline anyway, because the cost is ten seconds a fold and the §6 probe it came from is still the fastest way to learn what shape a dataset has.

The blend that made things worse

The first version of the blending code did the obvious thing: convert each model's predictions to ranks, average them with equal weight, done. Rank averaging is a sound instinct for AUC — the metric only cares about ordering, so averaging ranks avoids letting a badly-calibrated model dominate just because its probabilities are more extreme.

Run on a 5% subsample, it produced this:

The blend was 0.0053 worse than simply using LightGBM. In hindsight this is arithmetic, not misfortune. Equal weights are only the right answer when the members are about equally good. Give a 0.9482 model and a 0.9286 model half the vote each and the result lands between them; the weaker model does not contribute enough independent information to pay for the 50% of the ordering it just took over.

The fix is to let the data set the weights. I replaced the average with Caruana greedy ensemble selection: start from nothing, and repeatedly add whichever model most improves out-of-fold AUC, with replacement. A model that helps gets picked repeatedly and ends up with a large weight; a model that helps slightly gets picked once; a model that helps not at all never gets picked.

for _ in range(rounds):
    pick the model m maximising AUC of (running_mean * n + rank[m]) / (n+1)
    if adding it doesn't improve OOF AUC: stop
    fold it into the running mean

The property that matters is the stopping rule. If no addition improves the score, the loop breaks immediately and returns the single best model. The method is allowed to decline to blend. An equal-weight average has no way to express "actually, just use LightGBM", which is exactly the answer it needed to give here.

A false alarm

Worth recording because the reasoning failed in a way that looked like success.

The logistic model emitted a wall of warnings on every fit:

RuntimeWarning: divide by zero encountered in matmul
RuntimeWarning: overflow encountered in matmul
RuntimeWarning: invalid value encountered in matmul

My first diagnosis: the pipeline spline-expands every numeric column, including the binary missingness indicators added in §5. Splines over a column that only takes values 0 and 1 produce basis functions that are constant, which have zero variance, which StandardScaler then divides by. This is a genuinely plausible story, it fits the symptom, and the fix — route binary columns around the spline step — is a real improvement.

It also did not change anything. The warnings continued unchanged.

So I stopped theorising and instrumented the pipeline, checking for non-finite values at each stage:

Clean at every stage. So the input to the matrix multiply was finite and the output was finite, yet numpy insisted something had divided by zero. The decisive test is three lines — multiply a matrix of random normals by a random vector, assert both are finite, and see whether the warning fires:

numpy 2.0.2, BLAS: accelerate
result finite: True
warnings raised on provably-finite inputs:
  ['divide by zero encountered in matmul',
   'overflow encountered in matmul',
   'invalid value encountered in matmul']

The warnings are spurious, and they are a known defect rather than anything to do with this project. numpy builds on macOS dispatch matrix multiplication to Apple's Accelerate framework, and Accelerate raises floating-point status flags on operations whose results are perfectly finite; numpy faithfully reports the flags it finds. It is tracked upstream as numpy issues #28687 and #28790, where it reproduces on identity matrices from 15×15 upward while 14×14 stays silent — a size threshold that points at a vectorised code path rather than at any property of the data. Apple is reported to be aware of it. Nothing was ever wrong with the data or the model.

Silencing them took three attempts, and each one failed in a way worth knowing about. The obvious move is a warnings filter matched to the message:

warnings.filterwarnings("ignore", message=".*encountered in matmul",
                        category=RuntimeWarning)

That works in isolation and does nothing here, because scikit-learn calls warnings.simplefilter("always") inside its own fitting code, which overrides the filter. The second attempt swaps to numpy's own floating-point error control, np.errstate, which is immune to whatever the warnings module is doing — and that also didn't work, for a reason that took a proper experiment to find:

The culprit was n_jobs=-1 on the logistic regression, which I had added reflexively. It pushes the fit into joblib workers, and numpy's floating-point state is context-local — it does not follow the work across that boundary, so the errstate wrapping the call applies to nothing that matters. The parameter was pointless anyway: n_jobs only affects multiclass one-vs-rest fitting, and this problem is binary. Removing it and keeping np.errstate, scoped to just this fit, gives a clean run with an identical AUC of 0.92803.

Where it stands

The baseline, cross-validated on the full 691k rows:

The fold spread is 0.0018, which is tight enough that improvements smaller than about 0.0005 should not be believed from a single 5-fold run.

Running all four models and letting the greedy selector weight them:

The three tree models land within 0.0009 of each other, with XGBoost narrowly ahead — and CatBoost taking four and a half times as long as either competitor to finish last of the three.

The result worth pointing at is the last weight in that column. The logistic model was given zero weight. The selector evaluated it, found it made every candidate blend worse, and never picked it — which is precisely the behaviour the equal-weight average could not express, and precisely the case that cost 0.0053 back in §10. The argument from §9 that it would bring useful diversity turned out to be wrong: at 0.932 it is simply too far behind, and whatever the smooth-function story gave it, the trees had already captured.

What the leaderboard said

Both files were then submitted — the blend, and XGBoost alone as a control on whether the blend's margin was real.

The local estimates were conservative, not optimistic. Both score about 0.0014 higher on the leaderboard than out of fold, and the two gaps agree to within 0.0002. That consistency is the good outcome: cross-validation here cannot be read as an absolute performance estimate, but it ranks candidates faithfully, which is the only thing it was used for.

The gap is not a validation flaw. Each test prediction averages the five fold-models, while each out-of-fold prediction comes from exactly one — so the submitted file is a five-model bag and the OOF score is not. The leaderboard is measuring a slightly stronger object than the cross-validation is.

And the blend's advantage evaporated. It beat XGBoost alone by +0.00022 out of fold and by +0.00001 on held-out data. The caveat in the box above turned out to be the whole story: the weights were chosen to maximise OOF AUC, so almost all of that margin was selection optimism. On the leaderboard the blend and its dominant member are indistinguishable, and the two extra tree models cost 6.5 minutes per fold to achieve that.

What I'd do next

Roughly in order of expected return per hour:

1. Move to 10-fold. The clearest remaining gain. A control run showed that cutting the training set by 12% costs 0.00035 AUC, so the model is still data-limited at 691k rows and 90% training folds should beat 80% ones by roughly +0.0002 — more than the entire three-model blend delivered.
2. Drop the three categorical columns. Permutation importance measured them at 0.00000–0.00003. Free speedup, no measurable cost.
3. Tune hyperparameters. The current values are hand-picked defaults, never searched. On a dataset this size that is usually worth a few ten-thousandths, occasionally more.
4. Multi-seed bagging. Repeat the CV with several seeds and average. Reliable, dull, and roughly the size of the fold noise.
5. Test --extra. 7,500 real rows against 691k synthetic ones. Probably nothing, but it is one command.
6. A neural net. An MLP would bring genuinely different inductive bias to the blend — and now that the blend can weight members properly, a mediocre one can be added without risk of dragging the score down.

Limitations

• Only the public leaderboard is known. That is scored on a subset of the test set; the private split that decides the competition is still unseen. The validation itself checks out — a separate leakage sweep found `id` worthless as a predictor (AUC 0.5007), no duplicate rows, and only two exact train/test matches.
• Single seed, single fold scheme. One 5-fold split at seed 42. This limits unpaired comparisons, where the 0.0018 fold spread sets the noise floor. Comparisons on identical folds resolve far smaller effects: the feature-engineering ablation measured +0.00057 with a paired standard deviation of 0.00008.
• The equal-weight blend failure was measured on a 5% subsample, not the full dataset. The direction of the effect is what mattered, and greedy selection is not worse than equal weights in any case, but the exact magnitude at full scale is unmeasured.
• The blend weights are fitted on the same out-of-fold predictions the blend is scored against. That makes 0.96514 optimistic by an unknown but probably small amount. Properly, weight selection belongs inside a nested split; at four members and a +0.0002 margin it was not worth the extra complexity, but the number should be read with that in mind.
• The blend's value is now measured at approximately zero. It is kept because greedy selection cannot do worse than its best member by construction, but on this problem the two extra tree models buy +0.00001.
• Hyperparameters are guesses. Reasonable ones, but no search was run.
• The environment is Python 3.9, inherited from the system interpreter. Everything installs and runs, but it is two years past its useful life and a 3.11+ rebuild would be faster.
• Everything here describes synthetic data. The suppression effect in §8 is a property of Kaggle's generative model, which was itself fit to a 7,500-row survey of unknown provenance. None of it is a finding about actual smartphone use, and it should not be read as one.

Figures computed from train.csv and test.csv as distributed for playground-series-s6e8, plus Smartphone_Usage_And_Addiction_Analysis_7500_Rows.csv (jayjoshi37/smartphone-usage-and-addiction-prediction) for §7. Code in src/: eda.py for §4–5, probe.py for §6, train.py for §8–11.