Gradient Boosting
for Small Data
XGBoost and LightGBM fail when you have 50–500 samples. SmallGBM is built from the ground up for data-scarce regimes — robust leaf weights, stochastic splits, native C core, and scikit-learn API.
Why SmallGBM?
Robust Leaf Weights
Median for small leaves, inverse-distance weighted mean for larger ones, plus signal-adaptive shrinkage. One outlier can't destroy the estimate.
C Core
Histogram-based tree with 256 quantile bins, compiled on install. 2.7× faster training and up to 5× faster inference than XGBoost on small data.
scikit-learn Compatible
Drop-in replacement: same fit, predict,
predict_proba interface. Works with GridSearchCV, pipelines, everything.
| Model | AUC | Fit (ms) | Predict (ms) |
|---|---|---|---|
| SmallGBM | 0.9101 | 13.4 | 0.29 |
| XGBoost | 0.9036 | 36.5 | 0.46 |
| RandomForest | 0.9007 | 28.8 | 1.57 |
| LightGBM | 0.8958 | 34.3 | 0.57 |
22 datasets (15 synthetic + 7 real-world) · 5-fold cross-validation · same default hyperparameters for all models.
Getting Started
Installation
pip install smallgbm
Requires a C compiler on PATH (cc, clang, or gcc).
On macOS: xcode-select --install.
Classification — 5 lines
model = SmallGBMClassifier()
model.fit(X_train, y_train)
proba = model.predict_proba(X_test)
preds = model.predict(X_test)
Regression is not recommended. SmallGBMRegressor exists in the API
but is not validated — robust leaf regularization, tuned for classification, does not transfer.
See Limitations.
API Reference
SmallGBMClassifier
| Parameter | Default | Description |
|---|---|---|
| n_estimators | 50 | Number of boosting rounds |
| max_depth | 3 | Maximum tree depth |
| min_samples_leaf | 3 | Minimum samples per leaf |
| learning_rate | 0.1 | Shrinkage factor for each tree |
| sigma_prior | 0.5 | Regularization strength |
| colsample_bytree | 0.5 | Feature fraction per tree |
| random_state | None | Reproducibility |
| auto_scale | False | RobustScaler internally |
When NOT to use SmallGBM
Regression
SmallGBMRegressor exists in the API but is not validated and
not recommended. On 26 small regression datasets it scores normalized MAE of
0.61 vs 0.39 for XGBoost (mean rank 4.46 vs 2.31). The robust leaf
regularization, tuned for classification, does not transfer to regression.
Large datasets (n > 1000)
SmallGBM is designed for small data. On larger sets, XGBoost and LightGBM are optimized for scale and will train orders of magnitude faster. Example: at n = 200k, XGBoost reaches 0.923 AUC in 0.71s, while SmallGBM gets 0.884 in 69s. This is expected — robust leaf regularization is a small-sample technique.