Optuna generally optimizes hyperparameter tuning search spaces more efficiently than Random Search by adaptively exploring promising regions, leading to faster convergence to optimal hyperparameters. While Random Search samples uniformly, Optuna uses intelligent sampling strategies and pruning to accelerate the optimization process.
Random Search
Random Search explores the hyperparameter space by sampling values from specified distributions for a fixed number of iterations. It is straightforward to implement and highly parallelizable. Its strength lies in its ability to find good solutions in high-dimensional spaces where only a few hyperparameters significantly impact performance, as it doesn't get stuck in local optima like grid search might.
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform
# Define the model
model = RandomForestClassifier(random_state=42)
# Define the hyperparameter space
param_distributions = {
'n_estimators': randint(50, 200),
'max_depth': randint(5, 20),
'min_samples_split': uniform(0.01, 0.1),
'criterion': ['gini', 'entropy']
}
# Setup RandomizedSearchCV
random_search = RandomizedSearchCV(
estimator=model,
param_distributions=param_distributions,
n_iter=100, # Number of parameter settings that are sampled
cv=5,
scoring='accuracy',
random_state=42,
n_jobs=-1 # Use all available CPU cores
)
# random_search.fit(X_train, y_train)Optuna
Optuna is a hyperparameter optimization framework that employs state-of-the-art sampling algorithms (like Tree-structured Parzen Estimator (TPE), Gaussian Process, or CMA-ES) and pruning mechanisms. It adaptively samples the search space based on the performance of previous trials, focusing on regions likely to yield better results. Pruning allows Optuna to stop unpromising trials early, saving computational resources.
import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
def objective(trial):
n_estimators = trial.suggest_int('n_estimators', 50, 200)
max_depth = trial.suggest_int('max_depth', 5, 20)
min_samples_split = trial.suggest_float('min_samples_split', 0.01, 0.1, log=False)
criterion = trial.suggest_categorical('criterion', ['gini', 'entropy'])
classifier = RandomForestClassifier(
n_estimators=n_estimators,
max_depth=max_depth,
min_samples_split=min_samples_split,
criterion=criterion,
random_state=42
)
score = cross_val_score(classifier, X, y, n_jobs=-1, cv=3).mean()
return score
# Create a study object and optimize the objective function
study = optuna.create_study(direction='maximize', sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective, n_trials=100)
# To enable pruning (e.g., for neural networks or iterative models)
# study = optuna.create_study(direction='maximize',
# sampler=optuna.samplers.TPESampler(seed=42),
# pruner=optuna.pruners.MedianPruner(n_startup_trials=5, n_warmup_steps=30, interval_steps=10))
# study.optimize(objective_with_intermediate_reports, n_trials=100)| Feature | Random Search | Optuna |
| :----------------- | :------------------------------------------- | :-------------------------------------------------------- |
| Strategy | Uniform sampling from distributions | Adaptive, intelligent sampling (e.g., TPE, GP, CMA-ES) |
| Efficiency | Less efficient for complex interactions | More efficient, faster convergence to optimal values |
| Setup Complexity | Low, direct parameter distributions | Moderate, requires defining an objective function |
| Pruning | No built-in mechanism | Yes, stops unpromising trials early (e.g., MedianPruner) |
| Parallelization | Embarrassingly parallel, simple to distribute | Easy, supports distributed optimization with RDB backend |
| Best Use Case | Quick exploration, high-dimensional spaces | Complex models, limited compute budget, iterative training |