# Questions tagged [hyperparameters]

63 questions

1

votes

1

answer

378

Views

### How do I optimize the hyperparameters of LightFM?

I am using the LightFM recommender library on my dataset, which gives me the results in the image below.
NUM_THREADS = 4
NUM_COMPONENTS = 30
NUM_EPOCHS = 5
ITEM_ALPHA = 1e-6
LEARNING_RATE = 0.005
LEARNING_SCHEDULE = 'adagrad'
RANDOM_SEED = 29031994
warp_model = LightFM(loss='warp',
learning_rate...

0

votes

1

answer

6

Views

### gridSearch performance measure effect

I have an assignment and it asks me to:
Improve the performance of the models from the previous stepwith
hyperparameter tuning and select a final optimal model using grid
search based on a metric (or metrics) that you choose. Choosing an
optimal model for a given task (comparing multiple regressor...

1

votes

1

answer

220

Views

### How to determine epoch hyperparameter from grid search result

I have run a grid search, with epochs as one of the hyper parameters. Now after choosing the best model, how can I determine which epoch was chosen for this particular model?
Below is the summary of the model
Model Details:
==============
H2OBinomialModel: deeplearning
Model ID: dl_grid_model_19
S...

1

votes

2

answer

185

Views

### Python / GPyOpt: Optimizing only one argument

I´m currently trying to find the minimum of some function f(arg1, arg2, arg3, ...) via Gaussian optimization using the GPyOpt module. While f(...) takes many input arguments, I only want to optimize a single one of them. How do you do that?
My current 'solution' is to put f(...) in a dummy class an...

1

votes

1

answer

179

Views

### GridSearch for doc2vec model built using gensim

I am trying to find best hyperparameters for my trained doc2vec gensim model which takes a document as an input and create its document embeddings. My train data consists of text documents but it doesn't have any labels. i.e. I just have 'X' but not 'y'.
I found some questions here related to what I...

1

votes

1

answer

81

Views

### ap_uniform_sampler() missing 1 required positional argument: 'high' in Ray Tune package for python

I am trying to use the Ray Tune package for hyperparameter tuning of a LSTM implemented using pure Tensorflow. I used the hyperband scheduler and HyperOptSearch algorithms for this and I am also using the trainable class method. When I try to run it I get the following error:
TypeError: ap_uniform_s...

1

votes

1

answer

688

Views

### Kernel parameters of Gaussian Process Regression: How to get them in Scikit-learn?

I use the squared exponential kernel or RBF in my regression operation using GaussianProcessRegressor of Scikit-learn. In addition, I use the internally available optimizer 'fmin_l_bfgs_b' (L-BFGS-B algorithm) to optimize the Kernel parameters. The kernel parameters are length scale and signal varia...

1

votes

1

answer

27

Views

### Randomized Search Get param not implemented

I am training my cnn model on some images and want to add randomized search for hyper parameter optimization but I am having trouble in using randomized search of hyper parameters. I am sharing my model and some code and Error I am having.
I have tried sklearn documentation example and other articl...

0

votes

0

answer

6

Views

### qloguniform search space setting issue in Hyperopt

I am working on using hyperopt to tune my ML model but having troubles in using the qloguniform as the search space. I am giving the example from official wiki and changed the search space.
import pickle
import time
#utf8
import pandas as pd
import numpy as np
from hyperopt import fmin, tpe, hp, S...

2

votes

1

answer

122

Views

### How to tune hyper-parameters when feeding data from flow_from_directory

I have the training data structured such flow_from_directory can be used and trains the network as well.
Now I wish to perform hyper-parameter tuning using GridSearchCV. When using GridSearchCV along with keras models the fit method expects array-like objects for input and labels.
Is there any way I...

17

votes

4

answer

6.4k

Views

### In what order should we tune hyperparameters in Neural Networks?

I have a quite simple ANN using Tensorflow and AdamOptimizer for a regression problem and I am now at the point to tune all the hyperparameters.
For now, I saw many different hyperparameters that I have to tune :
Learning rate : initial learning rate, learning rate decay
The AdamOptimizer needs 4...

2

votes

1

answer

67

Views

### Hyperparameter tuning on the whole dataset?

It may be a weird question because I don't fully understand hyperparameter-tuning yet.
Currently I'm using gridSearchCV of sklearn to tune the parameters of a randomForestClassifier like this:
gs = GridSearchCV(RandomForestClassifier(n_estimators=100, random_state=42), param_grid={'max_depth': rang...

2

votes

1

answer

275

Views

### Optimize the Kernel parameters of RBF kernel for GPR in scikit-learn using internally supported optimizers

The basic equation of square exponential or RBF kernel is as follows:
Here l is the length scale and sigma is the variance parameter. The length scale controls how two points appear to be similar as it simply magnifies the distance between x and x'. The variance parameter controls how smooth the fun...

5

votes

4

answer

3.5k

Views

### Hyper-parameter tuning using pure ranger package in R

Love the speed of the ranger package for random forest model creation, but can't see how to tune mtry or number of trees. I realize I can do this via caret's train() syntax, but I prefer the speed increase that comes from using pure ranger.
Here's my example of basic model creation using ranger (whi...

3

votes

2

answer

928

Views

### Hyperparameter in Voting classifier

So, I have a classifier which looks like
clf = VotingClassifier(estimators=[
('nn', MLPClassifier()),
('gboost', GradientBoostingClassifier()),
('lr', LogisticRegression()),
], voting='soft')
And I want to essentially tune the hyperparameters of each of the estimators.
Is there a way to tune the...

2

votes

1

answer

355

Views

### Sklearn MLP Classifier Hyperparameter Optimization (RandomizedSearchCV)

I have the following parameters set up :
parameter_space = {
'hidden_layer_sizes': [(sp_randint(100,600),sp_randint(100,600),), (sp_randint(100,600),)],
'activation': ['tanh', 'relu', 'logistic'],
'solver': ['sgd', 'adam', 'lbfgs'],
'alpha': stats.uniform(0.0001, 0.9),
'learning_rate': ['constant',...

12

votes

2

answer

12.3k

Views

### What is a good range of values for the svm.SVC() hyperparameters to be explored via GridSearchCV()?

I am running into the problem that the hyperparameters of my svm.SVC() are too wide such that the GridSearchCV() never gets completed! One idea is to use RandomizedSearchCV() instead. But again, my dataset is relative big such that 500 iterations take about 1 hour!
My question is, what is a good se...

3

votes

1

answer

680

Views

### Multidimensional hyperparameter search with vw-hypersearch in Vowpal Wabbit

vw-hypersearch is the Vowpal Wabbit wrapper intended to optimize hyperparameters in vw models: regularization rates, learning rates and decays, minibatches, bootstrap sizes etc. In the tutorial for vw-hypersearch there is a following example:
vw-hypersearch 1e-10 5e-4 vw --l1 % train.dat
Here %...

3

votes

2

answer

6.5k

Views

### How to use hyperopt for hyperparameter optimization of Keras deep learning network?

I want to build a non linear regression model using keras to predict a +ve continuous variable.
For the below model how do I select the following hyperparameters?
Number of Hidden layers and Neurons
Dropout ratio
Use BatchNormalization or not
Activation function out of linear, relu, tanh, sigmoid
Be...

13

votes

3

answer

4.1k

Views

### Hyperparameter optimization for Pytorch model

What is the best way to perform hyperparameter optimization for a Pytorch model? Implement e.g. Random Search myself? Use Skicit Learn? Or is there anything else I am not aware of?

4

votes

1

answer

83

Views

### Correlation among Hyperparameters of Classifiers

I am wondering whether there exists some correlation among the hyperparameters of two different classifiers.
For example: let us say that we run LogisticRegression on a dataset with best hyperparameters (by finding through GridSearch) and want to run another classifier like SVC (SVM classifier) on t...

2

votes

1

answer

629

Views

### Parameter selection and k-fold cross-validation

I have one dataset, and need to do cross-validation, for example, a 10-fold cross-validation, on the entire dataset. I would like to use radial basis function (RBF) kernel with parameter selection (there are two parameters for an RBF kernel: C and gamma). Usually, people select the hyperparameters o...

7

votes

2

answer

2.8k

Views

### How to pass elegantly Sklearn's GridseachCV's best parameters to another model?

I have found a set of best hyperparameters for my KNN estimator with Grid Search CV:
>>> knn_gridsearch_model.best_params_
{'algorithm': 'auto', 'metric': 'manhattan', 'n_neighbors': 3}
So far, so good. I want to train my final estimator with these new-found parameters. Is there a way to feed the ab...

2

votes

1

answer

447

Views

### Is it reasonable for l1/l2 regularization to cause all feature weights to be zero in vowpal wabbit?

I got a weird result from vw, which uses online learning scheme for logistic regression. And when I add --l1 or --l2 regularization then I got all predictions at 0.5 (that means all features are 0)
Here's my command:
vw -d training_data.txt --loss_function logistic -f model_l1 --invert_hash model_r...

3

votes

1

answer

2.5k

Views

### Is there a way to perform grid search hyper-parameter optimization on One-Class SVM

Is there a way to use GridSearchCV or any other built-in sklearn function to find the best hyper-parameters for OneClassSVM classifier?
What I currently do, is perform the search myself using train/test split like this:
Gamma and nu values are defined as:
gammas = np.logspace(-9, 3, 13)
nus = np.lin...

3

votes

0

answer

146

Views

### Compare ways to tune hyperparameters in scikit-learn

This post is about the differences between LogisticRegressionCV, GridSearchCV and cross_val_score. Consider the following setup:
import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression, LogisticRegressionCV
from sklearn.model_selection import t...

2

votes

1

answer

632

Views

### Get holdout loss in Vowpal Wabbit

I'm trying to implement grid search or more sophisticated hyperparameter search in Vowpal Wabbit. Is there a relatively simple way to get a loss function value obtained on a validation set (holdout in vw) for this purpose? VW must have computed it e.g. for every number of passes, because early stopp...

1

votes

1

answer

856

Views

### Putting together sklearn pipeline+nested cross-validation for KNN regression

I'm trying to figure out how to built a workflow for sklearn.neighbors.KNeighborsRegressor that includes:
normalize features
feature selection (best subset of 20 numeric features, no specific total)
cross-validates hyperparameter K in range 1 to 20
cross-validates model
uses RMSE as error metric
The...

3

votes

1

answer

170

Views

### Why is my mean test score at parameter tuning (cv) lower than on hold out test set (RandomForestClassifier)?

I'm doing hyperparameter tuning using RandomizedSearchCV (sklearn) with a 3 fold cross validation on my training set. After that I'm checking my score (accuracy, recall_weighted, cohen_kappa) on the test set. Surprisingly its always a bit higher than the best_score attribute of my RandomizedSearchCV...

1

votes

1

answer

1.3k

Views

### Grid Search the number of hidden layers with keras

I am trying to optimize the hyperparameters of my NN using Keras and sklearn.
I am wrapping up with KerasClassifier (it´s a classification problem).
I am trying to optimize the number of hidden layers.
I can´t figure it out how to do it with keras (actually I am wondering how to set up the functio...

1

votes

1

answer

182

Views

### Too many hyperparameter tuning metrics written out

Hyperparameter tuning job on Cloud ML Engine fails with the error message:
Too many hyperparameter tuning metrics were written by Hyperparameter Tuning Trial #...
How do I fix this?

2

votes

1

answer

87

Views

### Squared covariance function of Gaussian process using matlab?

This is my first attempt to write the covariance function. I have following values,
x = [-1.50 -1.0 -.75 -.40 -.25 0.00];
sf = 1.27;
ell = 1;
sn = 0.3;
The formula for squared exponential covariance function is
The matlab code for that I have written as :
K = sf^2*exp(-0.5*(squareform(pdist(x)).^2...

1

votes

2

answer

259

Views

### Hyperparameter tuning using MLR package

I want to tune hyperparameters for random forest using the MLR package. I have a few questions:
1) How do I decide which of the parameters I should tune? I heard something about keeping num.trees as high as computationally possible and tuning mtry? (I couldn't find anything online backing this up th...

2

votes

3

answer

675

Views

### Spark ML Linear Regression - What Hyper-parameters to Tune

I'm using the LinearRegression model in the Spark ML for predicting price. It is a single variate regression (x=time, y=price).
Assume my data is clean, what are the usual steps to take to improve this model?
So far, I tried tuning regularization parameter using cross-validation, and got rmse=15 giv...

2

votes

1

answer

1.5k

Views

### Function to determine a reasonable initial guess for scipy.optimize?

I'm using scipy.optimize.minimize to find the minimum of a 4D function that is rather sensitive to the initial guess used. If I vary it a little bit, the solution will change considerably.
There are many questions similar to this one already in SO (e.g.: 1, 2, 3), but no real answer.
In an old quest...

1

votes

1

answer

55

Views

### h2o Distributed Random Forest maximum features parameter

I am hyperparameter tuning a random forest and I would like to tune the parameter regarding the maximum features of each tree. By sklearn's documentation it is:
The number of features to consider when looking for the best split: If
int, then consider max_features features at each split.
If float, th...

2

votes

1

answer

354

Views

### LibSVM prediction accuracy

I am currently trying to run LibSVM located here: https://www.csie.ntu.edu.tw/~cjlin/libsvm
I only have access to MATLAB 2011b. When I try to run the example data file (heartscale) included with the LibSVM package with different C and gamma values I get the same accuracy results.
This happens for ot...

4

votes

2

answer

3k

Views

### How can we specify a custom lambda sequence to glmnet

I am new to the glmnet package in R, and wanted to specify a lambda function based on the suggestion in a published research paper to the glmnet.cv function. The documentation suggests that we can supply a decreasing sequence of lambdas as a parameter. However, in the documentation there are no exa...

2

votes

1

answer

141

Views

### Controlled Bayesian Optimization for Hyperparameter Tuning

What is the best way to use hyperparameter tuning using Bayesian Optimization with some heuristic selections to explore too?
In packages such as spearmint or hyperopt you can specify a range to explore but I want to also explore some heuristic values that do not necessarily belong to the range. Any...

2

votes

2

answer

2.9k

Views

### Pyspark - Get all parameters of models created with ParamGridBuilder

I'm using PySpark 2.0 for a Kaggle competition. I'd like to know the behavior of a model (RandomForest) depending on different parameters. ParamGridBuilder() allows to specify different values for a single parameters, and then perform (I guess) a Cartesian product of the entire set of parameters. As...