SciKit-learn grid search own scoring object syntax

Question

I want to search hyper-parameters in a convolutional net built with Keras. For this I am using the KerasClassifier and GridSearchCV from SciKit-learn in line with a good intro that is given here MachineLearningMastery. Typically SciKit-learn optimizes on 'accuracy', however my network runs image segmentation optimizing the Jaccard index. So I need to define my own scoring object for the grid search using make_scorer as explaned here make_scorer and here defining your scoring strategy. The code section below shows my implementation, but I am getting an error in model.compile(optimizer=optimizer, loss=eval_loss, metrics=(['eval_func']), where I do not know what to specify in metrics. Default is 'accuracy' but I would assume in my case this would be 'eval_func' (which works when not doing grid search) or 'score' but neither of these works in this case.

What is the right syntax?

def eval_func(y_true, y_pred):
    '''Evaluation function dice or jaccard, set with global var JACCARD=True'''
    if JACCARD:
        return jaccard_index(y_true, y_pred)
    else:
        return dice_coef(y_true, y_pred)


def get_unet(batch_size=32, decay=0, dropout_rate=0.5, weight_constraint=0):
    '''Create u-net model'''
    dim = 32    

    inputs = Input((3, image_cols, image_rows)) # modified to take 3 color channel input
    conv1 = Convolution2D(dim, 3, 3, activation='relu', border_mode='same', W_constraint=weight_constraint)(inputs)
    conv1 = Convolution2D(dim, 3, 3, activation='relu', border_mode='same', W_constraint=weight_constraint)(conv1)
    pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
    pool1 = Dropout(dropout_rate)(pool1) # dropout added to all layers

    ... more layers ...

    conv10 = Convolution2D(1, 1, 1, activation='sigmoid')(conv9)

    model = Model(input=inputs, output=conv10)

    optimizer = Adam(lr=LR, decay=decay)   
    model.compile(optimizer=optimizer, loss=eval_loss, metrics=(['eval_func'])

    return model

def run_grid_search():
    '''Optimize model parameters with grid search'''

    ... loading data ...

    model = KerasClassifier(build_fn=get_unet, verbose=1, nb_epoch=NUM_EPOCH, shuffle=True)
    # define grid search parameters
    batch_size = [16, 32, 48]
    decay = [0, 0.002, 0.004]
    param_grid = dict(batch_size=batch_size, decay=decay)

    # create scoring object
    score = make_scorer(eval_func, greater_is_better=True)

    grid = GridSearchCV(estimator=model, param_grid=param_grid, scoring=score, n_jobs=1, verbose=1)
    grid_result = grid.fit(X_aug, Y_aug)

Here is the last part of the error I am getting both using 'eval_func' and 'score':

File "C:\Program Files\Anaconda2\lib\site-packages\keras\metrics.py", line 216 , in get return get_from_module(identifier, globals(), 'metric') File "C:\Program Files\Anaconda2\lib\site-packages\keras\utils\generic_utils.p y", line 16, in get_from_module str(identifier)) Exception: Invalid metric: eval_func

user2653663 · Accepted Answer · 2018-09-12 08:38:21Z

1

You should unquote it when passing to compile. Keras recognizes metrics in quotation marks only if they're part of the Keras API. See Keras.io/metrics.

This is the problem:

model.compile(optimizer=optimizer, loss=eval_loss, metrics=(['eval_func'])

You should fix it to:

model.compile(optimizer=optimizer, loss=eval_loss, metrics=([eval_func])

Hope it helps!

edited Sep 12, 2018 at 8:38

user2653663

2,9481 gold badge20 silver badges23 bronze badges

answered Sep 12, 2018 at 6:19

Snir Hordan

112 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Mark Rotteveel Over a year ago

This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review

Snir Hordan Over a year ago

It does answer it because he passed the callback function in quotation marks to the compile method rather than without which should fix it.

Mark Rotteveel Over a year ago

I think you'll need to demonstrate your solution, showing specifically what needs to be done. As it stands it is not a real answer just a comment/hint.

user2653663 · Accepted Answer · 2018-09-12 08:16:23Z

0

By default scikit-learn uses the score function in the estimator, if it exists at all. So you could just override the score function in KerasClassifier.

class MyEstimator(KerasClassifier):
    def __init__(self, **kwargs):
        super(MyEstimator, self).__init__(**kwargs)

    def score(self, X, y):
        y_pred = self.predict(X)
        return eval_func(y, y_pred)

answered Sep 12, 2018 at 8:16

user2653663

2,9481 gold badge20 silver badges23 bronze badges

Collectives™ on Stack Overflow

SciKit-learn grid search own scoring object syntax

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related