A Crucial Problem: How to tune hyperparameters using TensorFlow?

As we know, we need hyperparameters, since they're a significant part of training algorithms. To some extent, they maintain the performance of Large Foundation Models. 

I assume the readers here have read my little notes on ANNs and DNNs (especially the hyperparameter tuning section). If you don't know where to start, please check these:

Basic ANNshttps://download.csdn.net/download/m0_62984100/88470062Using DNNs with confidencehttps://download.csdn.net/download/m0_62984100/88850904There're at least 3 ways to tune them. Suppose you're building a model from scratch, you may try ones in the following order:

1. Bayesian Optimization - hyperparameters - alpha=1e-4, beta=2.6 by default

    a. alpha - the expectation of observed performance measure noise

    b. beta - the tendency/balance to explore new solution during searching

2. Hyperband Search

3. Random Search

As mentioned before, we have built a ClassificationHyperModel class. Unfortunately, we cannot pass it a TensorFlow Dataset. To fix this, we should seperate the X and Y in each item of one.

Here's the original code:

class ClassificationHyperModel(kt.HyperModel):
    def build(self, hp):
        return build_model(hp)
    def fit(self, hp, model, X, Y, **kwargs):
        if hp.Boolean("normalize"):
            X = tf.keras.layers.Normalization()(X)
        return model.fit(X, Y, **kwargs)

[important!] and here's the modified one:

class ClassificationHyperModel(kt.HyperModel):
    def build(self, hp):
        return build_model(hp)
    def fit(self, hp, model, Data, **kwargs):
        if hp.Boolean("normalize"):
            Data = Data.map(lambda x,y: (tf.keras.layers.Normalization()(x),y), num_parallel_calls=tf.data.AUTOTUNE)
        return model.fit(Data, **kwargs)

After modification, you can pass a TensorFlow Dataset now!