6-1 Three Ways of Modeling

    For the models with sequenced structure, Sequential method should be given the highest priority.

    For the models with nonsequenced structures such as multiple input/output, shared weights, or residual connections, modeling with functional API is recommended.

    Modeling through child class of Model should be AVOIDED unless with special requirements. This method is flexible, but also fallible.

    1. tf.keras.backend.clear_session()
    3. model = models.Sequential()
    5. model.add(layers.Embedding(MAX_WORDS,7,input_length=MAX_LEN))
    6. model.add(layers.Conv1D(filters = 64,kernel_size = 5,activation = "relu"))
    7. model.add(layers.MaxPool1D(2))
    8. model.add(layers.Conv1D(filters = 32,kernel_size = 3,activation = "relu"))
    9. model.add(layers.MaxPool1D(2))
    10. model.add(layers.Flatten())
    11. model.add(layers.Dense(1,activation = "sigmoid"))
    13. model.compile(optimizer='Nadam',
    14.             loss='binary_crossentropy',
    15.             metrics=['accuracy',"AUC"])
    17. model.summary()

    1. import datetime
    2. baselogger = callbacks.BaseLogger(stateful_metrics=["AUC"])
    3. logdir = "../data/keras_model/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
    4. tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
    5. history = model.fit(ds_train,validation_data = ds_test,
    6.         epochs = 6,callbacks=[baselogger,tensorboard_callback])
    1. %matplotlib inline
    2. %config InlineBackend.figure_format = 'svg'
    4. import matplotlib.pyplot as plt
    6. def plot_metric(history, metric):
    7.     train_metrics = history.history[metric]
    8.     val_metrics = history.history['val_'+metric]
    9.     epochs = range(1, len(train_metrics) + 1)
    10.     plt.plot(epochs, train_metrics, 'bo--')
    11.     plt.plot(epochs, val_metrics, 'ro-')
    12.     plt.title('Training and validation '+ metric)
    13.     plt.xlabel("Epochs")
    14.     plt.ylabel(metric)
    15.     plt.legend(["train_"+metric, 'val_'+metric])
    16.     plt.show()
    1. plot_metric(history,"AUC")

    6-1 Three Ways of Modeling - 图2

    1. Model: "model"
    2. __________________________________________________________________________________________________
    3. Layer (type)                    Output Shape         Param #     Connected to                     
    4. ==================================================================================================
    5. input_1 (InputLayer)            [(None, 200)]        0                                            
    6. __________________________________________________________________________________________________
    7. embedding (Embedding)           (None, 200, 7)       70000       input_1[0][0]                    
    8. __________________________________________________________________________________________________
    9. separable_conv1d (SeparableConv (None, 198, 64)      533         embedding[0][0]                  
    10. __________________________________________________________________________________________________
    11. separable_conv1d_2 (SeparableCo (None, 196, 64)      547         embedding[0][0]                  
    12. __________________________________________________________________________________________________
    13. separable_conv1d_4 (SeparableCo (None, 194, 64)      561         embedding[0][0]                  
    15. __________________________________________________________________________________________________
    16. max_pooling1d_1 (MaxPooling1D)  (None, 39, 64)       0           separable_conv1d_2[0][0]         
    17. __________________________________________________________________________________________________
    18. max_pooling1d_2 (MaxPooling1D)  (None, 27, 64)       0           separable_conv1d_4[0][0]         
    19. __________________________________________________________________________________________________
    20. separable_conv1d_1 (SeparableCo (None, 64, 32)       2272        max_pooling1d[0][0]              
    21. __________________________________________________________________________________________________
    22. separable_conv1d_3 (SeparableCo (None, 35, 32)       2400        max_pooling1d_1[0][0]            
    23. __________________________________________________________________________________________________
    24. separable_conv1d_5 (SeparableCo (None, 21, 32)       2528        max_pooling1d_2[0][0]            
    25. __________________________________________________________________________________________________
    26. global_max_pooling1d (GlobalMax (None, 32)           0           separable_conv1d_1[0][0]         
    27. __________________________________________________________________________________________________
    28. global_max_pooling1d_1 (GlobalM (None, 32)           0           separable_conv1d_3[0][0]         
    29. __________________________________________________________________________________________________
    30. global_max_pooling1d_2 (GlobalM (None, 32)           0           separable_conv1d_5[0][0]         
    31. __________________________________________________________________________________________________
    32. concatenate (Concatenate)       (None, 96)           0           global_max_pooling1d[0][0]       
    33.                                                                  global_max_pooling1d_1[0][0]     
    34.                                                                  global_max_pooling1d_2[0][0]     
    35. __________________________________________________________________________________________________
    36. dense (Dense)                   (None, 1)            97          concatenate[0][0]                
    37. ==================================================================================================
    38. Total params: 78,938
    39. Trainable params: 78,938
    40. Non-trainable params: 0
    41. __________________________________________________________________________________________________

    1. import datetime
    2. logdir = "../data/keras_model/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
    3. tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
    4. history = model.fit(ds_train,validation_data = ds_test,epochs = 6,callbacks=[tensorboard_callback])
    1. Epoch 1/6
    2. 1000/1000 [==============================] - 32s 32ms/step - loss: 0.5527 - accuracy: 0.6758 - AUC: 0.7731 - val_loss: 0.3646 - val_accuracy: 0.8426 - val_AUC: 0.9192
    3. Epoch 2/6
    4. 1000/1000 [==============================] - 24s 24ms/step - loss: 0.3024 - accuracy: 0.8737 - AUC: 0.9444 - val_loss: 0.3281 - val_accuracy: 0.8644 - val_AUC: 0.9350
    5. Epoch 3/6
    6. 1000/1000 [==============================] - 24s 24ms/step - loss: 0.2158 - accuracy: 0.9159 - AUC: 0.9715 - val_loss: 0.3461 - val_accuracy: 0.8666 - val_AUC: 0.9363
    7. Epoch 4/6
    8. 1000/1000 [==============================] - 24s 24ms/step - loss: 0.1492 - accuracy: 0.9464 - AUC: 0.9859 - val_loss: 0.4017 - val_accuracy: 0.8568 - val_AUC: 0.9311
    9. Epoch 5/6
    10. 1000/1000 [==============================] - 24s 24ms/step - loss: 0.0944 - accuracy: 0.9696 - AUC: 0.9939 - val_loss: 0.4998 - val_accuracy: 0.8550 - val_AUC: 0.9233
    11. Epoch 6/6
    12. 1000/1000 [==============================] - 26s 26ms/step - loss: 0.0526 - accuracy: 0.9865 - AUC: 0.9977 - val_loss: 0.6463 - val_accuracy: 0.8462 - val_AUC: 0.9138
    1. plot_metric(history,"AUC")
    1. # Define a customized residual module as Layer
    3. class ResBlock(layers.Layer):
    4.     def __init__(self, kernel_size, **kwargs):
    5.         super(ResBlock, self).__init__(**kwargs)
    6.         self.kernel_size = kernel_size
    8.     def build(self,input_shape):
    9.         self.conv1 = layers.Conv1D(filters=64,kernel_size=self.kernel_size,
    10.                                    activation = "relu",padding="same")
    11.         self.conv2 = layers.Conv1D(filters=32,kernel_size=self.kernel_size,
    12.                                    activation = "relu",padding="same")
    13.         self.conv3 = layers.Conv1D(filters=input_shape[-1],
    14.                                    kernel_size=self.kernel_size,activation = "relu",padding="same")
    15.         self.maxpool = layers.MaxPool1D(2)
    16.         super(ResBlock,self).build(input_shape) # Identical to self.built = True
    18.     def call(self, inputs):
    19.         x = self.conv1(inputs)
    20.         x = self.conv3(x)
    21.         x = layers.Add()([inputs,x])
    22.         x = self.maxpool(x)
    23.         return x
    25.     # Need to define get_config method in order to sequentialize the model constructed from the customized Layer by Functional API.
    26.     def get_config(self):  
    27.         config = super(ResBlock, self).get_config()
    28.         config.update({'kernel_size': self.kernel_size})
    29.         return config
    1. # Test ResBlock
    2. resblock = ResBlock(kernel_size = 3)
    3. resblock.build(input_shape = (None,200,7))
    4. resblock.compute_output_shape(input_shape=(None,200,7))
    1. TensorShape([None, 100, 7])
    1. tf.keras.backend.clear_session()
    3. model = ImdbModel()
    4. model.build(input_shape =(None,200))
    5. model.summary()
    7. model.compile(optimizer='Nadam',
    8.             loss='binary_crossentropy',
    9.             metrics=['accuracy',"AUC"])
    1. Model: "imdb_model"
    2. _________________________________________________________________
    3. Layer (type)                 Output Shape              Param #   
    4. =================================================================
    5. embedding (Embedding)        multiple                  70000     
    6. _________________________________________________________________
    7. res_block (ResBlock)         multiple                  19143     
    8. _________________________________________________________________
    9. res_block_1 (ResBlock)       multiple                  13703     
    10. _________________________________________________________________
    11. dense (Dense)                multiple                  351       
    12. =================================================================
    13. Total params: 103,197
    14. Trainable params: 103,197
    15. Non-trainable params: 0
    16. _________________________________________________________________

    6-1 Three Ways of Modeling - 图5

    1. import datetime
    3. logdir = "../tflogs/keras_model/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
    4. tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
    5. history = model.fit(ds_train,validation_data = ds_test,
    6.                     epochs = 6,callbacks=[tensorboard_callback])
    1. Epoch 1/6
    2. 1000/1000 [==============================] - 47s 47ms/step - loss: 0.5629 - accuracy: 0.6618 - AUC: 0.7548 - val_loss: 0.3422 - val_accuracy: 0.8510 - val_AUC: 0.9286
    3. Epoch 2/6
    4. 1000/1000 [==============================] - 43s 43ms/step - loss: 0.2648 - accuracy: 0.8903 - AUC: 0.9576 - val_loss: 0.3276 - val_accuracy: 0.8650 - val_AUC: 0.9410
    5. Epoch 3/6
    6. 1000/1000 [==============================] - 42s 42ms/step - loss: 0.1573 - accuracy: 0.9439 - AUC: 0.9846 - val_loss: 0.3861 - val_accuracy: 0.8682 - val_AUC: 0.9390
    7. Epoch 4/6
    8. 1000/1000 [==============================] - 42s 42ms/step - loss: 0.0849 - accuracy: 0.9706 - AUC: 0.9950 - val_loss: 0.5324 - val_accuracy: 0.8616 - val_AUC: 0.9292
    9. Epoch 5/6
    10. 1000/1000 [==============================] - 43s 43ms/step - loss: 0.0393 - accuracy: 0.9876 - AUC: 0.9986 - val_loss: 0.7693 - val_accuracy: 0.8566 - val_AUC: 0.9132
    11. Epoch 6/6
    12. 1000/1000 [==============================] - 44s 44ms/step - loss: 0.0222 - accuracy: 0.9926 - AUC: 0.9994 - val_loss: 0.9328 - val_accuracy: 0.8584 - val_AUC: 0.9052
    1. plot_metric(history,"AUC")

    Please leave comments in the WeChat official account “Python与算法之美” (Elegance of Python and Algorithms) if you want to communicate with the author about the content. The author will try best to reply given the limited time available.

    image.png