MNIST Classifier¶
In this example, we will predict handwritten digits in the MNIST dataset using a multi-layer perceptron (MLP). This is similar to the tutorial notebook, but with an added comparison with a standard model.
Of course, other neural network architectures such as convolutional neural networks (CNNs) are better suited for this task, but for this example we will stick with MLPs.
Setup¶
First, let’s prepare the imports.
import keras
import numpy as np
2024-06-22 06:28:34.410764: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-06-22 06:28:34.411126: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-06-22 06:28:34.413415: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-06-22 06:28:34.443327: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-22 06:28:35.153872: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Define constants relating to the data.
NUM_CLASSES = 10 # 10 distinct classes, 0 to 9
INPUT_SHAPE = (28, 28) # 28 x 28 greyscale images
Load the data from the mnist dataset.
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
Perform some preprocessing.
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255
y_train = keras.utils.to_categorical(y_train, NUM_CLASSES)
y_test = keras.utils.to_categorical(y_test, NUM_CLASSES)
Defining the Model¶
As mentioned, we will be using a MLP for the model. However, instead of using keras’s default Dense layer, we will use keras_mml’s DenseMML layer (which stands for Dense Matrix-Multiplication-less). DenseMML is designed to be a direct replacement for Dense layers in fully-connected layers, so we don’t have to change the architecture of the model much.
import keras_mml
Define the Sequential model.
model = keras.Sequential(
[
keras.Input(shape=INPUT_SHAPE),
keras.layers.Flatten(),
keras_mml.layers.DenseMML(256),
keras_mml.layers.DenseMML(256),
keras_mml.layers.DenseMML(256),
keras.layers.Dense(NUM_CLASSES, activation="softmax"), # The last layer needs to be `Dense` for the output to work
],
name="Classifier-MML"
)
model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
model.summary()
Model: "Classifier-MML"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ flatten (Flatten) │ (None, 784) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_mml (DenseMML) │ (None, 256) │ 200,960 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_mml_1 (DenseMML) │ (None, 256) │ 65,792 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_mml_2 (DenseMML) │ (None, 256) │ 65,792 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense (Dense) │ (None, 10) │ 2,570 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 335,114 (1.28 MB)
Trainable params: 335,114 (1.28 MB)
Non-trainable params: 0 (0.00 B)
We can now train the model.
model.fit(x_train, y_train, batch_size=128, epochs=20, validation_split=0.1)
Epoch 1/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 4s 7ms/step - accuracy: 0.5580 - loss: 1.5937 - val_accuracy: 0.8792 - val_loss: 0.4351
Epoch 2/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.8768 - loss: 0.4416 - val_accuracy: 0.8997 - val_loss: 0.3379
Epoch 3/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9024 - loss: 0.3393 - val_accuracy: 0.9185 - val_loss: 0.2897
Epoch 4/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9121 - loss: 0.3059 - val_accuracy: 0.9250 - val_loss: 0.2600
Epoch 5/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9194 - loss: 0.2692 - val_accuracy: 0.9417 - val_loss: 0.2011
Epoch 6/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9280 - loss: 0.2414 - val_accuracy: 0.9450 - val_loss: 0.1895
Epoch 7/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9319 - loss: 0.2255 - val_accuracy: 0.9368 - val_loss: 0.2094
Epoch 8/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9350 - loss: 0.2182 - val_accuracy: 0.9453 - val_loss: 0.1827
Epoch 9/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9372 - loss: 0.2084 - val_accuracy: 0.9333 - val_loss: 0.2263
Epoch 10/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 6ms/step - accuracy: 0.9328 - loss: 0.2149 - val_accuracy: 0.9483 - val_loss: 0.1709
Epoch 11/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9438 - loss: 0.1825 - val_accuracy: 0.9467 - val_loss: 0.1738
Epoch 12/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9430 - loss: 0.1863 - val_accuracy: 0.9488 - val_loss: 0.1801
Epoch 13/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 5s 6ms/step - accuracy: 0.9454 - loss: 0.1793 - val_accuracy: 0.9483 - val_loss: 0.1670
Epoch 14/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9485 - loss: 0.1700 - val_accuracy: 0.9537 - val_loss: 0.1532
Epoch 15/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9454 - loss: 0.1784 - val_accuracy: 0.9543 - val_loss: 0.1569
Epoch 16/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 6ms/step - accuracy: 0.9499 - loss: 0.1650 - val_accuracy: 0.9488 - val_loss: 0.1692
Epoch 17/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9498 - loss: 0.1637 - val_accuracy: 0.9540 - val_loss: 0.1545
Epoch 18/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.9505 - loss: 0.1601 - val_accuracy: 0.9475 - val_loss: 0.1764
Epoch 19/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 6ms/step - accuracy: 0.9491 - loss: 0.1603 - val_accuracy: 0.9530 - val_loss: 0.1568
Epoch 20/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 6ms/step - accuracy: 0.9527 - loss: 0.1533 - val_accuracy: 0.9568 - val_loss: 0.1516
<keras.src.callbacks.history.History at 0x7fb34a92d930>
Once the model is trained, let’s evaluate it.
score = model.evaluate(x_test, y_test, verbose=0)
print("Test loss:", score[0])
print("Test accuracy:", score[1])
Test loss: 0.18468262255191803
Test accuracy: 0.9449999928474426
A Comparison - An MLP Using Normal Dense Layers¶
Let’s compare our model’s performance to a model that uses the regular Dense layers.
model = keras.Sequential(
[
keras.Input(shape=INPUT_SHAPE),
keras.layers.Flatten(),
keras.layers.Dense(256),
keras.layers.Dense(256),
keras.layers.Dense(256),
keras.layers.Dense(NUM_CLASSES, activation="softmax"),
],
name="Classifier-Normal"
)
model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
model.summary()
Model: "Classifier-Normal"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ flatten_1 (Flatten) │ (None, 784) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 256) │ 200,960 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 256) │ 65,792 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_3 (Dense) │ (None, 256) │ 65,792 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_4 (Dense) │ (None, 10) │ 2,570 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 335,114 (1.28 MB)
Trainable params: 335,114 (1.28 MB)
Non-trainable params: 0 (0.00 B)
We’ll train the model using the same batch_size, epochs, and validation_split.
model.fit(x_train, y_train, batch_size=128, epochs=20, validation_split=0.1)
Epoch 1/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 5ms/step - accuracy: 0.8458 - loss: 0.5085 - val_accuracy: 0.9172 - val_loss: 0.2930
Epoch 2/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.9036 - loss: 0.3322 - val_accuracy: 0.9267 - val_loss: 0.2615
Epoch 3/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.9113 - loss: 0.3115 - val_accuracy: 0.9320 - val_loss: 0.2490
Epoch 4/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.9122 - loss: 0.3044 - val_accuracy: 0.9300 - val_loss: 0.2516
Epoch 5/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.9142 - loss: 0.3023 - val_accuracy: 0.9253 - val_loss: 0.2622
Epoch 6/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.9166 - loss: 0.2931 - val_accuracy: 0.9267 - val_loss: 0.2594
Epoch 7/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.9178 - loss: 0.2913 - val_accuracy: 0.9267 - val_loss: 0.2613
Epoch 8/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.9178 - loss: 0.2854 - val_accuracy: 0.9292 - val_loss: 0.2605
Epoch 9/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.9201 - loss: 0.2866 - val_accuracy: 0.9328 - val_loss: 0.2462
Epoch 10/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 3s 4ms/step - accuracy: 0.9218 - loss: 0.2766 - val_accuracy: 0.9302 - val_loss: 0.2546
Epoch 11/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.9206 - loss: 0.2796 - val_accuracy: 0.9263 - val_loss: 0.2593
Epoch 12/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.9205 - loss: 0.2800 - val_accuracy: 0.9245 - val_loss: 0.2725
Epoch 13/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.9201 - loss: 0.2770 - val_accuracy: 0.9288 - val_loss: 0.2512
Epoch 14/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.9231 - loss: 0.2727 - val_accuracy: 0.9333 - val_loss: 0.2423
Epoch 15/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.9245 - loss: 0.2643 - val_accuracy: 0.9322 - val_loss: 0.2563
Epoch 16/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.9245 - loss: 0.2661 - val_accuracy: 0.9275 - val_loss: 0.2556
Epoch 17/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.9225 - loss: 0.2721 - val_accuracy: 0.9278 - val_loss: 0.2672
Epoch 18/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.9248 - loss: 0.2717 - val_accuracy: 0.9302 - val_loss: 0.2469
Epoch 19/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.9256 - loss: 0.2618 - val_accuracy: 0.9327 - val_loss: 0.2480
Epoch 20/20
422/422 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.9253 - loss: 0.2634 - val_accuracy: 0.9307 - val_loss: 0.2595
<keras.src.callbacks.history.History at 0x7fb33c5162c0>
Again, we evaluate the model.
score = model.evaluate(x_test, y_test, verbose=0)
print("Test loss:", score[0])
print("Test accuracy:", score[1])
Test loss: 0.3022889792919159
Test accuracy: 0.9175999760627747
Notice that the accuracy of the normal model is actually less accurate than the MML model. Regardless, this shows that, even though the model itself does not use matrix multiplications at all, our model performs similarly to the standard model.
Conclusion¶
In this example, we have seen how to use DenseMML to create a simple MNIST handwritten digits classifier. We also compared the performance of DenseMML with the standard Dense layer and found that their performances are similar.