Columbia University

ECBM E4040 Neural Networks and Deep Learning. Fall 2020.

Name : Yi-Pei Chan

ECBM E4040 - Task 3: Convolutional Neural Network (CNN)

In this task, you are going to first practice the forward/backward propagation of the convolutional operations with NumPy. After that, we will introduce TensorFlow with which you will create your CNN model for an image classification task.

CNNs:

This is one of the good posts describing CNNs:

https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/

Convolutional neural networks (CNNs) are highly effective for image processing tasks.

Remember when one builds a MLP model, each connection is multiplied by its own weight. When the input dimension or the first layer is very large, we need a giant matrix to store the weights. This could easily become a problem in image processing since the dimension of a vectorized image could easily exceed 1000 (consider CIFAR-10 which has images of shape 32×32=1024, yet the resolution is so low).

In CNN, the weights are shared: the same filter (also known as 'weights' or 'kernel') moves over the input, and at each position an output value is calculated. This means that the same weights are repetitively applied to the entire input, therefore saving a lot of memory.

Image source: here

Convolution: In the picture above, the input is a 7-by-7 image, and the filter is shown as a blue 3-by-3 grid. The filter overlaps with the top-left corner of the input, and we perform an element-wise multiplication followed by a summation, then put the sum into the output matrix. The filter then moves several pixels right, covering a new input area so a new sum can be derived.

Training: One thing to remember is that there would be a lot of filters for each layer in a CNN, and the goal of training is to find the best filters for your task. Each filter tries to capture one specific feature. Typically, in the first convolutional layer which directly looks at your input, the filters try to capture information about color and edges which we know as local features; in higher layers, due to the effect of max-pooling, the receptive-fields of filters becomes large so more global and complex features can be detected.

Architecture: For classification tasks, a CNN usually starts with convolution followed by max-pooling. After that, the feature maps will be flattened so that we could append fully connected layers. Common activation functions include ReLu, ELU in the convolution layers, and softmax in the fully connected layers (to calculate the classification scores).

---

Terminology

• Convolution: element-wise multiplication followed by summation of your input and one of your filters in the CNN context.
• Filter/kernel/weights: a grid or a set of grids typically smaller than your input size that moves over the input space to generate output. Each filter captures one type of feature.
• Feature/feature maps: the output of a hidden layer. Think of it as another representation of your data.
• Pooling: an downsampling operation that joins local information together, so the higher layers' receptive fields can be bigger. The most seen pooling operation is max-pooling, which outputs the maximum of all values inside the pool.
• Flatten: a junction between convolution layers and fully connected layers. Used to turn 2-D feature maps into 1-D. For tasks such as image segmentation where the output also needs to be 2-D, this won't be used.
• Border mode: usually refers to 'VALID' or 'SAME'. Under 'VALID' mode, only when the filter and the input fully overlap can a convolution be conducted; under 'SAME' mode, the output size is the same as the input size (only when the stride is 1), and when the filter and the input don't fully overlap (happens at the edge/corner of input) we pad zeroes (or other designated numbers) and then do convolution.

import numpy as np
import tensorflow as tf

print(tf.__version__)

2.2.0

tf.test.gpu_device_name()

''

Part 1: Getting a sense of convolution

conv2d feedforward

Implement a NumPy naive 2-D convolution feedforward function. We ask you to simply do the element-wise multiplication and summation. Do not worry about the efficiency of your functions. Use as many loops as you like.

__TODO:__ Complete the function conv2d_forward in utils/layer_funcs.py. After that, run the following cell blocks in Jupyter notebook, which will give the output of your convolution function. Detailed instructions have been given in the comments of layer_func.py. The instructors will look at the output to give credits for this task.

def conv2d_forward(x, w, b, pad, stride):
    """
    A Numpy implementation of 2-D image convolution.
    By 'convolution', simple element-wise multiplication and summation will suffice.
    The border mode is 'valid' - Your convolution only happens when your input and your filter fully overlap.
    Another thing to remember is that in TensorFlow, 'padding' means border mode (VALID or SAME). For this practice,
    'pad' means the number rows/columns of zeroes to concatenate before/after the edge of input.

    Inputs:
    :param x: Input data. Should have size (batch, height, width, channels).
    :param w: Filter. Should have size (filter_height, filter_width, channels, num_of_filters).
    :param b: Bias term. Should have size (num_of_filters, ).
    :param pad: Integer. The number of zeroes to pad along the height and width axis.
    :param stride: Integer. The number of pixels to move between 2 neighboring receptive fields.

    :return: A 4-D array. Should have size (batch, new_height, new_width, num_of_filters).

    Note:
    To calculate the output shape of your convolution, you need the following equations:
    new_height = ((height - filter_height + 2 * pad) // stride) + 1
    new_width = ((width - filter_width + 2 * pad) // stride) + 1
    For reference, visit this website:
    https://adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks-Part-2/
    """
    #######################################################################
    #                         TODO: YOUR CODE HERE                        #
    #######################################################################
    #raise NotImplementedError
      
    N, H, W, C = x.shape
    HH, WW, CC, F  = w.shape
    print("H, W, C", H, W, C )
    print("HH, WW, CC, F", HH, WW, CC,F)

    H_out = 1 + (H + 2 * pad - HH) / stride
    W_out = 1 + (W + 2 * pad - WW) / stride

    out = np.zeros((N, int(H_out), int(W_out), F))

    # padding
    x_with_pad = np.pad(x, ((0,0),(pad,pad),(pad,pad),(0,0)), 'constant', constant_values=0)
    _, H, W, _ = x_with_pad.shape
    #print("x_with_pad", x_with_pad.shape)

    # convolving

    for i in range(0, N):
        x_data = x_with_pad[i]
        xx, yy = -1, -1
        for j in range(0, H-HH+1, stride):
            yy += 1
            for k in range(0, W-WW+1, stride):
                xx += 1
                x_rf = x_data[j:j+HH, k:k+WW,:]

                for l in range(0, F):
                    conv_value = np.sum(x_rf * w[:,:,:,l]) + b[l]
                    out[i, yy, xx, l] = conv_value

            xx = -1
        yy =-1

    cache = (x, w, b, pad, stride)

    return(out)    
    #######################################################################
    #                           END OF YOUR CODE                          #
    #######################################################################
    

# tf 2.2.0 implementation
from utils.layer_funcs import conv2d_forward
# Set test parameters.
x_shape = (2, 5, 5, 3) #(batch, height, width, channels)
w_shape = (3, 3, 3, 5) #(filter_height, filter_width, channels, num_of_filters)
channels = w_shape[-1]

x = np.linspace(-0.1, 0.5, num=np.prod(x_shape)).reshape(x_shape)
w = np.linspace(-0.2, 0.3, num=np.prod(w_shape)).reshape(w_shape)
b = np.linspace(-0.1, 0.2, num=channels)
pad = 1
stride = 2
your_feedforward = conv2d_forward(x, w, b, pad, stride)

print("Your feedforward result (size :{}) is: ".format(your_feedforward.shape))
print(your_feedforward)

H, W, C 5 5 3
HH, WW, CC, F 3 3 3 5
Your feedforward result (size :(2, 3, 3, 5)) is: 
[[[[-0.17767004 -0.10534459 -0.03301913  0.03930632  0.11163177]
   [-0.14008665 -0.0678814   0.00432385  0.0765291   0.14873435]
   [-0.09471101 -0.0207628   0.05318542  0.12713363  0.20108184]]

  [[ 0.06484073  0.14191425  0.21898778  0.2960613   0.37313483]
   [ 0.1523525   0.23228839  0.31222428  0.39216017  0.47209606]
   [ 0.04323099  0.12273866  0.20224632  0.28175398  0.36126164]]

  [[-0.08652509 -0.00608585  0.0743534   0.15479265  0.23523189]
   [-0.14941751 -0.06504157  0.01933437  0.10371031  0.18808625]
   [-0.19829711 -0.1162351  -0.03417309  0.04788891  0.12995092]]]


 [[[ 0.38218171  0.46803015  0.55387859  0.63972704  0.72557548]
   [ 0.54755735  0.64004708  0.73253681  0.82502655  0.91751628]
   [ 0.2622959   0.3497671   0.43723831  0.52470951  0.61218071]]

  [[ 0.44821747  0.54557548  0.64293349  0.7402915   0.8376495 ]
   [ 0.49921717  0.60957979  0.7199424   0.83030502  0.94066763]
   [ 0.12234048  0.22213263  0.32192477  0.42171692  0.52150907]]

  [[-0.13520785 -0.04124562  0.05271662  0.14667885  0.24064109]
   [-0.37457528 -0.26991486 -0.16525443 -0.06059401  0.04406641]
   [-0.4498247  -0.35423971 -0.25865471 -0.16306972 -0.06748472]]]]

######################################################
# Verification/checking code. Do not modify it       #
######################################################

X_tf = tf.constant(x, shape=x_shape)
w_tf = tf.constant(w, shape=w_shape)
b_tf = tf.constant(b, shape=channels)

def conv2d_forward_2(x, w, b, stride):
    # stride in tf.nn.conv2d is in the format: [1, x_movement, y_movement, 1]
    feedforward = tf.nn.conv2d(x, w, [1, stride, stride, 1], padding = "SAME")
    # add bias to the conv network
    feedforward = tf.nn.bias_add(feedforward, b)
    return feedforward
print("Is your feedforward correct? {}".format(np.allclose(your_feedforward, conv2d_forward_2(X_tf, w_tf, b_tf, stride))))

Is your feedforward correct? True

conv2d backpropagation (optional, bonus +10 points)

This function is optional, but a bonus 10 points will be given if you solve it correctly.

Implement a NumPy naive 2-D convolution backpropagation function. Again, don't worry about the efficiency.

__TODO:__ Complete the function conv2d_backward in utils/layer_funcs.py. After that, run the following cell blocks, which will give the output of your backpropagation. Detailed instructions have been given in the comments of layer_func.py. We need to judge your output to give you credits.

from utils.layer_funcs import conv2d_backward
# Set test parameters. Please don't change it.
np.random.seed(234)
d_top = np.random.normal(size=your_feedforward.shape)
your_dw, your_db, d_w_shape = conv2d_backward(d_top, x, w, b, pad, stride)

print("Your weights' gradients result (size :{}) is: ".format(d_w_shape))
print(your_dw)
print("Your biases' gradients result is: ")
print(your_db)

Your weights' gradients result (size :(3, 3, 3, 5)) is: 
[[[[-1.5313261  -0.70268556  0.29999267  0.45013902  0.02658925]
   [-1.54455043 -0.70704637  0.30712039  0.44705678  0.03977736]
   [-1.55777476 -0.71140718  0.31424811  0.44397453  0.05296546]]

  [[-1.26240393  0.57156177  0.10980501  0.66016111  0.86998156]
   [-1.26837463  0.58999709  0.11420201  0.66061212  0.88652154]
   [-1.27434533  0.60843241  0.11859902  0.66106313  0.90306152]]

  [[-0.51455235  0.97098958  0.25403568  0.30196443  1.63905567]
   [-0.51641576  0.991624    0.25206962  0.29911685  1.66167077]
   [-0.51827917  1.01225842  0.25010356  0.29626927  1.68428587]]]


 [[[-1.33339813 -0.50007572  0.83947043  0.15420289  0.19049381]
   [-1.36002174 -0.50099608  0.85300835  0.15279624  0.20119824]
   [-1.38664535 -0.50191643  0.86654628  0.1513896   0.21190267]]

  [[-0.92133176  1.09775008  0.67767936  0.23495075  1.52139887]
   [-0.93428666  1.11277168  0.69159827  0.2388919   1.54379329]
   [-0.94724155  1.12779328  0.70551718  0.24283304  1.56618771]]

  [[-0.2520005   1.01625621  0.51526494  0.1086168   2.41361816]
   [-0.26009521  1.02145266  0.52509012  0.10833538  2.45126551]
   [-0.26818993  1.02664911  0.53491531  0.10805395  2.48891287]]]


 [[[-0.94006704  0.51280795  1.25553856  0.29744473  0.58078218]
   [-0.96476826  0.52117651  1.26996836  0.3005436   0.58922554]
   [-0.98946947  0.52954507  1.28439816  0.30364248  0.59766889]]

  [[-1.04756957  1.02580803  1.08727124  0.48606264  1.85262242]
   [-1.06648102  1.03809799  1.10101584  0.49427999  1.8736956 ]
   [-1.08539248  1.05038795  1.11476044  0.50249734  1.89476877]]

  [[-0.63752738  0.1031137   0.13798971  0.30340504  1.81163083]
   [-0.65128985  0.10057825  0.14401772  0.30607337  1.84024718]
   [-0.66505232  0.09804279  0.15004573  0.3087417   1.86886353]]]]
Your biases' gradients result is: 
[-3.21713208  3.73036445  3.45652862  0.97871818  5.56128036]

####################################################
# Verification/checking code. Don't change it.     #
####################################################
d_top_tf = tf.constant(d_top, shape=your_feedforward.shape)
def conv2d_backward_2(x, w, b, d, stride):
    # stride in tf implementation is in the format: [1, x_movement, y_movement, 1]

    dw_tf =  tf.compat.v1.nn.conv2d_backprop_filter(x, w, d, [1, stride, stride, 1], padding = "SAME")
    with tf.GradientTape() as g:
        g.watch(b)
        y = conv2d_forward_2(X_tf, w_tf, b, stride) * d
    dy_dx = g.gradient(y, b)
    return dw_tf, dy_dx

print("Is your weights' gradients correct? {}".format(np.allclose(your_dw, conv2d_backward_2(X_tf, w_shape, b_tf, d_top_tf, stride)[0])))
print("Is your biases' gradients correct? {}".format(np.allclose(your_db, conv2d_backward_2(X_tf, w_shape, b_tf, d_top_tf, stride)[1])))

Is your weights' gradients correct? True
Is your biases' gradients correct? True

max pool feedforward

Implement a NumPy naive max pool feedforward function. We ask you to simply find the max in your pooling window. Also, don't need to worry about the efficiency of your function. Use loops as many as you like.

__TODO:__ Finish the function max_pool_forward in utils/layer_funcs.py. After that, run the following cell blocks, which will give the output of your max pool function. Detailed instructions have been given in the comments of layer_func.py. We need to judge your output to give you credits.

def max_pool_forward(x, pool_size, stride):
    """
    A Numpy implementation of 2-D image max pooling.

    Inputs:
    :params x: Input data. Should have size (batch, height, width, channels).
    :params pool_size: Integer. The size of a window in which you will perform max operations.
    :params stride: Integer. The number of pixels to move between 2 neighboring receptive fields.
    :return :A 4-D array. Should have size (batch, new_height, new_width, num_of_filters).
    """
    #######################################################################
    #                         TODO: YOUR CODE HERE                        #
    #######################################################################
    n, h_in, w_in, c = x.shape
    
    h_pool = pool_size
    w_pool = pool_size
    h_out = 1 + (h_in - h_pool) // stride
    w_out = 1 + (w_in - w_pool) // stride
    output = np.zeros((n, h_out, w_out, c))
    
    for i in range(h_out):
        for j in range(w_out):
            h_start = i * stride
            h_end = h_start + h_pool
            w_start = j * stride
            w_end = w_start + w_pool
            x_prev_slice = x[:, h_start:h_end, w_start:w_end, :]
            output[:, i, j, :] = np.max(x_prev_slice, axis=(1, 2))
    return output
    
    #######################################################################
    #                           END OF YOUR CODE                          #
    #######################################################################

from utils.layer_funcs import max_pool_forward

# Set test parameters.
x_shape = (2, 5, 5, 3) #(batch, height, width, channels)
x = np.linspace(-0.5, 0.5, num=np.prod(x_shape)).reshape(x_shape)
pool_size = 2
stride = 2

your_feedforward = max_pool_forward(x, pool_size, stride)

print("Your feedforward result (size :{}) is: ".format(your_feedforward.shape))
print(your_feedforward)

Your feedforward result (size :(2, 2, 2, 3)) is: 
[[[[-0.37919463 -0.37248322 -0.36577181]
   [-0.33892617 -0.33221477 -0.32550336]]

  [[-0.17785235 -0.17114094 -0.16442953]
   [-0.13758389 -0.13087248 -0.12416107]]]


 [[[ 0.12416107  0.13087248  0.13758389]
   [ 0.16442953  0.17114094  0.17785235]]

  [[ 0.32550336  0.33221477  0.33892617]
   [ 0.36577181  0.37248322  0.37919463]]]]

####################################################
# Verification/checking code. Don't change it.     #
####################################################
X_tf = tf.constant(x, shape=x_shape)

def maxpool_forward_2(x, pool_size, stride):
    maxpool_forward = tf.nn.max_pool(x, [1, pool_size, pool_size, 1], [1, stride, stride, 1], padding='VALID')
    return maxpool_forward

## Print validation result
print("Is your feedforward correct? {}".format(np.allclose(your_feedforward, maxpool_forward_2(X_tf, pool_size, stride))))

Is your feedforward correct? True

max pool backpropogation (optional, bonus +10 points)

This function is optional, but a bonus 10 points will be given if you solve it correctly.

Implement a Numpy naive max pooling backpropagation function. Again, don't worry about the efficiency.

__TODO:__ Finish the function max_pool_backward in utils/layer_funcs.py. After that, run the following cell blocks, which will give the output of your backpropagation. Detailed instructions have been given in the comments of layer_func.py. We need to judge your output to give you credits.

from utils.layer_funcs import max_pool_backward

# Set test parameters. Please don't change it.
np.random.seed(234)
dout = np.random.normal(size=your_feedforward.shape)
dx = max_pool_backward(dout, x, pool_size, stride)

print("Your inputs' gradients result (size :{}) is: ".format(dx.shape))
print(dx)

Your inputs' gradients result (size :(2, 5, 5, 3)) is: 
[[[[ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]]

  [[ 0.          0.          0.        ]
   [ 0.81879162 -1.04355064  0.3509007 ]
   [ 0.          0.          0.        ]
   [ 0.92157829 -0.08738186 -3.12888464]
   [ 0.          0.          0.        ]]

  [[ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]]

  [[ 0.          0.          0.        ]
   [-0.96973267  0.93466579  0.04386634]
   [ 0.          0.          0.        ]
   [ 1.4252155  -0.55706272  0.92682445]
   [ 0.          0.          0.        ]]

  [[ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]]]


 [[[ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]]

  [[ 0.          0.          0.        ]
   [-1.28355374  1.09625686 -1.93247255]
   [ 0.          0.          0.        ]
   [ 0.4789592   1.34458964 -0.17542066]
   [ 0.          0.          0.        ]]

  [[ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]]

  [[ 0.          0.          0.        ]
   [-0.08270438 -0.88845473 -0.30076649]
   [ 0.          0.          0.        ]
   [ 0.90837517 -0.64559131 -1.32361363]
   [ 0.          0.          0.        ]]

  [[ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]
   [ 0.          0.          0.        ]]]]

#######################################
# Checking code. Don't change it.     #
#######################################
d_out_tf = tf.constant(dout, shape=your_feedforward.shape)

def max_pool_backward_2(x, d):
    with tf.GradientTape() as g:
        g.watch(x)
        y = maxpool_forward_2(X_tf, pool_size, stride) * d
    dy_dx = g.gradient(y, x)
    return dy_dx
# ## Print validation result
print('*'*50)
print("Is your inputs' gradients correct? {}".format(np.allclose(dx, max_pool_backward_2(X_tf, d_out_tf))))

**************************************************
Is your inputs' gradients correct? True

Part 2: TensorFlow CNN

In this part we will construct the CNN in TensorFlow. We will implement a CNN similar to the LeNet structure.

TensorFlow offers many useful resources and functions which help developers build the net in a high-level fashion, such as functions in the layer module. However, we will build the network by ourself for this homework for better understanding. By utilizing functions in tf.nn that exist for Neural Network structuring and training, we can build out our own layers and network modules rather quickly.

Also, we will introduce a visualization tool called Tensorboard. You can use TensorBoard to visualize your TensorFlow graph, plot quantitative metrics about the execution of your graph, and show additional data that pass through it.

Resources and References:

• TensorBoard: Visualizing Learning
  
• Convolutional Neural Networks (LeNet) - DeepLearning 0.1 documentation
  
• LeNet-5, convolutional neural networks

Quick guide for Tensorboard

Tensorboard is a powerful tool provided by TensorFlow. It allows developers to check their graph and trend of parameters. This guide will give you a basic under standing on how to set up Tensorboard graph in your code, start tensorboard on your local machine/GCP instance and how to access the interface.

For complete instructions, check the official guide on Tensorflow web site here.

How to start tensorboard

Local

To start your Tensorboard on your local machine, you need to specify a log directory for the service to fetch the graph. For example, in your command line, type:

$ tensorboard --logdir="~/log"

Then, Tensorboard will start running. By default, it will be running on port 6006:

TensorBoard 1.8.0 at http://localhost:6006 (Press CTRL+C to quit)

Make sure Tensorboard is running, you can visit http://localhost:6006 In your browser and you should be able to see the main page of Tensorboard. If the page is shown as below, it means Tensorboard is running correctly. The report is due to lack of event file, but we can just leave it there for now.

GCP

To set up the Tensorboard on GCP is the same as above. However, we're not able to check the Tensorboard UI directly through our browser. In order to visit the page through our local browser, we should link the port of our local machine to the port on GCP. It is similar to what we did previously for Jupyter Notebook.

In the command line on your local machine, type:

$ gcloud compute ssh --ssh-flag="-L 9999:localhost:9999 -L 9998:localhost:6006" "ecbm4040@YOUR_INSTANCE"

This will bind your port of your local machine to the port on GCP instance. In this case, your local port 9999 is binded with 9999 on GCP, while local port 9998 is binded with 6006 on GCP. You can change whatever port you like as long as it does not confilct with your local services.

After connecting to GCP using the command, you will be able to see the result page.

Export Tensorboard events into log directory

To write summary data for visualization in TensorBoard, we should use class tf.summary. This class will save your network graph sturcuture and all the variable summary.

For example, the following code from the LeNet_trainer.py will save all the parameter summary and marked with iteration_total. These data will be displayed in the Tensorboard later on.

# ... previous code ...
# ...
                # Set up summary writers to write the summaries to disk in a different logs directory:
                def summary(self):
                    self.current_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
                    self.train_log_dir = 'logs/gradient_tape/' + self.current_time + '/train'
                    self.test_log_dir = 'logs/gradient_tape/' + self.current_time + '/test'
                    self.train_summary_writer = tf.summary.create_file_writer(self.train_log_dir)
                    self.test_summary_writer = tf.summary.create_file_writer(self.test_log_dir)

                # Use tf.summary.scalar() to log metrics (loss and accuracy) during training/testing within the scope of
                # the summary writers to write the summaries to disk.
                def train_epoch(self, epoch):
                    ...previous code...
                    for images, labels in train_ds:
                    self.train_step(images, labels)
                    with self.train_summary_writer.as_default():
                        tf.summary.scalar('loss', self.train_loss.result(), step=epoch)
                        tf.summary.scalar('accuracy', self.train_accuracy.result(), step=epoch)

                    for test_images, test_labels in test_ds:
                        self.test_step(test_images, test_labels)

                    with self.test_summary_writer.as_default():
                        tf.summary.scalar('loss', self.test_loss.result(), step=epoch)
                        tf.summary.scalar('accuracy', self.test_accuracy.result(), step=epoch)

Check the graph and summary in Tensorboard

After executing the program once, you should able to see the metrics displayed in the tensorboard.

Also, you may able zoom in or zoom out or click into the layer block to check all the variables and tensor operations in the graph, check the trend of the variables and the distribution of those in Scalar, Distributions and Histograms. You may explore the tensorboard by yourself and take advantage to it for debuging the nerwork structure.

__TODO:__ You will try to achieve your own CNN model that has similar structure to LeNet, show the model graph in tensorboard, and get a model with 90% or higher accuracy using the data we provide you.

An example code is included in utils/neuralnets/cnn/model_LeNet.py. This sample is used as a guide line for how to build a Neural Net model in Tensorflow. Feel free to copy or utilize the code we give you.

__TODO:__

1. Edit the file utils/neuralnets/cnn/my_model_LeNet.py. Create your own CNN that is based on the LeNet structure to achieve at least 90% test accuracy.
2. Print out the training process and the best validation accuracy, save the .meta model in model/ folder.
3. Attach a screenshot of your tensorboard graph in the markdown cell below. Double click the cell and replace the example image with your own image. Here is a Markdown Cheetsheet that may also help.

Hint:

1. You can copy and edit the code from model_LeNet.py

Sequential implementation.

import datetime
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten, Conv2D, AveragePooling2D
from tensorflow.keras import Model
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.losses import categorical_crossentropy
import datetime

fashion_mnist = tf.keras.datasets.fashion_mnist

(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

# Add a channels dimension
x_train = x_train[..., tf.newaxis].astype("float32")
x_test = x_test[..., tf.newaxis].astype("float32")

print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
print(x_train[0].shape, 'image shape')

x_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
(28, 28, 1) image shape

from utils.neuralnets.cnn.LeNet_trainer import MyLeNet_trainer

# Need to check whether the updated model be imported.


trainer = MyLeNet_trainer(x_train,y_train,x_test,y_test, epochs=20,batch_size=256,lr=1e-3)
trainer.run()

This is LeNet in the 3rd file.
Training Epoch 1
Loss: 1.6570978164672852, Accuracy: 81.09833526611328, Test Loss: 1.6153115034103394, Test Accuracy: 84.8499984741211
Training Epoch 2
Loss: 1.5940110683441162, Accuracy: 86.961669921875, Test Loss: 1.5854144096374512, Test Accuracy: 87.8499984741211
Training Epoch 3
Loss: 1.578779935836792, Accuracy: 88.52499389648438, Test Loss: 1.584221363067627, Test Accuracy: 87.81999969482422
Training Epoch 4
Loss: 1.572750210762024, Accuracy: 89.01666259765625, Test Loss: 1.5856096744537354, Test Accuracy: 87.62999725341797
Training Epoch 5
Loss: 1.566405177116394, Accuracy: 89.59333038330078, Test Loss: 1.5740115642547607, Test Accuracy: 88.77999877929688
Training Epoch 6
Loss: 1.5609363317489624, Accuracy: 90.10833740234375, Test Loss: 1.5735224485397339, Test Accuracy: 88.73999786376953
Training Epoch 7
Loss: 1.558279275894165, Accuracy: 90.35833740234375, Test Loss: 1.5756558179855347, Test Accuracy: 88.51000213623047
Training Epoch 8
Loss: 1.5545527935028076, Accuracy: 90.7683334350586, Test Loss: 1.5662305355072021, Test Accuracy: 89.52999877929688
Training Epoch 9
Loss: 1.5514495372772217, Accuracy: 91.07499694824219, Test Loss: 1.5698177814483643, Test Accuracy: 89.01000213623047
Training Epoch 10
Loss: 1.5491091012954712, Accuracy: 91.23500061035156, Test Loss: 1.5692371129989624, Test Accuracy: 89.1300048828125
Training Epoch 11
Loss: 1.5475701093673706, Accuracy: 91.375, Test Loss: 1.5615917444229126, Test Accuracy: 89.87000274658203
Training Epoch 12
Loss: 1.5452070236206055, Accuracy: 91.63166809082031, Test Loss: 1.5612808465957642, Test Accuracy: 89.93000030517578
Training Epoch 13
Loss: 1.5431175231933594, Accuracy: 91.9000015258789, Test Loss: 1.5664880275726318, Test Accuracy: 89.45999908447266
Training Epoch 14
Loss: 1.542034387588501, Accuracy: 91.95832824707031, Test Loss: 1.5646589994430542, Test Accuracy: 89.63999938964844
Training Epoch 15
Loss: 1.5401023626327515, Accuracy: 92.14500427246094, Test Loss: 1.5643384456634521, Test Accuracy: 89.72000122070312
Training Epoch 16
Loss: 1.5394166707992554, Accuracy: 92.19999694824219, Test Loss: 1.5595780611038208, Test Accuracy: 90.27999877929688
Training Epoch 17
Loss: 1.5374468564987183, Accuracy: 92.38999938964844, Test Loss: 1.5598125457763672, Test Accuracy: 90.09000396728516
Training Epoch 18
Loss: 1.5356484651565552, Accuracy: 92.56666564941406, Test Loss: 1.5573948621749878, Test Accuracy: 90.37000274658203
Training Epoch 19
Loss: 1.5341793298721313, Accuracy: 92.74333190917969, Test Loss: 1.5576778650283813, Test Accuracy: 90.29000091552734
Training Epoch 20
Loss: 1.5334855318069458, Accuracy: 92.80833435058594, Test Loss: 1.5561723709106445, Test Accuracy: 90.41000366210938

Notes: Don't forget to set up a firewall for Tensorboard just like the way we did for jupyter notebook. This time the port is 6006.

%load_ext tensorboard
# Load tensorboard on google cloud with specific port
%tensorboard --logdir=logs/gradient_tape/ --port=6006 --bind_all

__TODO:__ Show accuracy and attach the tensorboard graph.

[Insert your screenshot here.]

# Put the screenshot here
#