[학습] 모두를 위한 딥러닝 #9 Lab 10. NN, ReLu, Xavier, Dropout and Adam

Lab 10  NN, ReLu, Xavier, Dropout, and Adam

1.  softmax for MNIST

# define cost/loss & optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
    logits=hypothesis, labels=Y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

# initialize
sess = tf.Session()
sess.run(tf.global_variables_initializer())

 

Epoch: 0014 cost = 0.419000337

Epoch: 0015 cost = 0.406490815

Learning Finished!

Accuracy: 0.9035

 

 

 

2.  NN for MNIST with ReLU

 

여러개의 계산 단계를 거침

 

# input place holders

X = tf.placeholder(tf.float32, [None, 784])

Y = tf.placeholder(tf.float32, [None, 10])

# weights & bias for nn layers

W1 = tf.Variable(tf.random_normal([784, 256]))

b1 = tf.Variable(tf.random_normal([256]))

L1 = tf.nn.relu(tf.matmul(X, W1) + b1)

W2 = tf.Variable(tf.random_normal([256, 256]))

b2 = tf.Variable(tf.random_normal([256]))

L2 = tf.nn.relu(tf.matmul(L1, W2) + b2)

W3 = tf.Variable(tf.random_normal([256, 10]))

b3 = tf.Variable(tf.random_normal([10]))

hypothesis = tf.matmul(L2, W3) + b3

# define cost/loss & optimizer

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(

  logits=hypothesis, labels=Y))

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

 

Epoch: 0014 cost = 0.624131458

Epoch: 0015 cost = 0.454633765

Learning Finished!

Accuracy: 0.9455

 

 

 

3.  Xavier, Adam for MNIST

 

초기값이 초기부터 낮음 

 

# input place holders

X = tf.placeholder(tf.float32, [None, 784])

Y = tf.placeholder(tf.float32, [None, 10])

# weights & bias for nn layers

# http://stackoverflow.com/questions/33640581

W1 = tf.get_variable("W1", shape=[784, 256],

                   initializer=tf.contrib.layers.xavier_initializer())

b1 = tf.Variable(tf.random_normal([256]))

L1 = tf.nn.relu(tf.matmul(X, W1) + b1)

W2 = tf.get_variable("W2", shape=[256, 256],

                   initializer=tf.contrib.layers.xavier_initializer())

b2 = tf.Variable(tf.random_normal([256]))

L2 = tf.nn.relu(tf.matmul(L1, W2) + b2)

W3 = tf.get_variable("W3", shape=[256, 10],

                   initializer=tf.contrib.layers.xavier_initializer())

b3 = tf.Variable(tf.random_normal([10]))

hypothesis = tf.matmul(L2, W3) + b3

# define cost/loss & optimizer

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(

  logits=hypothesis, labels=Y))

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

 

 

Epoch: 0001 cost = 0.301498963 Epoch: 0002 cost = 0.107252513 Epoch: 0003 cost = 0.064888892 .... Epoch: 0014 cost = 0.002714260 Epoch: 0015 cost = 0.004707661 Learning Finished! Accuracy: 0.9783

Epoch: 0001 cost = 141.207671860 Epoch: 0002 cost = 38.788445864 Epoch: 0003 cost = 23.977515479

 

4.  Deep NN for MNIST

단순한 Deep 만으로는 효과가 나지 않고 Dropout 적용를 통해서 효과 확인

# dropout (keep_prob) rate  0.7 on training, but should be 1 for testing

keep_prob = tf.placeholder(tf.float32)

W1 = tf.get_variable("W1", shape=[784, 512])

b1 = tf.Variable(tf.random_normal([512]))

L1 = tf.nn.relu(tf.matmul(X, W1) + b1)

L1 = tf.nn.dropout(L1, keep_prob=keep_prob)

W2 = tf.get_variable("W2", shape=[512, 512])

b2 = tf.Variable(tf.random_normal([512]))

L2 = tf.nn.relu(tf.matmul(L1, W2) + b2)

L2 = tf.nn.dropout(L2, keep_prob=keep_prob)

…

# train my model

for epoch in range(training_epochs):

   ...

  for i in range(total_batch):

      batch_xs, batch_ys = mnist.train.next_batch(batch_size)

      feed_dict = {X: batch_xs, Y: batch_ys, keep_prob: 0.7}

      c, _ = sess.run([cost, optimizer], feed_dict=feed_dict)

      avg_cost += c / total_batch

# Test model and check accuracy

correct_prediction = tf.equal(tf.argmax(hypothesis, 1), tf.argmax(Y, 1))

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

print('Accuracy:', sess.run(accuracy, feed_dict={

    X: mnist.test.images, Y: mnist.test.labels, keep_prob: 1}))

 

Epoch: 0014 cost = 0.041290121

Epoch: 0015 cost = 0.043621063

Learning Finished!

Accuracy: 0.9804!!

 

 

5.  Optimizer : Adam 추천 

   

6. Summary

• Softmax VS Neural Nets for MNIST, 90% and 94.5%

• Xavier initialization: 97.8%

• Deep Neural Nets with Dropout: 98%

• Adam and other optimizers

• Exercise: Batch Normalization