TF: tips 2 - 개념 정리

TF: tips 2 - 개념 정리

2017, May 01    

tf tips. 2 - TensorFlow 개념 정리

tensorflow를 알기 전에는 보통 sklearn을 사용해서 데이터 분석이나 머신러닝을 했었다. 일반적으로 jupyter notebook 환경에서 데이터를 읽고 전처리한 다음, 트레이닝과 테스트 셋으로 나눠서 모델을 학습시키고 성능을 측정한다. 노트북 환경에서는 코드를 입력하면 바로 결과가 출력되기 때문에 데이터가 처리되는 흐름을 직관적으로 이해할 수 있어 편리했다. 그런데 tensorflow는 조금 다른 방식으로 사용해야 하기 때문에, 익숙해지기 조금 어려운 부분이 있다. 간단히 기록을 남겨둔다.

회사에서 업무를 처리할 때 분산처리 컴퓨팅 환경인 spark를 사용한다. spark가 대용량 데이터를 빠르게 처리할 수 있는 힘은 여러 대의 머신에 연산을 할당하고 이를 취합하여 결과를 내는 병렬처리에 있는데, 이 과정에서 lazy evaluation이라는 개념이 사용된다. 즉, 연산을 시켰을 때 이를 그때그때 처리해서 메모리에 들고 있는 것이 아니라, 처리해야 하는 명령을 차곡차곡 쌓다가, show등 과 같은 명령어가 실행되면 가장 최적화된 방식으로 계산을 수행한다.

정확히 같은 개념으로 만들어졌는지는 모르나, tensorflow로 비슷한 면이 있다. 변수를 할당하고 이를 곱했을 때 바로 결과물이 나오지 않는다. tf.Session()으로 세션을 열고 그 안에서 명령을 실행하면 그제서야 결과값이 산출된다. 곱셈같은 간단한 연산을 할 때는 오히려 번거로우나, 내가 원하는 데이터 처리 플로우를 짜고 실행할 수 있다는 장점이 있다. tensorflow를 사용할 때는 오퍼레이션 프로세스를 먼저 정의하고, 세션에서 실행한다는 점을 잊지말자.

예를 들어 다음과 같은 간단한 곱셈 연산을 tensorflow로 해보자.

import tensorflow as tf
a = tf.constant([2])
b = tf.constant([3])

res = tf.multiply(a, b)
res
<tf.Tensor 'Mul:0' shape=(1,) dtype=int32>

res를 실행해도 그 결과가 6이 바로 튀어나오지 않는다. 그냥 그 타입과 쉐입만 출력될 뿐이다. 아직 계산이 이루어지지 않았다.

with tf.Session() as sess:
    tf_res = sess.run(res)
    print(tf_res)
[6]

tf.Session()을 열고, run으로 res를 실행시킨다. 이 부분이 처음에 이해가 바로 되지 않을수도 있는데, 위에서 정의한 resab를 곱한다는 오퍼레이션을 정의한 것으로 이해하면 된다. 즉, 정의한 오퍼레이션을 세션안에서 돌리는 것을 바로 위에서 실행한 셈이다. 한가지 팁으로 with 구문을 사용하면 sess.close()로 세션을 다시 닫지 않아도 되어 매우 편리하다.

간단한 Logistic Regression

tensorflow는 딥러닝 라이브러리로 보통 알려져 있으나, 사실 범용적인 연산을 위한 도구다. MNIST 데이터셋을 가지고 간단하게 Logistic Regression을 만들어 돌려보자.

데이터셋

import matplotlib.pyplot as plt
%matplotlib inline
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz
mnist.train.images[0].shape
(784,)
plt.imshow(mnist.train.images[0].reshape([28,28]))
<matplotlib.image.AxesImage at 0x122e0b630>

png

tensorflow에는 MNIST 데이터가 이미 들어있어서 위와 같이 편리하게 가져올 수 있다. 하나의 손글씨는 위와 같이 784개의 float으로 된 벡터로 표현되어 있다. 이를 시각화하려면 28x28로 쉐입을 변환한다음 plt.imshow를 쓰면 된다.

오퍼레이션 정의

앞서 tensorflow를 사용할 때는 오퍼레이션을 정의하고, 세션에서 이를 실행시킨다고 했다. 기억해야 할 점은, 연산뿐 아니라 인풋 데이터의 형태 역시 정의해줘야 한다.

x = tf.placeholder(tf.float32, [None, 784])

입력 데이터인 xtf.placeholder로, 그 데이터 타입은 float32이며, 형태가 [None, 784]라고 정의했다. tf.placeholder는 이제 만들 데이터 모델에 뚫린 구멍이라고 생각하면 된다. 이 구멍을 통해 우리가 데이터를 집어넣을거다. 그리고 그 데이터의 타입은 float32이며 형태가 [None, 784]라는 얘기다. 784는 하나의 이미지가 784개 픽셀로 구성되어있으니 알겠는데, 왜 None일까? 그 이유는 뒤에 우리가 batch로 모델을 학습시킬 것인데, 이 batch의 크기가 변할 수 있다. 그 값이 변할 수 있으므로 여기서는 None을 넘긴다고 보면 된다.

간단한 logitic regression은 입력 데이터에 웨이트를 곱한다음 바이어스를 더하는 식으로 구성할 수 있다. 그 결과로 출력된 값에 softmax를 입혀서, 0부터 9까지 총 10개의 클래스가 각각 가질 수 있는 확률을 산출하면 된다.

W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))

W에는 웨이트를, b에는 바이어스를 할당하는데 여기서는 placeholder 대신에 Variable을 쓴다. 그 이유는 이 값들은 우리가 넘겨주는 것이 아닌 모델이 학습하면서 업데이트할 값이기 때문이다. 보통 tf.random_normal같은 것을 사용해서 초기값을 설정하는데, 여기서는 간단한 예시이므로 모두 0으로 초기화해서 설정한다.

입력 데이터인 xtf.matmulW를 곱하고 b를 더한다음, tf.nn.softmax로 소프트맥스를 얹는다.

y = tf.nn.softmax(tf.matmul(x, W) + b)

값을 출력하기만 하면 의미가 없다. 모델이 출력한 값(y)과 실제 값(y_)의 차이를 계산해서, 차이가 최소화되는 방향으로 모델이 학습되어야 머신러닝을 하는 의미가 있다. 먼저 y_를 설정한다. 이 값은 MNIST 데이터셋이 제공하는 0-9 사이의 레이블 값으로, 클래스가 10개이므로 인풋데이터와 마찬가지로 [None, 10]을 설정한다.

y_ = tf.placeholder(tf.float32, [None, 10])

보통 regression 문제에서는 예측값과 실제값의 차이를 MSE나 RMSE로 계산하지만, MNIST 문제는 클래시피케이션 문제이므로 다른 방법으로 차이를 계산해야 한다. 여기서는 cross entropy를 사용해 모델의 정확성을 측정한다. 이는 예측값의 로그에 실제값을 곱한 값을 모두 합치고 그 평균값으로 계산한다. reduction_indicesreduce_sum이 합하는 축을 의미한다.

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))

cross_entropy가 최저가 되는 지점을 찾아야 한다. tensorflow는 이를 위한 아주 편리한 기능을 제공한다. tf.train.GradientDescentOptimizer를 불러 learning_rate을 지정한다음, cross_entropy를 최소화하라고만 하면 된다.

train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

그 다음은 성능 측정이다. 트레이닝 셋이 아닌 모델이 아직 보지 못한 테스트셋을 대상으로 accuracy를 산출한다. tf.argmax를 사용해서 소프트맥스로 계산한 확률 중 가장 높은 값과 실제값을 비교해서 correct_prediction을 계산하고, 이를 평균내어 accuracy를 구한다.

correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

오퍼레이션 실행

이제 tf.Session()으로 세션을 열어 앞서 정의한 오퍼레이션을 돌리기만 하면 된다. 그런데 여기서 기억해야할 사항이 있다. 앞서 VariableWb를 정의했는데, 이 값을 초기화(initialize)해줘야 한다.

또, tensorflow는 앞서 정의한 바와 같이 GradientDescent 방식으로 최적의 해를 점진적으로 찾아가는 방식을 취하므로, epoch을 돌면서 batch데이터를 기반으로 학습을 수행한다.

즉, 이 과정을 간단히 정리하면, 먼저 세션을 열고, 초기값을 이니셜라이즈한 다음, 각 epoch에 대해 트레이닝 셋을 불러와서 모델을 학습시킨다. 그리고 적절한 스텝마다 테스트셋을 대상으로 Validation Accuracy를 측정한다. 세션을 열기 전에 epoch 횟수같은 하이퍼파라미터 일부를 먼저 정의해주면 편하다.

epoch = 1000
batch_size = 100
loss_list = []
val_list = []
with tf.Session() as sess:
    tf.global_variables_initializer().run()
    
    for e in range(epoch):
        batch_xs, batch_ys = mnist.train.next_batch(batch_size)
        _, loss = sess.run([train_step, cross_entropy],
                          feed_dict={x: batch_xs, y_: batch_ys})
        loss_list.append({"epoch":e, "loss": loss})
        print("epoch: %s // training loss: %s" %(e, loss))
        
        if (e % 10 == 0) | (e == epoch-1):
            val_acc = (sess.run(accuracy,
                               feed_dict={x: mnist.test.images, y_: mnist.test.labels}))
            val_list.append({"epoch":e, "val_acc": val_acc})
            print("epoch: %s // valiation accuracy: %s" %(e, val_acc))
epoch: 0 // training loss: 2.30259
epoch: 0 // valiation accuracy: 0.4075
epoch: 1 // training loss: 1.89025
epoch: 2 // training loss: 1.91388
epoch: 3 // training loss: 1.67165
epoch: 4 // training loss: 1.29189
epoch: 5 // training loss: 1.37689
epoch: 6 // training loss: 1.2659
epoch: 7 // training loss: 1.3202
epoch: 8 // training loss: 0.992359
epoch: 9 // training loss: 0.804411
epoch: 10 // training loss: 0.671933
epoch: 10 // valiation accuracy: 0.7311
epoch: 11 // training loss: 0.647577
epoch: 12 // training loss: 0.752242
epoch: 13 // training loss: 0.651919
epoch: 14 // training loss: 0.753956
epoch: 15 // training loss: 0.456011
epoch: 16 // training loss: 0.640342
epoch: 17 // training loss: 0.608281
epoch: 18 // training loss: 0.770371
epoch: 19 // training loss: 0.671142
epoch: 20 // training loss: 0.778107
epoch: 20 // valiation accuracy: 0.8062
epoch: 21 // training loss: 0.738233
epoch: 22 // training loss: 0.894308
epoch: 23 // training loss: 0.763473
epoch: 24 // training loss: 0.581832
epoch: 25 // training loss: 0.517427
epoch: 26 // training loss: 0.657688
epoch: 27 // training loss: 0.602923
epoch: 28 // training loss: 0.65742
epoch: 29 // training loss: 0.57282
epoch: 30 // training loss: 0.422698
epoch: 30 // valiation accuracy: 0.8515
epoch: 31 // training loss: 0.641175
epoch: 32 // training loss: 0.770473
epoch: 33 // training loss: 0.482948
epoch: 34 // training loss: 0.60727
epoch: 35 // training loss: 0.38558
epoch: 36 // training loss: 0.649918
epoch: 37 // training loss: 0.938527
epoch: 38 // training loss: 0.697327
epoch: 39 // training loss: 0.534539
epoch: 40 // training loss: 0.386488
epoch: 40 // valiation accuracy: 0.869
epoch: 41 // training loss: 0.440741
epoch: 42 // training loss: 0.581701
epoch: 43 // training loss: 0.525693
epoch: 44 // training loss: 0.572505
epoch: 45 // training loss: 0.474357
epoch: 46 // training loss: 0.403282
epoch: 47 // training loss: 0.48757
epoch: 48 // training loss: 0.3819
epoch: 49 // training loss: 0.338289
epoch: 50 // training loss: 0.447385
epoch: 50 // valiation accuracy: 0.8725
epoch: 51 // training loss: 0.369722
epoch: 52 // training loss: 0.53857
epoch: 53 // training loss: 0.28952
epoch: 54 // training loss: 0.366428
epoch: 55 // training loss: 0.272401
epoch: 56 // training loss: 0.347079
epoch: 57 // training loss: 0.543697
epoch: 58 // training loss: 0.356406
epoch: 59 // training loss: 0.378793
epoch: 60 // training loss: 0.297911
epoch: 60 // valiation accuracy: 0.8772
epoch: 61 // training loss: 0.349307
epoch: 62 // training loss: 0.476142
epoch: 63 // training loss: 0.348735
epoch: 64 // training loss: 0.31766
epoch: 65 // training loss: 0.615396
epoch: 66 // training loss: 0.529816
epoch: 67 // training loss: 0.58085
epoch: 68 // training loss: 0.415417
epoch: 69 // training loss: 0.394774
epoch: 70 // training loss: 0.403933
epoch: 70 // valiation accuracy: 0.8629
epoch: 71 // training loss: 0.390535
epoch: 72 // training loss: 0.502363
epoch: 73 // training loss: 0.518818
epoch: 74 // training loss: 0.479361
epoch: 75 // training loss: 0.558443
epoch: 76 // training loss: 0.686607
epoch: 77 // training loss: 0.475164
epoch: 78 // training loss: 0.36374
epoch: 79 // training loss: 0.540127
epoch: 80 // training loss: 0.729924
epoch: 80 // valiation accuracy: 0.8829
epoch: 81 // training loss: 0.568572
epoch: 82 // training loss: 0.309707
epoch: 83 // training loss: 0.471674
epoch: 84 // training loss: 0.283315
epoch: 85 // training loss: 0.326462
epoch: 86 // training loss: 0.513753
epoch: 87 // training loss: 0.498824
epoch: 88 // training loss: 0.412309
epoch: 89 // training loss: 0.65897
epoch: 90 // training loss: 0.505785
epoch: 90 // valiation accuracy: 0.882
epoch: 91 // training loss: 0.516804
epoch: 92 // training loss: 0.532921
epoch: 93 // training loss: 0.659676
epoch: 94 // training loss: 0.391469
epoch: 95 // training loss: 0.600153
epoch: 96 // training loss: 0.668893
epoch: 97 // training loss: 0.680492
epoch: 98 // training loss: 0.522317
epoch: 99 // training loss: 0.351204
epoch: 100 // training loss: 0.265373
epoch: 100 // valiation accuracy: 0.8948
epoch: 101 // training loss: 0.528275
epoch: 102 // training loss: 0.405095
epoch: 103 // training loss: 0.340304
epoch: 104 // training loss: 0.370836
epoch: 105 // training loss: 0.384847
epoch: 106 // training loss: 0.304942
epoch: 107 // training loss: 0.580107
epoch: 108 // training loss: 0.512071
epoch: 109 // training loss: 0.538989
epoch: 110 // training loss: 0.483554
epoch: 110 // valiation accuracy: 0.896
epoch: 111 // training loss: 0.333933
epoch: 112 // training loss: 0.363784
epoch: 113 // training loss: 0.306778
epoch: 114 // training loss: 0.346928
epoch: 115 // training loss: 0.442725
epoch: 116 // training loss: 0.460574
epoch: 117 // training loss: 0.454377
epoch: 118 // training loss: 0.378364
epoch: 119 // training loss: 0.487164
epoch: 120 // training loss: 0.446942
epoch: 120 // valiation accuracy: 0.8975
epoch: 121 // training loss: 0.417804
epoch: 122 // training loss: 0.475671
epoch: 123 // training loss: 0.224116
epoch: 124 // training loss: 0.486217
epoch: 125 // training loss: 0.599233
epoch: 126 // training loss: 0.434598
epoch: 127 // training loss: 0.486575
epoch: 128 // training loss: 0.521056
epoch: 129 // training loss: 0.277014
epoch: 130 // training loss: 0.345924
epoch: 130 // valiation accuracy: 0.8966
epoch: 131 // training loss: 0.247196
epoch: 132 // training loss: 0.315561
epoch: 133 // training loss: 0.27315
epoch: 134 // training loss: 0.425995
epoch: 135 // training loss: 0.39344
epoch: 136 // training loss: 0.314865
epoch: 137 // training loss: 0.305612
epoch: 138 // training loss: 0.321085
epoch: 139 // training loss: 0.280464
epoch: 140 // training loss: 0.418325
epoch: 140 // valiation accuracy: 0.9014
epoch: 141 // training loss: 0.343054
epoch: 142 // training loss: 0.417974
epoch: 143 // training loss: 0.365933
epoch: 144 // training loss: 0.295097
epoch: 145 // training loss: 0.483411
epoch: 146 // training loss: 0.233611
epoch: 147 // training loss: 0.179555
epoch: 148 // training loss: 0.405014
epoch: 149 // training loss: 0.415728
epoch: 150 // training loss: 0.524662
epoch: 150 // valiation accuracy: 0.8926
epoch: 151 // training loss: 0.482645
epoch: 152 // training loss: 0.39626
epoch: 153 // training loss: 0.291398
epoch: 154 // training loss: 0.214721
epoch: 155 // training loss: 0.422442
epoch: 156 // training loss: 0.351581
epoch: 157 // training loss: 0.451332
epoch: 158 // training loss: 0.461752
epoch: 159 // training loss: 0.666114
epoch: 160 // training loss: 0.426322
epoch: 160 // valiation accuracy: 0.8992
epoch: 161 // training loss: 0.368248
epoch: 162 // training loss: 0.221329
epoch: 163 // training loss: 0.402565
epoch: 164 // training loss: 0.252571
epoch: 165 // training loss: 0.398761
epoch: 166 // training loss: 0.318086
epoch: 167 // training loss: 0.241069
epoch: 168 // training loss: 0.19282
epoch: 169 // training loss: 0.336817
epoch: 170 // training loss: 0.2269
epoch: 170 // valiation accuracy: 0.902
epoch: 171 // training loss: 0.455558
epoch: 172 // training loss: 0.502173
epoch: 173 // training loss: 0.192502
epoch: 174 // training loss: 0.469071
epoch: 175 // training loss: 0.74537
epoch: 176 // training loss: 0.364719
epoch: 177 // training loss: 0.382647
epoch: 178 // training loss: 0.24532
epoch: 179 // training loss: 0.19762
epoch: 180 // training loss: 0.449055
epoch: 180 // valiation accuracy: 0.9029
epoch: 181 // training loss: 0.368268
epoch: 182 // training loss: 0.246325
epoch: 183 // training loss: 0.313043
epoch: 184 // training loss: 0.332933
epoch: 185 // training loss: 0.328881
epoch: 186 // training loss: 0.367361
epoch: 187 // training loss: 0.457497
epoch: 188 // training loss: 0.431087
epoch: 189 // training loss: 0.392294
epoch: 190 // training loss: 0.352272
epoch: 190 // valiation accuracy: 0.9044
epoch: 191 // training loss: 0.281507
epoch: 192 // training loss: 0.488805
epoch: 193 // training loss: 0.275736
epoch: 194 // training loss: 0.371847
epoch: 195 // training loss: 0.371266
epoch: 196 // training loss: 0.457774
epoch: 197 // training loss: 0.557697
epoch: 198 // training loss: 0.339276
epoch: 199 // training loss: 0.438118
epoch: 200 // training loss: 0.290492
epoch: 200 // valiation accuracy: 0.9031
epoch: 201 // training loss: 0.307506
epoch: 202 // training loss: 0.319735
epoch: 203 // training loss: 0.371756
epoch: 204 // training loss: 0.228875
epoch: 205 // training loss: 0.295086
epoch: 206 // training loss: 0.264416
epoch: 207 // training loss: 0.283733
epoch: 208 // training loss: 0.352447
epoch: 209 // training loss: 0.30849
epoch: 210 // training loss: 0.248146
epoch: 210 // valiation accuracy: 0.9048
epoch: 211 // training loss: 0.239028
epoch: 212 // training loss: 0.286788
epoch: 213 // training loss: 0.520755
epoch: 214 // training loss: 0.383323
epoch: 215 // training loss: 0.399157
epoch: 216 // training loss: 0.526942
epoch: 217 // training loss: 0.528346
epoch: 218 // training loss: 0.497659
epoch: 219 // training loss: 0.256993
epoch: 220 // training loss: 0.226957
epoch: 220 // valiation accuracy: 0.9026
epoch: 221 // training loss: 0.538898
epoch: 222 // training loss: 0.359762
epoch: 223 // training loss: 0.284109
epoch: 224 // training loss: 0.362353
epoch: 225 // training loss: 0.298059
epoch: 226 // training loss: 0.381414
epoch: 227 // training loss: 0.369553
epoch: 228 // training loss: 0.28617
epoch: 229 // training loss: 0.249294
epoch: 230 // training loss: 0.188754
epoch: 230 // valiation accuracy: 0.9063
epoch: 231 // training loss: 0.392173
epoch: 232 // training loss: 0.240153
epoch: 233 // training loss: 0.538358
epoch: 234 // training loss: 0.294288
epoch: 235 // training loss: 0.305329
epoch: 236 // training loss: 0.668392
epoch: 237 // training loss: 0.33744
epoch: 238 // training loss: 0.215423
epoch: 239 // training loss: 0.296849
epoch: 240 // training loss: 0.329205
epoch: 240 // valiation accuracy: 0.9061
epoch: 241 // training loss: 0.489052
epoch: 242 // training loss: 0.384589
epoch: 243 // training loss: 0.394772
epoch: 244 // training loss: 0.290132
epoch: 245 // training loss: 0.34469
epoch: 246 // training loss: 0.284525
epoch: 247 // training loss: 0.465197
epoch: 248 // training loss: 0.50372
epoch: 249 // training loss: 0.543966
epoch: 250 // training loss: 0.4005
epoch: 250 // valiation accuracy: 0.8989
epoch: 251 // training loss: 0.512154
epoch: 252 // training loss: 0.328465
epoch: 253 // training loss: 0.268634
epoch: 254 // training loss: 0.335728
epoch: 255 // training loss: 0.354798
epoch: 256 // training loss: 0.540582
epoch: 257 // training loss: 0.33408
epoch: 258 // training loss: 0.471362
epoch: 259 // training loss: 0.329235
epoch: 260 // training loss: 0.363629
epoch: 260 // valiation accuracy: 0.9101
epoch: 261 // training loss: 0.587542
epoch: 262 // training loss: 0.416833
epoch: 263 // training loss: 0.686309
epoch: 264 // training loss: 0.404131
epoch: 265 // training loss: 0.288432
epoch: 266 // training loss: 0.484675
epoch: 267 // training loss: 0.562658
epoch: 268 // training loss: 0.351446
epoch: 269 // training loss: 0.328309
epoch: 270 // training loss: 0.355802
epoch: 270 // valiation accuracy: 0.9093
epoch: 271 // training loss: 0.464158
epoch: 272 // training loss: 0.487349
epoch: 273 // training loss: 0.508012
epoch: 274 // training loss: 0.524916
epoch: 275 // training loss: 0.321248
epoch: 276 // training loss: 0.340699
epoch: 277 // training loss: 0.446558
epoch: 278 // training loss: 0.320727
epoch: 279 // training loss: 0.288172
epoch: 280 // training loss: 0.331293
epoch: 280 // valiation accuracy: 0.9045
epoch: 281 // training loss: 0.263185
epoch: 282 // training loss: 0.378145
epoch: 283 // training loss: 0.446045
epoch: 284 // training loss: 0.333479
epoch: 285 // training loss: 0.326221
epoch: 286 // training loss: 0.210879
epoch: 287 // training loss: 0.328871
epoch: 288 // training loss: 0.14169
epoch: 289 // training loss: 0.165044
epoch: 290 // training loss: 0.288847
epoch: 290 // valiation accuracy: 0.9098
epoch: 291 // training loss: 0.21646
epoch: 292 // training loss: 0.220626
epoch: 293 // training loss: 0.241128
epoch: 294 // training loss: 0.444429
epoch: 295 // training loss: 0.515431
epoch: 296 // training loss: 0.504415
epoch: 297 // training loss: 0.377201
epoch: 298 // training loss: 0.533099
epoch: 299 // training loss: 0.318074
epoch: 300 // training loss: 0.23355
epoch: 300 // valiation accuracy: 0.9074
epoch: 301 // training loss: 0.315848
epoch: 302 // training loss: 0.33829
epoch: 303 // training loss: 0.255283
epoch: 304 // training loss: 0.428463
epoch: 305 // training loss: 0.198863
epoch: 306 // training loss: 0.377765
epoch: 307 // training loss: 0.249294
epoch: 308 // training loss: 0.226611
epoch: 309 // training loss: 0.336695
epoch: 310 // training loss: 0.395869
epoch: 310 // valiation accuracy: 0.9034
epoch: 311 // training loss: 0.302651
epoch: 312 // training loss: 0.229119
epoch: 313 // training loss: 0.229551
epoch: 314 // training loss: 0.429277
epoch: 315 // training loss: 0.225359
epoch: 316 // training loss: 0.139762
epoch: 317 // training loss: 0.335283
epoch: 318 // training loss: 0.339385
epoch: 319 // training loss: 0.211959
epoch: 320 // training loss: 0.381679
epoch: 320 // valiation accuracy: 0.9051
epoch: 321 // training loss: 0.431686
epoch: 322 // training loss: 0.447361
epoch: 323 // training loss: 0.79335
epoch: 324 // training loss: 0.748342
epoch: 325 // training loss: 0.489049
epoch: 326 // training loss: 0.314964
epoch: 327 // training loss: 0.367029
epoch: 328 // training loss: 0.352018
epoch: 329 // training loss: 0.402933
epoch: 330 // training loss: 0.33667
epoch: 330 // valiation accuracy: 0.9064
epoch: 331 // training loss: 0.185718
epoch: 332 // training loss: 0.362338
epoch: 333 // training loss: 0.37964
epoch: 334 // training loss: 0.217633
epoch: 335 // training loss: 0.5304
epoch: 336 // training loss: 0.446779
epoch: 337 // training loss: 0.296315
epoch: 338 // training loss: 0.174377
epoch: 339 // training loss: 0.216697
epoch: 340 // training loss: 0.223447
epoch: 340 // valiation accuracy: 0.9109
epoch: 341 // training loss: 0.299302
epoch: 342 // training loss: 0.262589
epoch: 343 // training loss: 0.620306
epoch: 344 // training loss: 0.638271
epoch: 345 // training loss: 0.290869
epoch: 346 // training loss: 0.395634
epoch: 347 // training loss: 0.358023
epoch: 348 // training loss: 0.347988
epoch: 349 // training loss: 0.440436
epoch: 350 // training loss: 0.307607
epoch: 350 // valiation accuracy: 0.9063
epoch: 351 // training loss: 0.299477
epoch: 352 // training loss: 0.259027
epoch: 353 // training loss: 0.312394
epoch: 354 // training loss: 0.335824
epoch: 355 // training loss: 0.35878
epoch: 356 // training loss: 0.257237
epoch: 357 // training loss: 0.290096
epoch: 358 // training loss: 0.208629
epoch: 359 // training loss: 0.30477
epoch: 360 // training loss: 0.371225
epoch: 360 // valiation accuracy: 0.9093
epoch: 361 // training loss: 0.232679
epoch: 362 // training loss: 0.475089
epoch: 363 // training loss: 0.403038
epoch: 364 // training loss: 0.528409
epoch: 365 // training loss: 0.448161
epoch: 366 // training loss: 0.234477
epoch: 367 // training loss: 0.330411
epoch: 368 // training loss: 0.251709
epoch: 369 // training loss: 0.375114
epoch: 370 // training loss: 0.378745
epoch: 370 // valiation accuracy: 0.9062
epoch: 371 // training loss: 0.418635
epoch: 372 // training loss: 0.268309
epoch: 373 // training loss: 0.549332
epoch: 374 // training loss: 0.526262
epoch: 375 // training loss: 0.433894
epoch: 376 // training loss: 0.258967
epoch: 377 // training loss: 0.379973
epoch: 378 // training loss: 0.377516
epoch: 379 // training loss: 0.459351
epoch: 380 // training loss: 0.359856
epoch: 380 // valiation accuracy: 0.9061
epoch: 381 // training loss: 0.302653
epoch: 382 // training loss: 0.348976
epoch: 383 // training loss: 0.175941
epoch: 384 // training loss: 0.168434
epoch: 385 // training loss: 0.293789
epoch: 386 // training loss: 0.30845
epoch: 387 // training loss: 0.253575
epoch: 388 // training loss: 0.415584
epoch: 389 // training loss: 0.422475
epoch: 390 // training loss: 0.235039
epoch: 390 // valiation accuracy: 0.911
epoch: 391 // training loss: 0.390794
epoch: 392 // training loss: 0.361338
epoch: 393 // training loss: 0.447388
epoch: 394 // training loss: 0.411813
epoch: 395 // training loss: 0.23702
epoch: 396 // training loss: 0.174408
epoch: 397 // training loss: 0.2817
epoch: 398 // training loss: 0.460931
epoch: 399 // training loss: 0.368317
epoch: 400 // training loss: 0.544249
epoch: 400 // valiation accuracy: 0.9037
epoch: 401 // training loss: 0.390912
epoch: 402 // training loss: 0.280986
epoch: 403 // training loss: 0.202746
epoch: 404 // training loss: 0.500325
epoch: 405 // training loss: 0.393544
epoch: 406 // training loss: 0.372975
epoch: 407 // training loss: 0.308857
epoch: 408 // training loss: 0.488195
epoch: 409 // training loss: 0.401988
epoch: 410 // training loss: 0.464
epoch: 410 // valiation accuracy: 0.9112
epoch: 411 // training loss: 0.37533
epoch: 412 // training loss: 0.594764
epoch: 413 // training loss: 0.438606
epoch: 414 // training loss: 0.38879
epoch: 415 // training loss: 0.177234
epoch: 416 // training loss: 0.242151
epoch: 417 // training loss: 0.308295
epoch: 418 // training loss: 0.210013
epoch: 419 // training loss: 0.273072
epoch: 420 // training loss: 0.283622
epoch: 420 // valiation accuracy: 0.9099
epoch: 421 // training loss: 0.238682
epoch: 422 // training loss: 0.460756
epoch: 423 // training loss: 0.36098
epoch: 424 // training loss: 0.364719
epoch: 425 // training loss: 0.312515
epoch: 426 // training loss: 0.483459
epoch: 427 // training loss: 0.383897
epoch: 428 // training loss: 0.171218
epoch: 429 // training loss: 0.501915
epoch: 430 // training loss: 0.270101
epoch: 430 // valiation accuracy: 0.9093
epoch: 431 // training loss: 0.179957
epoch: 432 // training loss: 0.19547
epoch: 433 // training loss: 0.516618
epoch: 434 // training loss: 0.137617
epoch: 435 // training loss: 0.30432
epoch: 436 // training loss: 0.268143
epoch: 437 // training loss: 0.168118
epoch: 438 // training loss: 0.276059
epoch: 439 // training loss: 0.520615
epoch: 440 // training loss: 0.594093
epoch: 440 // valiation accuracy: 0.9051
epoch: 441 // training loss: 0.289449
epoch: 442 // training loss: 0.350477
epoch: 443 // training loss: 0.197343
epoch: 444 // training loss: 0.431612
epoch: 445 // training loss: 0.641088
epoch: 446 // training loss: 0.499685
epoch: 447 // training loss: 0.228935
epoch: 448 // training loss: 0.586801
epoch: 449 // training loss: 0.292608
epoch: 450 // training loss: 0.374081
epoch: 450 // valiation accuracy: 0.9102
epoch: 451 // training loss: 0.248285
epoch: 452 // training loss: 0.509349
epoch: 453 // training loss: 0.55924
epoch: 454 // training loss: 0.531526
epoch: 455 // training loss: 0.459834
epoch: 456 // training loss: 0.353635
epoch: 457 // training loss: 0.332087
epoch: 458 // training loss: 0.258256
epoch: 459 // training loss: 0.170479
epoch: 460 // training loss: 0.191993
epoch: 460 // valiation accuracy: 0.9114
epoch: 461 // training loss: 0.324622
epoch: 462 // training loss: 0.463683
epoch: 463 // training loss: 0.310086
epoch: 464 // training loss: 0.213323
epoch: 465 // training loss: 0.245221
epoch: 466 // training loss: 0.284963
epoch: 467 // training loss: 0.333131
epoch: 468 // training loss: 0.187963
epoch: 469 // training loss: 0.430781
epoch: 470 // training loss: 0.406526
epoch: 470 // valiation accuracy: 0.9122
epoch: 471 // training loss: 0.526605
epoch: 472 // training loss: 0.401463
epoch: 473 // training loss: 0.46112
epoch: 474 // training loss: 0.196326
epoch: 475 // training loss: 0.217856
epoch: 476 // training loss: 0.369439
epoch: 477 // training loss: 0.382322
epoch: 478 // training loss: 0.537759
epoch: 479 // training loss: 0.705745
epoch: 480 // training loss: 0.289019
epoch: 480 // valiation accuracy: 0.9123
epoch: 481 // training loss: 0.381119
epoch: 482 // training loss: 0.276746
epoch: 483 // training loss: 0.163582
epoch: 484 // training loss: 0.195151
epoch: 485 // training loss: 0.439112
epoch: 486 // training loss: 0.293756
epoch: 487 // training loss: 0.245189
epoch: 488 // training loss: 0.351554
epoch: 489 // training loss: 0.477671
epoch: 490 // training loss: 0.428187
epoch: 490 // valiation accuracy: 0.9134
epoch: 491 // training loss: 0.316922
epoch: 492 // training loss: 0.232455
epoch: 493 // training loss: 0.284692
epoch: 494 // training loss: 0.240343
epoch: 495 // training loss: 0.424937
epoch: 496 // training loss: 0.212188
epoch: 497 // training loss: 0.2011
epoch: 498 // training loss: 0.382962
epoch: 499 // training loss: 0.567673
epoch: 500 // training loss: 0.274724
epoch: 500 // valiation accuracy: 0.9125
epoch: 501 // training loss: 0.304383
epoch: 502 // training loss: 0.205526
epoch: 503 // training loss: 0.348608
epoch: 504 // training loss: 0.228986
epoch: 505 // training loss: 0.314203
epoch: 506 // training loss: 0.247295
epoch: 507 // training loss: 0.380325
epoch: 508 // training loss: 0.267708
epoch: 509 // training loss: 0.166405
epoch: 510 // training loss: 0.236964
epoch: 510 // valiation accuracy: 0.9151
epoch: 511 // training loss: 0.180482
epoch: 512 // training loss: 0.371421
epoch: 513 // training loss: 0.280234
epoch: 514 // training loss: 0.503608
epoch: 515 // training loss: 0.173826
epoch: 516 // training loss: 0.368072
epoch: 517 // training loss: 0.152101
epoch: 518 // training loss: 0.258091
epoch: 519 // training loss: 0.279088
epoch: 520 // training loss: 0.320104
epoch: 520 // valiation accuracy: 0.9128
epoch: 521 // training loss: 0.143165
epoch: 522 // training loss: 0.270832
epoch: 523 // training loss: 0.282506
epoch: 524 // training loss: 0.246422
epoch: 525 // training loss: 0.263665
epoch: 526 // training loss: 0.386061
epoch: 527 // training loss: 0.404298
epoch: 528 // training loss: 0.204111
epoch: 529 // training loss: 0.20647
epoch: 530 // training loss: 0.239274
epoch: 530 // valiation accuracy: 0.9144
epoch: 531 // training loss: 0.206007
epoch: 532 // training loss: 0.127365
epoch: 533 // training loss: 0.15613
epoch: 534 // training loss: 0.210701
epoch: 535 // training loss: 0.262719
epoch: 536 // training loss: 0.170202
epoch: 537 // training loss: 0.18292
epoch: 538 // training loss: 0.284873
epoch: 539 // training loss: 0.0464507
epoch: 540 // training loss: 0.0876424
epoch: 540 // valiation accuracy: 0.9131
epoch: 541 // training loss: 0.0747684
epoch: 542 // training loss: 0.222247
epoch: 543 // training loss: 0.448983
epoch: 544 // training loss: 0.334583
epoch: 545 // training loss: 0.0847737
epoch: 546 // training loss: 0.181981
epoch: 547 // training loss: 0.758941
epoch: 548 // training loss: 0.0368999
epoch: 549 // training loss: 0.354437
epoch: 550 // training loss: 0.321239
epoch: 550 // valiation accuracy: 0.9091
epoch: 551 // training loss: 0.303209
epoch: 552 // training loss: 0.255811
epoch: 553 // training loss: 0.378652
epoch: 554 // training loss: 0.215324
epoch: 555 // training loss: 0.459478
epoch: 556 // training loss: 0.360689
epoch: 557 // training loss: 0.378055
epoch: 558 // training loss: 0.173029
epoch: 559 // training loss: 0.325457
epoch: 560 // training loss: 0.190759
epoch: 560 // valiation accuracy: 0.9155
epoch: 561 // training loss: 0.285212
epoch: 562 // training loss: 0.344628
epoch: 563 // training loss: 0.254008
epoch: 564 // training loss: 0.405413
epoch: 565 // training loss: 0.209497
epoch: 566 // training loss: 0.274154
epoch: 567 // training loss: 0.355594
epoch: 568 // training loss: 0.522452
epoch: 569 // training loss: 0.310229
epoch: 570 // training loss: 0.338279
epoch: 570 // valiation accuracy: 0.9136
epoch: 571 // training loss: 0.262892
epoch: 572 // training loss: 0.377455
epoch: 573 // training loss: 0.321644
epoch: 574 // training loss: 0.209261
epoch: 575 // training loss: 0.386532
epoch: 576 // training loss: 0.452714
epoch: 577 // training loss: 0.232019
epoch: 578 // training loss: 0.343246
epoch: 579 // training loss: 0.252109
epoch: 580 // training loss: 0.277323
epoch: 580 // valiation accuracy: 0.9149
epoch: 581 // training loss: 0.402863
epoch: 582 // training loss: 0.353771
epoch: 583 // training loss: 0.281938
epoch: 584 // training loss: 0.522824
epoch: 585 // training loss: 0.15761
epoch: 586 // training loss: 0.422179
epoch: 587 // training loss: 0.176933
epoch: 588 // training loss: 0.48073
epoch: 589 // training loss: 0.518381
epoch: 590 // training loss: 0.509177
epoch: 590 // valiation accuracy: 0.9159
epoch: 591 // training loss: 0.347066
epoch: 592 // training loss: 0.364841
epoch: 593 // training loss: 0.223496
epoch: 594 // training loss: 0.19468
epoch: 595 // training loss: 0.188415
epoch: 596 // training loss: 0.336507
epoch: 597 // training loss: 0.490244
epoch: 598 // training loss: 0.405952
epoch: 599 // training loss: 0.325237
epoch: 600 // training loss: 0.269126
epoch: 600 // valiation accuracy: 0.91
epoch: 601 // training loss: 0.421665
epoch: 602 // training loss: 0.263576
epoch: 603 // training loss: 0.331408
epoch: 604 // training loss: 0.30647
epoch: 605 // training loss: 0.463769
epoch: 606 // training loss: 0.420117
epoch: 607 // training loss: 0.197874
epoch: 608 // training loss: 0.292864
epoch: 609 // training loss: 0.323753
epoch: 610 // training loss: 0.396536
epoch: 610 // valiation accuracy: 0.914
epoch: 611 // training loss: 0.249945
epoch: 612 // training loss: 0.305937
epoch: 613 // training loss: 0.330648
epoch: 614 // training loss: 0.275228
epoch: 615 // training loss: 0.290297
epoch: 616 // training loss: 0.325221
epoch: 617 // training loss: 0.531182
epoch: 618 // training loss: 0.21734
epoch: 619 // training loss: 0.277858
epoch: 620 // training loss: 0.42165
epoch: 620 // valiation accuracy: 0.9148
epoch: 621 // training loss: 0.311067
epoch: 622 // training loss: 0.319813
epoch: 623 // training loss: 0.347459
epoch: 624 // training loss: 0.470038
epoch: 625 // training loss: 0.238993
epoch: 626 // training loss: 0.211119
epoch: 627 // training loss: 0.281111
epoch: 628 // training loss: 0.23323
epoch: 629 // training loss: 0.355397
epoch: 630 // training loss: 0.344143
epoch: 630 // valiation accuracy: 0.9176
epoch: 631 // training loss: 0.30489
epoch: 632 // training loss: 0.521164
epoch: 633 // training loss: 0.229398
epoch: 634 // training loss: 0.268505
epoch: 635 // training loss: 0.387548
epoch: 636 // training loss: 0.265114
epoch: 637 // training loss: 0.249718
epoch: 638 // training loss: 0.472774
epoch: 639 // training loss: 0.28948
epoch: 640 // training loss: 0.245843
epoch: 640 // valiation accuracy: 0.9159
epoch: 641 // training loss: 0.309779
epoch: 642 // training loss: 0.377556
epoch: 643 // training loss: 0.287313
epoch: 644 // training loss: 0.303592
epoch: 645 // training loss: 0.246277
epoch: 646 // training loss: 0.422191
epoch: 647 // training loss: 0.5053
epoch: 648 // training loss: 0.350562
epoch: 649 // training loss: 0.315248
epoch: 650 // training loss: 0.437099
epoch: 650 // valiation accuracy: 0.9176
epoch: 651 // training loss: 0.269042
epoch: 652 // training loss: 0.253894
epoch: 653 // training loss: 0.43648
epoch: 654 // training loss: 0.348148
epoch: 655 // training loss: 0.350213
epoch: 656 // training loss: 0.220559
epoch: 657 // training loss: 0.25412
epoch: 658 // training loss: 0.44877
epoch: 659 // training loss: 0.337493
epoch: 660 // training loss: 0.290692
epoch: 660 // valiation accuracy: 0.9187
epoch: 661 // training loss: 0.294348
epoch: 662 // training loss: 0.241651
epoch: 663 // training loss: 0.239684
epoch: 664 // training loss: 0.332293
epoch: 665 // training loss: 0.400288
epoch: 666 // training loss: 0.411176
epoch: 667 // training loss: 0.22484
epoch: 668 // training loss: 0.342249
epoch: 669 // training loss: 0.244806
epoch: 670 // training loss: 0.19291
epoch: 670 // valiation accuracy: 0.9163
epoch: 671 // training loss: 0.355272
epoch: 672 // training loss: 0.182569
epoch: 673 // training loss: 0.261831
epoch: 674 // training loss: 0.399687
epoch: 675 // training loss: 0.250179
epoch: 676 // training loss: 0.216578
epoch: 677 // training loss: 0.341717
epoch: 678 // training loss: 0.279368
epoch: 679 // training loss: 0.322786
epoch: 680 // training loss: 0.259673
epoch: 680 // valiation accuracy: 0.9173
epoch: 681 // training loss: 0.21153
epoch: 682 // training loss: 0.394257
epoch: 683 // training loss: 0.25517
epoch: 684 // training loss: 0.442028
epoch: 685 // training loss: 0.201213
epoch: 686 // training loss: 0.302617
epoch: 687 // training loss: 0.239355
epoch: 688 // training loss: 0.507132
epoch: 689 // training loss: 0.36232
epoch: 690 // training loss: 0.298419
epoch: 690 // valiation accuracy: 0.9175
epoch: 691 // training loss: 0.299023
epoch: 692 // training loss: 0.22171
epoch: 693 // training loss: 0.364151
epoch: 694 // training loss: 0.382103
epoch: 695 // training loss: 0.227174
epoch: 696 // training loss: 0.351615
epoch: 697 // training loss: 0.225897
epoch: 698 // training loss: 0.462026
epoch: 699 // training loss: 0.255931
epoch: 700 // training loss: 0.314025
epoch: 700 // valiation accuracy: 0.9146
epoch: 701 // training loss: 0.329374
epoch: 702 // training loss: 0.299969
epoch: 703 // training loss: 0.228691
epoch: 704 // training loss: 0.273108
epoch: 705 // training loss: 0.300357
epoch: 706 // training loss: 0.247372
epoch: 707 // training loss: 0.294509
epoch: 708 // training loss: 0.161373
epoch: 709 // training loss: 0.47532
epoch: 710 // training loss: 0.289207
epoch: 710 // valiation accuracy: 0.9172
epoch: 711 // training loss: 0.434644
epoch: 712 // training loss: 0.268992
epoch: 713 // training loss: 0.157123
epoch: 714 // training loss: 0.255926
epoch: 715 // training loss: 0.266053
epoch: 716 // training loss: 0.174128
epoch: 717 // training loss: 0.28774
epoch: 718 // training loss: 0.242172
epoch: 719 // training loss: 0.317374
epoch: 720 // training loss: 0.348031
epoch: 720 // valiation accuracy: 0.9142
epoch: 721 // training loss: 0.456451
epoch: 722 // training loss: 0.264645
epoch: 723 // training loss: 0.522754
epoch: 724 // training loss: 0.357017
epoch: 725 // training loss: 0.30548
epoch: 726 // training loss: 0.369508
epoch: 727 // training loss: 0.324748
epoch: 728 // training loss: 0.325461
epoch: 729 // training loss: 0.305876
epoch: 730 // training loss: 0.193584
epoch: 730 // valiation accuracy: 0.9123
epoch: 731 // training loss: 0.359518
epoch: 732 // training loss: 0.350012
epoch: 733 // training loss: 0.200813
epoch: 734 // training loss: 0.251379
epoch: 735 // training loss: 0.241113
epoch: 736 // training loss: 0.297143
epoch: 737 // training loss: 0.279785
epoch: 738 // training loss: 0.181268
epoch: 739 // training loss: 0.260586
epoch: 740 // training loss: 0.333464
epoch: 740 // valiation accuracy: 0.9166
epoch: 741 // training loss: 0.412619
epoch: 742 // training loss: 0.297434
epoch: 743 // training loss: 0.351488
epoch: 744 // training loss: 0.244739
epoch: 745 // training loss: 0.223248
epoch: 746 // training loss: 0.556826
epoch: 747 // training loss: 0.262486
epoch: 748 // training loss: 0.253501
epoch: 749 // training loss: 0.197081
epoch: 750 // training loss: 0.353667
epoch: 750 // valiation accuracy: 0.9151
epoch: 751 // training loss: 0.307547
epoch: 752 // training loss: 0.248525
epoch: 753 // training loss: 0.244327
epoch: 754 // training loss: 0.257453
epoch: 755 // training loss: 0.341852
epoch: 756 // training loss: 0.235645
epoch: 757 // training loss: 0.439557
epoch: 758 // training loss: 0.236175
epoch: 759 // training loss: 0.231559
epoch: 760 // training loss: 0.327537
epoch: 760 // valiation accuracy: 0.9171
epoch: 761 // training loss: 0.377447
epoch: 762 // training loss: 0.357607
epoch: 763 // training loss: 0.263838
epoch: 764 // training loss: 0.349594
epoch: 765 // training loss: 0.291628
epoch: 766 // training loss: 0.314571
epoch: 767 // training loss: 0.315908
epoch: 768 // training loss: 0.205324
epoch: 769 // training loss: 0.116325
epoch: 770 // training loss: 0.252982
epoch: 770 // valiation accuracy: 0.9155
epoch: 771 // training loss: 0.235351
epoch: 772 // training loss: 0.357128
epoch: 773 // training loss: 0.238146
epoch: 774 // training loss: 0.200822
epoch: 775 // training loss: 0.447385
epoch: 776 // training loss: 0.352931
epoch: 777 // training loss: 0.47296
epoch: 778 // training loss: 0.599095
epoch: 779 // training loss: 0.334219
epoch: 780 // training loss: 0.387223
epoch: 780 // valiation accuracy: 0.9117
epoch: 781 // training loss: 0.367202
epoch: 782 // training loss: 0.206517
epoch: 783 // training loss: 0.284378
epoch: 784 // training loss: 0.43637
epoch: 785 // training loss: 0.299164
epoch: 786 // training loss: 0.192641
epoch: 787 // training loss: 0.310452
epoch: 788 // training loss: 0.259478
epoch: 789 // training loss: 0.202565
epoch: 790 // training loss: 0.325167
epoch: 790 // valiation accuracy: 0.9145
epoch: 791 // training loss: 0.493588
epoch: 792 // training loss: 0.261219
epoch: 793 // training loss: 0.298195
epoch: 794 // training loss: 0.260487
epoch: 795 // training loss: 0.150157
epoch: 796 // training loss: 0.35048
epoch: 797 // training loss: 0.28302
epoch: 798 // training loss: 0.25549
epoch: 799 // training loss: 0.319797
epoch: 800 // training loss: 0.444557
epoch: 800 // valiation accuracy: 0.9145
epoch: 801 // training loss: 0.314499
epoch: 802 // training loss: 0.29731
epoch: 803 // training loss: 0.334452
epoch: 804 // training loss: 0.294237
epoch: 805 // training loss: 0.375911
epoch: 806 // training loss: 0.403276
epoch: 807 // training loss: 0.343874
epoch: 808 // training loss: 0.355797
epoch: 809 // training loss: 0.380446
epoch: 810 // training loss: 0.432105
epoch: 810 // valiation accuracy: 0.9141
epoch: 811 // training loss: 0.530816
epoch: 812 // training loss: 0.226125
epoch: 813 // training loss: 0.242994
epoch: 814 // training loss: 0.222467
epoch: 815 // training loss: 0.285732
epoch: 816 // training loss: 0.284995
epoch: 817 // training loss: 0.338328
epoch: 818 // training loss: 0.335841
epoch: 819 // training loss: 0.33023
epoch: 820 // training loss: 0.213055
epoch: 820 // valiation accuracy: 0.9182
epoch: 821 // training loss: 0.366059
epoch: 822 // training loss: 0.411662
epoch: 823 // training loss: 0.273435
epoch: 824 // training loss: 0.222467
epoch: 825 // training loss: 0.300958
epoch: 826 // training loss: 0.374035
epoch: 827 // training loss: 0.33496
epoch: 828 // training loss: 0.265396
epoch: 829 // training loss: 0.436607
epoch: 830 // training loss: 0.396394
epoch: 830 // valiation accuracy: 0.9152
epoch: 831 // training loss: 0.303396
epoch: 832 // training loss: 0.271111
epoch: 833 // training loss: 0.388439
epoch: 834 // training loss: 0.353509
epoch: 835 // training loss: 0.400967
epoch: 836 // training loss: 0.214627
epoch: 837 // training loss: 0.291021
epoch: 838 // training loss: 0.218683
epoch: 839 // training loss: 0.299856
epoch: 840 // training loss: 0.277438
epoch: 840 // valiation accuracy: 0.9192
epoch: 841 // training loss: 0.234711
epoch: 842 // training loss: 0.276452
epoch: 843 // training loss: 0.268529
epoch: 844 // training loss: 0.133021
epoch: 845 // training loss: 0.218733
epoch: 846 // training loss: 0.273772
epoch: 847 // training loss: 0.192891
epoch: 848 // training loss: 0.35287
epoch: 849 // training loss: 0.37159
epoch: 850 // training loss: 0.236738
epoch: 850 // valiation accuracy: 0.9133
epoch: 851 // training loss: 0.143845
epoch: 852 // training loss: 0.381442
epoch: 853 // training loss: 0.220074
epoch: 854 // training loss: 0.284869
epoch: 855 // training loss: 0.287929
epoch: 856 // training loss: 0.341605
epoch: 857 // training loss: 0.204805
epoch: 858 // training loss: 0.271469
epoch: 859 // training loss: 0.216062
epoch: 860 // training loss: 0.357835
epoch: 860 // valiation accuracy: 0.917
epoch: 861 // training loss: 0.344944
epoch: 862 // training loss: 0.217034
epoch: 863 // training loss: 0.175147
epoch: 864 // training loss: 0.225614
epoch: 865 // training loss: 0.528958
epoch: 866 // training loss: 0.263254
epoch: 867 // training loss: 0.27402
epoch: 868 // training loss: 0.29169
epoch: 869 // training loss: 0.258024
epoch: 870 // training loss: 0.37508
epoch: 870 // valiation accuracy: 0.9145
epoch: 871 // training loss: 0.256477
epoch: 872 // training loss: 0.303802
epoch: 873 // training loss: 0.422129
epoch: 874 // training loss: 0.429617
epoch: 875 // training loss: 0.228193
epoch: 876 // training loss: 0.243051
epoch: 877 // training loss: 0.491666
epoch: 878 // training loss: 0.357437
epoch: 879 // training loss: 0.256116
epoch: 880 // training loss: 0.224811
epoch: 880 // valiation accuracy: 0.9179
epoch: 881 // training loss: 0.349687
epoch: 882 // training loss: 0.381856
epoch: 883 // training loss: 0.249269
epoch: 884 // training loss: 0.459491
epoch: 885 // training loss: 0.39744
epoch: 886 // training loss: 0.207812
epoch: 887 // training loss: 0.244035
epoch: 888 // training loss: 0.185522
epoch: 889 // training loss: 0.363434
epoch: 890 // training loss: 0.215938
epoch: 890 // valiation accuracy: 0.9195
epoch: 891 // training loss: 0.229964
epoch: 892 // training loss: 0.271469
epoch: 893 // training loss: 0.377995
epoch: 894 // training loss: 0.354329
epoch: 895 // training loss: 0.439852
epoch: 896 // training loss: 0.306809
epoch: 897 // training loss: 0.42546
epoch: 898 // training loss: 0.341679
epoch: 899 // training loss: 0.229034
epoch: 900 // training loss: 0.163432
epoch: 900 // valiation accuracy: 0.9172
epoch: 901 // training loss: 0.226936
epoch: 902 // training loss: 0.258233
epoch: 903 // training loss: 0.290312
epoch: 904 // training loss: 0.190728
epoch: 905 // training loss: 0.278983
epoch: 906 // training loss: 0.330679
epoch: 907 // training loss: 0.303222
epoch: 908 // training loss: 0.308874
epoch: 909 // training loss: 0.289852
epoch: 910 // training loss: 0.258356
epoch: 910 // valiation accuracy: 0.913
epoch: 911 // training loss: 0.25502
epoch: 912 // training loss: 0.35167
epoch: 913 // training loss: 0.552955
epoch: 914 // training loss: 0.197594
epoch: 915 // training loss: 0.343562
epoch: 916 // training loss: 0.18248
epoch: 917 // training loss: 0.385704
epoch: 918 // training loss: 0.34092
epoch: 919 // training loss: 0.325594
epoch: 920 // training loss: 0.32262
epoch: 920 // valiation accuracy: 0.9159
epoch: 921 // training loss: 0.306224
epoch: 922 // training loss: 0.355399
epoch: 923 // training loss: 0.223486
epoch: 924 // training loss: 0.355086
epoch: 925 // training loss: 0.338075
epoch: 926 // training loss: 0.185627
epoch: 927 // training loss: 0.35801
epoch: 928 // training loss: 0.414034
epoch: 929 // training loss: 0.325052
epoch: 930 // training loss: 0.274868
epoch: 930 // valiation accuracy: 0.9172
epoch: 931 // training loss: 0.34556
epoch: 932 // training loss: 0.319834
epoch: 933 // training loss: 0.404393
epoch: 934 // training loss: 0.262425
epoch: 935 // training loss: 0.256887
epoch: 936 // training loss: 0.258105
epoch: 937 // training loss: 0.482647
epoch: 938 // training loss: 0.27303
epoch: 939 // training loss: 0.37681
epoch: 940 // training loss: 0.301088
epoch: 940 // valiation accuracy: 0.918
epoch: 941 // training loss: 0.315677
epoch: 942 // training loss: 0.30684
epoch: 943 // training loss: 0.218828
epoch: 944 // training loss: 0.379115
epoch: 945 // training loss: 0.477654
epoch: 946 // training loss: 0.275431
epoch: 947 // training loss: 0.302095
epoch: 948 // training loss: 0.338761
epoch: 949 // training loss: 0.284811
epoch: 950 // training loss: 0.237696
epoch: 950 // valiation accuracy: 0.9189
epoch: 951 // training loss: 0.372927
epoch: 952 // training loss: 0.260738
epoch: 953 // training loss: 0.267696
epoch: 954 // training loss: 0.421447
epoch: 955 // training loss: 0.304379
epoch: 956 // training loss: 0.298697
epoch: 957 // training loss: 0.362644
epoch: 958 // training loss: 0.30521
epoch: 959 // training loss: 0.334756
epoch: 960 // training loss: 0.33426
epoch: 960 // valiation accuracy: 0.919
epoch: 961 // training loss: 0.350213
epoch: 962 // training loss: 0.353042
epoch: 963 // training loss: 0.172833
epoch: 964 // training loss: 0.382719
epoch: 965 // training loss: 0.366468
epoch: 966 // training loss: 0.249598
epoch: 967 // training loss: 0.366024
epoch: 968 // training loss: 0.243455
epoch: 969 // training loss: 0.251323
epoch: 970 // training loss: 0.307071
epoch: 970 // valiation accuracy: 0.9153
epoch: 971 // training loss: 0.412847
epoch: 972 // training loss: 0.360264
epoch: 973 // training loss: 0.22645
epoch: 974 // training loss: 0.192934
epoch: 975 // training loss: 0.215387
epoch: 976 // training loss: 0.209343
epoch: 977 // training loss: 0.291857
epoch: 978 // training loss: 0.311712
epoch: 979 // training loss: 0.285559
epoch: 980 // training loss: 0.127981
epoch: 980 // valiation accuracy: 0.92
epoch: 981 // training loss: 0.269311
epoch: 982 // training loss: 0.336116
epoch: 983 // training loss: 0.270161
epoch: 984 // training loss: 0.28874
epoch: 985 // training loss: 0.14331
epoch: 986 // training loss: 0.474125
epoch: 987 // training loss: 0.331437
epoch: 988 // training loss: 0.221905
epoch: 989 // training loss: 0.354028
epoch: 990 // training loss: 0.225112
epoch: 990 // valiation accuracy: 0.9182
epoch: 991 // training loss: 0.33868
epoch: 992 // training loss: 0.214263
epoch: 993 // training loss: 0.236308
epoch: 994 // training loss: 0.419525
epoch: 995 // training loss: 0.316515
epoch: 996 // training loss: 0.282098
epoch: 997 // training loss: 0.337346
epoch: 998 // training loss: 0.351964
epoch: 999 // training loss: 0.170097
epoch: 999 // valiation accuracy: 0.9185

결과 확인

epoch을 돌면서 중간 중간 validation accuracy를 출력하게 했는데 0.92 정도에 수렴하는 것으로 보아 막 만든 모델 치고 성능이 나쁘지 않음을 확인할 수 있었다. 학습이 진행되면서 lossvalidation accuracy가 어떻게 달라지는지 확인하기 위해 loss_listval_list를 만들어 두었다. pandas로 이를 읽은 다음 플롯화해서 아래와 같이 살펴볼 수 있다.

import pandas as pd

l_df = pd.DataFrame.from_dict(loss_list)
l_df.set_index('epoch', inplace=True)

v_df = pd.DataFrame.from_dict(val_list)
v_df.set_index('epoch', inplace=True)
plt.plot(l_df, label="loss")
plt.plot(v_df, label="val_acc")
plt.legend()
plt.show()

png

epoch을 20번만 돌아도 loss가 급격하게 떨어지면서 val_acc가 크게 상승하는 것을 볼 수 있다.

마무리

MNIST 데이터셋과 tensorflow를 사용해 간단한 logistic regression 모델을 만들어보았다. 실제 사용하게 될 DNN, CNN, RNN은 더 복잡한 형태를 띄고 있겠지만, tensorflow를 사용하는 이상, 이와 같은 구조를 취하고 있을 것이니, 오퍼레이션 구성 -> 실행이라는 기본 구조를 항상 기억하자.