tensorflow2学习笔记---tensorflow2的变化

tensorflow2的变化

官网

Effective TensorFlow 2 | TensorFlow Core

主要的变化

API Cleanup

删除了一些API：

• tf.app
• tf.flags
• tf.logging

将tf下的一些function移动到了子包，例如tf.math。还有一些功能用其他function代替，如tf.summary、tf.keras.metrics和tf.keras.optimizers。

Eager execution

1.x需要用户手动创建graph，并且使用seesion来调用逻辑。2.0的所有方法会立即执行，seesion相关的内容会在框架内部实现。所有代码都按顺序执行

No more globals

默认命名空间不会维持用户的变量，如果它出了自己的作用域将会被回收

Functions，not seesions

在2.0中可以通过tf.function()将Python方法标记为JIT编译，并运行在一个独立的graph中。

好处：

• 性能：方法执行效率提升（node pruning, kernel fusion）
• 可移植性：方法可以导出导入

# TensorFlow 1.X
outputs = session.run(f(placeholder), feed_dict={placeholder: input})
# TensorFlow 2.0
outputs = f(input)

2.x的官网demo

都是对手写数字的识别，第一个使用NN，第二个使用CNN

高层API

import tensorflow as tf
from util import DataLoader as dl

mnist = tf.keras.datasets.mnist

# 获取手写数字数据 x_train的维度是(60000, 28, 28)
(x_train, y_train), (x_test, y_test) = dl.load_data()
print(x_train.shape)

# 将数据从整型转换成浮点型
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)

model.evaluate(x_test, y_test, verbose=2)

底层API

import tensorflow as tf
from tensorflow.keras.layers import Dense, Flatten, Conv2D
from tensorflow.keras import Model
from util import DataLoader as dl

(x_train, y_train), (x_test, y_test) = dl.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

# 为什么要增加一个维度？？？
x_train = x_train[..., tf.newaxis].astype("float32")
x_test = x_test[..., tf.newaxis].astype("float32")

# 数据分片，shuffle：缓冲区大小，用于打乱数据顺序。batch：每个分片的数据量
train_ds = tf.data.Dataset.from_tensor_slices(
    (x_train, y_train)).shuffle(10000).batch(32)

test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32)

# 定义模型
class MyModel(Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = Conv2D(32, 3, activation='relu')
        self.flatten = Flatten()
        self.d1 = Dense(128, activation='relu')
        self.d2 = Dense(10)

    def call(self, x):
        x = self.conv1(x)
        x = self.flatten(x)
        x = self.d1(x)
        return self.d2(x)

# 选择损失函数和梯度下降算法
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
optimizer = tf.keras.optimizers.Adam()

# 衡量指标来度量损失值和准确率
train_loss = tf.keras.metrics.Mean(name='train_loss')
train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')

test_loss = tf.keras.metrics.Mean(name='test_loss')
test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='test_accuracy')

model = MyModel()

@tf.function
def train_step(images, labels):
    with tf.GradientTape() as tape:
        predictions = model(images)
        # 当前权重计算出y 和 y^的差异值
        loss = loss_object(labels, predictions)
        # 对参数求导
        gradients = tape.gradient(loss, model.trainable_variables)
        # 梯度下降对参数进行修正，zip是啥意思？？？
        optimizer.apply_gradients(zip(gradients, model.trainable_variables))

        print('before metrics')
        train_loss(loss)
        train_accuracy(labels, predictions)

@tf.function
def test_step(images, labels):
    predictions = model(images)
    t_loss = loss_object(labels, predictions)

    test_loss(t_loss)
    test_accuracy(labels, predictions)

EPOCHS = 5
for epoch in range(EPOCHS):
    train_loss.reset_states()
    train_accuracy.reset_states()
    test_loss.reset_states()
    test_accuracy.reset_states()

    for images, labels in train_ds:
        train_step(images, labels)

    for test_images, test_labels in test_ds:
        test_step(test_images, test_labels)

    template = 'Epoch{},Loss:{},Accuracy:{},Test Loss:{},Test Accuracy:{}'
    print(template.format(epoch + 1, train_loss.result(), train_accuracy.result(), test_loss.result(),
                          test_accuracy.result()))

2.0使用的一些建议

Refactor your code into smaller funtions

可以将函数进行拆分，然后在最上层进行调用，并只在最上层的函数加上@tf.function注解

Use Keras layers and models to manage variables

Keras和model提供了variables和trainable_variables两个属性，它们递归收集了所有变量。

对比是否使用Keras

这种方式需要单独初始化各个参数，包括其维度和初始值，并自己管理参数

def dense(x, W, b):
  return tf.nn.sigmoid(tf.matmul(x, W) + b)

@tf.function
def multilayer_perceptron(x, w0, b0, w1, b1, w2, b2 ...):
  x = dense(x, w0, b0)
  x = dense(x, w1, b1)
  x = dense(x, w2, b2)
  ...

# You still have to manage w_i and b_i, and their shapes are defined far away from the code.

这种方式只需要设定好每个hidden layer的units数量，训练时喂数据就行，参数由keras自己管理。并且可以获取每一层的参数

# Each layer can be called, with a signature equivalent to linear(x)
layers = [tf.keras.layers.Dense(hidden_size, activation=tf.nn.sigmoid) for _ in range(n)]
perceptron = tf.keras.Sequential(layers)

# layers[3].trainable_variables => returns [w3, b3]
# perceptron.trainable_variables => returns [w0, b0, ...]

layers/model是继承自tf.train.Checkpointable，它已经被@tf.function标记。下面这句话没读太懂

Keras layers/models inherit from tf.train.Checkpointable and are integrated with @tf.function, which makes it possible to directly checkpoint or export SavedModels from Keras objects. You do not necessarily have to use Keras's .fit() API to take advantage of these integrations.

Conbine tf.data.Datasets and @tf.function

被@tf.function标记的方法，其中的循环/迭代 会被替换成AutoGraph，其中会对硬盘中的数据进行异步的预读和流处理，不用太担心内存问题。

@tf.function
def train(model, dataset, optimizer):
  for x, y in dataset:

...省略部分

Take advantage of AutoGraph with Python control flow

没看懂

AutoGraph provides a way to convert data-dependent control flow into graph-mode equivalents like tf.cond and tf.while_loop.
One common place where data-dependent control flow appears is in sequence models. tf.keras.layers.RNN wraps an RNN cell, allowing you to either statically or dynamically unroll the recurrence. For demonstration's sake, you could reimplement dynamic unroll as follows:

tf.metrics aggregates data and tf.summary logs them

tf.metrics会收集数据，tf.summary会记录数据。与1.x不同，summary会直接发送给writer，而不经过context manager的转发。

summary_writer = tf.summary.create_file_writer('/tmp/summaries')
with summary_writer.as_default():
  tf.summary.scalar('loss', 0.1, step=42)

tf.mertrics会统计值，并在调用result()方法时返回，调用reset_states()方法进行清空。

def train(model, optimizer, dataset, log_freq=10):
	# 创建metrics
  avg_loss = tf.keras.metrics.Mean(name='loss', dtype=tf.float32)
  for images, labels in dataset:
    loss = train_step(model, optimizer, images, labels)

		# 记录当前损失值
    avg_loss.update_state(loss)

    if tf.equal(optimizer.iterations % log_freq, 0):
			# 循环10此记录一次summary，使用metrics的result()获取记录的数据
      tf.summary.scalar('loss', avg_loss.result(), step=optimizer.iterations)
			# 清空metrics
      avg_loss.reset_states()

def test(model, test_x, test_y, step_num):
  # training=False is only needed if there are layers with different
  # behavior during training versus inference (e.g. Dropout).
  loss = loss_fn(model(test_x, training=False), test_y)
  tf.summary.scalar('loss', loss, step=step_num)

# 创建对应的writer
train_summary_writer = tf.summary.create_file_writer('/tmp/summaries/train')
test_summary_writer = tf.summary.create_file_writer('/tmp/summaries/test')

with train_summary_writer.as_default():
  train(model, optimizer, dataset)

with test_summary_writer.as_default():
  test(model, test_x, test_y, optimizer.iterations)

生成可视化数据

tensorboard --logdir /tmp/summaries

Use tf.config.experimental_run_funtions_eagerly() when debugging

tf.function和tf.keras的调试

因为有Eager execution，大部分方法都可以一步步执行来进行调试。但部分方法，如tf.function和tf.keras使用Graph执行，如果需要调试，可以将tf.config.experimental_run_functions_eagerly(True)设置上，让这些方法也使用Eager execution。

tf.function

@tf.function
def f(x):
  if x > 0:
    import pdb
    pdb.set_trace()
    x = x + 1
  return x

tf.config.experimental_run_functions_eagerly(True)
f(tf.constant(1))

tf.keras

class CustomModel(tf.keras.models.Model):

  @tf.function
  def call(self, input_data):
    if tf.reduce_mean(input_data) > 0:
      return input_data
    else:
      import pdb
      pdb.set_trace()
      return input_data // 2

tf.config.experimental_run_functions_eagerly(True)
model = CustomModel()
model(tf.constant([-2, -4]))