docs/image_summaries.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "djUvWu41mtXa"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "su2RaORHpReL"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NztQK2uFpXT-"
      },
      "source": [
        "# Displaying image data in TensorBoard\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tensorboard/image_summaries\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorboard/blob/master/docs/image_summaries.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorboard/blob/master/docs/image_summaries.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorboard/docs/image_summaries.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eDXRFe_qp5C3"
      },
      "source": [
        "## Overview\n",
        "\n",
        "Using the **TensorFlow Image Summary API,** you can easily log tensors and arbitrary images and view them in TensorBoard. This can be extremely helpful to sample and examine your input data, or to [visualize layer weights](http://cs231n.github.io/understanding-cnn/) and [generated tensors](https://hub.packtpub.com/generative-adversarial-networks-using-keras/). You can also log diagnostic data as images that can be helpful in the course of your model development.\n",
        "\n",
        "In this tutorial, you will learn how to use the Image Summary API to visualize tensors as images. You will also learn how to take an arbitrary image, convert it to a tensor, and visualize it in TensorBoard. You will work through a simple but real example that uses Image Summaries to help you understand how your model is performing.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dG-nnZK9qW9z"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "3U5gdCw_nSG3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TensorFlow 2.x selected.\n"
          ]
        }
      ],
      "source": [
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "\n",
        "# Load the TensorBoard notebook extension.\n",
        "%load_ext tensorboard"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "1qIKtOBrqc9Y"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TensorFlow version:  2.2\n"
          ]
        }
      ],
      "source": [
        "from datetime import datetime\n",
        "import io\n",
        "import itertools\n",
        "from packaging import version\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import sklearn.metrics\n",
        "\n",
        "print(\"TensorFlow version: \", tf.__version__)\n",
        "assert version.parse(tf.__version__).release[0] >= 2, \\\n",
        "    \"This notebook requires TensorFlow 2.0 or above.\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tq0gyXOGZ3-h"
      },
      "source": [
        "# Download the Fashion-MNIST dataset\n",
        "\n",
        "You're going to construct a simple neural network to classify images in the the [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) dataset. This dataset consist of 70,000 28x28 grayscale images of fashion products from 10 categories, with 7,000 images per category.\n",
        "\n",
        "First, download the data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "VmEQwCon3i7m"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz\n",
            "32768/29515 [=================================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz\n",
            "26427392/26421880 [==============================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz\n",
            "8192/5148 [===============================================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz\n",
            "4423680/4422102 [==============================] - 0s 0us/step\n"
          ]
        }
      ],
      "source": [
        "# Download the data. The data is already divided into train and test.\n",
        "# The labels are integers representing classes.\n",
        "fashion_mnist = keras.datasets.fashion_mnist\n",
        "(train_images, train_labels), (test_images, test_labels) = \\\n",
        "    fashion_mnist.load_data()\n",
        "\n",
        "# Names of the integer classes, i.e., 0 -> T-short/top, 1 -> Trouser, etc.\n",
        "class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', \n",
        "    'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qNsjMY0364j4"
      },
      "source": [
        "## Visualizing a single image\n",
        "\n",
        "To understand how the Image Summary API works, you're now going to simply log the first training image in your training set in TensorBoard.\n",
        "\n",
        "Before you do that, examine the shape of your training data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "FxMPcdmvBn9t"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape:  (28, 28)\n",
            "Label:  9 -> Ankle boot\n"
          ]
        }
      ],
      "source": [
        "print(\"Shape: \", train_images[0].shape)\n",
        "print(\"Label: \", train_labels[0], \"->\", class_names[train_labels[0]])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4F8zbUKfBuUt"
      },
      "source": [
        "Notice that the shape of each image in the data set is a rank-2 tensor of shape (28, 28), representing the height and the width.\n",
        "\n",
        "However, ```tf.summary.image()``` expects a rank-4 tensor containing ```(batch_size, height, width, channels)```. Therefore, the tensors need to be reshaped. \n",
        "\n",
        "You're logging only one image, so ```batch_size``` is 1. The images are grayscale, so set ```channels``` to 1."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5yPh-7EWB8IK"
      },
      "outputs": [],
      "source": [
        "# Reshape the image for the Summary API.\n",
        "img = np.reshape(train_images[0], (-1, 28, 28, 1))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAdJDY3FCCwt"
      },
      "source": [
        "You're now ready to log this image and view it in TensorBoard."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IJNpyVyxbVtT"
      },
      "outputs": [],
      "source": [
        "# Clear out any prior log data.\n",
        "!rm -rf logs\n",
        "\n",
        "# Sets up a timestamped log directory.\n",
        "logdir = \"logs/train_data/\" + datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n",
        "# Creates a file writer for the log directory.\n",
        "file_writer = tf.summary.create_file_writer(logdir)\n",
        "\n",
        "# Using the file writer, log the reshaped image.\n",
        "with file_writer.as_default():\n",
        "  tf.summary.image(\"Training data\", img, step=0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rngALbRogXe6"
      },
      "source": [
        "Now, use TensorBoard to examine the image. Wait a few seconds for the UI to spin up."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T_X-wIy-lD9f"
      },
      "outputs": [],
      "source": [
        "%tensorboard --logdir logs/train_data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c8n8YqGlT3-c"
      },
      "source": [
        "<!-- <img class=\"tfo-display-only-on-site\" src=\"https://github.com/tensorflow/tensorboard/blob/master/docs/images/images_single.png?raw=1\"/> -->"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "34enxJjjgWi7"
      },
      "source": [
        "The \"Time Series\" dashboard displays the image you just logged. It's an \"ankle boot\". \n",
        "\n",
        "The image is scaled to a default size for easier viewing. If you want to view the unscaled original image, check \"Show actual image size\" at the bottom of the \"Settings\" panel on the right. \n",
        "\n",
        "Play with the brightness and contrast sliders to see how they affect the image pixels."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bjACE1lAsqUd"
      },
      "source": [
        "## Visualizing multiple images\n",
        "\n",
        "Logging one tensor is great, but what if you wanted to log multiple training examples?\n",
        "\n",
        "Simply specify the number of images you want to log when passing data to ```tf.summary.image()```."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iHUjCXbetIpb"
      },
      "outputs": [],
      "source": [
        "with file_writer.as_default():\n",
        "  # Don't forget to reshape.\n",
        "  images = np.reshape(train_images[0:25], (-1, 28, 28, 1))\n",
        "  tf.summary.image(\"25 training data examples\", images, max_outputs=25, step=0)\n",
        "\n",
        "%tensorboard --logdir logs/train_data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fr6LFQG9UD6z"
      },
      "source": [
        "<!-- <img class=\"tfo-display-only-on-site\" src=\"https://github.com/tensorflow/tensorboard/blob/master/docs/images/images_multiple.png?raw=1\"/> -->"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c-7sZs3XuBBy"
      },
      "source": [
        "## Logging arbitrary image data\n",
        "\n",
        "What if you want to visualize an image that's not a tensor, such as an image generated by [matplotlib](https://matplotlib.org/)?\n",
        "\n",
        "You need some boilerplate code to convert the plot to a tensor, but after that, you're good to go.\n",
        "\n",
        "In the code below, you'll log the first 25 images as a nice grid using matplotlib's ```subplot()``` function. You'll then view the grid in TensorBoard:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F5U_5WKt8bdQ"
      },
      "outputs": [],
      "source": [
        "# Clear out prior logging data.\n",
        "!rm -rf logs/plots\n",
        "\n",
        "logdir = \"logs/plots/\" + datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n",
        "file_writer = tf.summary.create_file_writer(logdir)\n",
        "\n",
        "def plot_to_image(figure):\n",
        "  \"\"\"Converts the matplotlib plot specified by 'figure' to a PNG image and\n",
        "  returns it. The supplied figure is closed and inaccessible after this call.\"\"\"\n",
        "  # Save the plot to a PNG in memory.\n",
        "  buf = io.BytesIO()\n",
        "  plt.savefig(buf, format='png')\n",
        "  # Closing the figure prevents it from being displayed directly inside\n",
        "  # the notebook.\n",
        "  plt.close(figure)\n",
        "  buf.seek(0)\n",
        "  # Convert PNG buffer to TF image\n",
        "  image = tf.image.decode_png(buf.getvalue(), channels=4)\n",
        "  # Add the batch dimension\n",
        "  image = tf.expand_dims(image, 0)\n",
        "  return image\n",
        "\n",
        "def image_grid():\n",
        "  \"\"\"Return a 5x5 grid of the MNIST images as a matplotlib figure.\"\"\"\n",
        "  # Create a figure to contain the plot.\n",
        "  figure = plt.figure(figsize=(10,10))\n",
        "  for i in range(25):\n",
        "    # Start next subplot.\n",
        "    plt.subplot(5, 5, i + 1, title=class_names[train_labels[i]])\n",
        "    plt.xticks([])\n",
        "    plt.yticks([])\n",
        "    plt.grid(False)\n",
        "    plt.imshow(train_images[i], cmap=plt.cm.binary)\n",
        "  \n",
        "  return figure\n",
        "\n",
        "# Prepare the plot\n",
        "figure = image_grid()\n",
        "# Convert to image and log\n",
        "with file_writer.as_default():\n",
        "  tf.summary.image(\"Training data\", plot_to_image(figure), step=0)\n",
        "\n",
        "%tensorboard --logdir logs/plots"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o_tIghRsXY7S"
      },
      "source": [
        "<!-- <img class=\"tfo-display-only-on-site\" src=\"https://github.com/tensorflow/tensorboard/blob/master/docs/images/images_arbitrary.png?raw=1\"/> -->"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vZx70BC1zhgW"
      },
      "source": [
        "## Building an image classifier\n",
        "\n",
        "Now put this all together with a real example. After all, you're here to do machine learning and not plot pretty pictures!\n",
        "\n",
        "You're going to use image summaries to understand how well your model is doing while training a simple classifier for the Fashion-MNIST dataset. \n",
        "\n",
        "First, create a very simple model and compile it, setting up the optimizer and loss function. The compile step also specifies that you want to log the accuracy of the classifier along the way."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R74hPWJHzgvZ"
      },
      "outputs": [],
      "source": [
        "model = keras.models.Sequential([\n",
        "    keras.layers.Flatten(input_shape=(28, 28)),\n",
        "    keras.layers.Dense(32, activation='relu'),\n",
        "    keras.layers.Dense(10, activation='softmax')\n",
        "])\n",
        "\n",
        "model.compile(\n",
        "    optimizer='adam', \n",
        "    loss='sparse_categorical_crossentropy',\n",
        "    metrics=['accuracy']\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SdT_PpZB1UMn"
      },
      "source": [
        "When training a classifier, it's useful to see the [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix). The confusion matrix gives you detailed knowledge of how your classifier is performing on test data.\n",
        "\n",
        "Define a function that calculates the confusion matrix. You'll use a convenient [Scikit-learn](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html) function to do this, and then plot it using matplotlib."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rBiXP8-UO8t6"
      },
      "outputs": [],
      "source": [
        "def plot_confusion_matrix(cm, class_names):\n",
        "  \"\"\"\n",
        "  Returns a matplotlib figure containing the plotted confusion matrix.\n",
        "\n",
        "  Args:\n",
        "    cm (array, shape = [n, n]): a confusion matrix of integer classes\n",
        "    class_names (array, shape = [n]): String names of the integer classes\n",
        "  \"\"\"\n",
        "  figure = plt.figure(figsize=(8, 8))\n",
        "  plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)\n",
        "  plt.title(\"Confusion matrix\")\n",
        "  plt.colorbar()\n",
        "  tick_marks = np.arange(len(class_names))\n",
        "  plt.xticks(tick_marks, class_names, rotation=45)\n",
        "  plt.yticks(tick_marks, class_names)\n",
        "\n",
        "  # Compute the labels from the normalized confusion matrix.\n",
        "  labels = np.around(cm.astype('float') / cm.sum(axis=1)[:, np.newaxis], decimals=2)\n",
        "\n",
        "  # Use white text if squares are dark; otherwise black.\n",
        "  threshold = cm.max() / 2.\n",
        "  for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
        "    color = \"white\" if cm[i, j] > threshold else \"black\"\n",
        "    plt.text(j, i, labels[i, j], horizontalalignment=\"center\", color=color)\n",
        "\n",
        "  plt.tight_layout()\n",
        "  plt.ylabel('True label')\n",
        "  plt.xlabel('Predicted label')\n",
        "  return figure"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6lOAl_v26QGq"
      },
      "source": [
        "You're now ready to train the classifier and regularly log the confusion matrix along the way.\n",
        "\n",
        "Here's what you'll do:\n",
        "\n",
        "1. Create the [Keras TensorBoard callback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard) to log basic metrics\n",
        "2. Create a [Keras LambdaCallback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LambdaCallback) to log the confusion matrix at the end of every epoch\n",
        "3. Train the model using Model.fit(), making sure to pass both callbacks\n",
        "\n",
        "As training progresses, scroll down to see TensorBoard start up."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "utd-vH6hn5RY"
      },
      "outputs": [],
      "source": [
        "# Clear out prior logging data.\n",
        "!rm -rf logs/image\n",
        "\n",
        "logdir = \"logs/image/\" + datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n",
        "# Define the basic TensorBoard callback.\n",
        "tensorboard_callback = keras.callbacks.TensorBoard(log_dir=logdir)\n",
        "file_writer_cm = tf.summary.create_file_writer(logdir + '/cm')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bXQ7-9CF0TPA"
      },
      "outputs": [],
      "source": [
        "def log_confusion_matrix(epoch, logs):\n",
        "  # Use the model to predict the values from the validation dataset.\n",
        "  test_pred_raw = model.predict(test_images)\n",
        "  test_pred = np.argmax(test_pred_raw, axis=1)\n",
        "\n",
        "  # Calculate the confusion matrix.\n",
        "  cm = sklearn.metrics.confusion_matrix(test_labels, test_pred)\n",
        "  # Log the confusion matrix as an image summary.\n",
        "  figure = plot_confusion_matrix(cm, class_names=class_names)\n",
        "  cm_image = plot_to_image(figure)\n",
        "\n",
        "  # Log the confusion matrix as an image summary.\n",
        "  with file_writer_cm.as_default():\n",
        "    tf.summary.image(\"epoch_confusion_matrix\", cm_image, step=epoch)\n",
        "\n",
        "# Define the per-epoch callback.\n",
        "cm_callback = keras.callbacks.LambdaCallback(on_epoch_end=log_confusion_matrix)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k6CV7dy-oJZu"
      },
      "outputs": [],
      "source": [
        "# Start TensorBoard.\n",
        "%tensorboard --logdir logs/image\n",
        "\n",
        "# Train the classifier.\n",
        "model.fit(\n",
        "    train_images,\n",
        "    train_labels,\n",
        "    epochs=5,\n",
        "    verbose=0, # Suppress chatty output\n",
        "    callbacks=[tensorboard_callback, cm_callback],\n",
        "    validation_data=(test_images, test_labels),\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o7PnxGf8Ur6F"
      },
      "source": [
        "<!-- <img class=\"tfo-display-only-on-site\" src=\"https://github.com/tensorflow/tensorboard/blob/master/docs/images/images_accuracy.png?raw=1\"/> -->\n",
        "\n",
        "<!-- <img class=\"tfo-display-only-on-site\" src=\"https://github.com/tensorflow/tensorboard/blob/master/docs/images/images_cm.png?raw=1\"/> -->"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6URWgszz9Jut"
      },
      "source": [
        "Notice that accuracy is climbing on both train and validation sets. That's a good sign. But how is the model performing on specific subsets of the data?\n",
        "\n",
        "Scroll down the \"Time Series\" dashboard to visualize your logged confusion matrices.\n",
        "Check \"Show actual image size\" at the bottom of the \"Settings\" panel to see the confusion matrix at full size.\n",
        "\n",
        "By default the dashboard shows the image summary for the last logged step or epoch. Use the slider to view earlier confusion matrices. Notice how the matrix changes significantly as training progresses, with darker squares coalescing along the diagonal, and the rest of the matrix tending toward 0 and white. This means that your classifier is improving as training progresses! Great work!\n",
        "\n",
        "The confusion matrix shows that this simple model has some problems. Despite the great progress, Shirts, T-Shirts, and Pullovers are getting confused with each other. The model needs more work.\n",
        "\n",
        "If you're interested, try to improve this model with a [convolutional network](https://medium.com/tensorflow/hello-deep-learning-fashion-mnist-with-keras-50fcff8cd74a) (CNN)."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "image_summaries.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}