Visualizing neural nets trained in Cortex
Doing data science work in Clojure is pretty effective as the tooling is excellent in all parts of the data pipeline such as
- data wrangling (using Incanter / core.matrix / sql or nosql )
- building machine learning models (using clj-ml)
- visualization (Gorilla-repl)
If one needed to train deep neural networks, the obvious choices are tool-kits in other languages such as Tensorflow/Keras/Theano in Python.
However in the past year, Cortex, a neural network library in Clojure has gained traction. It now supports feed-forward networks, CNNs as well as other important features such as multiple optimizers (e.g. Adam optimiser) and weight initialization schemes such as Xavier.
I found that the Tensorboard tool (part of the Tensorflow project) is extremely useful for observing how neural networks train (or don’t train, which is often the case). While training networks with Cortex, I missed the ability to graph various metrics to track progress of training. I discovered that Tensorboard is quite a flexible tool which could be leveraged to visualize metrics generated by Cortex. What follows is a short note on getting Tensorboard and Cortex to work together.
First off, install Tensorflow using the instructions provided. virtualenv is usually the quickest method.
Cortex has multiple projects/interfaces, and the experiment interface (which is geared to quick prototyping) has APIs to plug in custom listeners. We can use the tensorboard listener.
Tensorboard creates its visualizations by reading files that are have the string *tfevents* in the file name. These files contain event data for different kinds of events such as
- scalar valued metrics (e.g. accuracy, train loss, test loss)
- vector valued metrics (e.g. gradients, weights)
- raw data such as images, text, audio.
The tensorboard listener in Cortex has out of the box support for train/test loss and weights/gradients for the layers. Let’s take a look an example of how to instrument Cortex code for emitting tfevents.
The following code snippet creates an event stream and write a single scalar value to the event stream. Of course, multiple events can be written to the event stream.
To view the file on Tensorboard, start Tensorboard and point to the same root directory
In a different post, we shall learn how to send other events such as gradients and buffers to be viewed on Tensorboard.