Can I Have Hooks / Callbacks Outside A Session Or Between The Creation Of One?

With train_and_evaluate() it is possible to execute a schedule which trains and evaluates a model according to the specifications I am passing down. There are hooks I can register

Solution 1:

I guess you could implement the begin method of your own SessionHook subclass.

For the sake of the example I used the iris code (see this doc).

import tensorflow as tf

def the_callback_i_want():
    # You need to work in a new graph so let's create a new one
    g = tf.Graph()
    with g.as_default():
        x = tf.get_variable("x", ())
        x = tf.assign_add(x, 1)
        init = tf.global_variables_initializer()
        with tf.Session() as sess:   
            sess.run(init)
            print("I'm here !", sess.run(x))


class MyHook(tf.train.SessionRunHook):

  def begin(self):
    """Called once before using the session.

    When called, the default graph is the one that will be launched in the
    session.  The hook can modify the graph by adding new operations to it.
    After the `begin()` call the graph will be finalized and the other callbacks
    can not modify the graph anymore. Second call of `begin()` on the same
    graph, should not change the graph.
    """
    the_callback_i_want()


import iris_data
# Fetch the data
(train_x, train_y), (test_x, test_y) = iris_data.load_data()

# Feature columns describe how to use the input.
my_feature_columns = []
for key in train_x.keys():
    my_feature_columns.append(tf.feature_column.numeric_column(key=key))

# Build 2 hidden layer DNN with 10, 10 units respectively.
classifier = tf.estimator.DNNClassifier(
    feature_columns=my_feature_columns, hidden_units=[10, 10],  n_classes=3)

# Fetch the data
(train_x, train_y), (test_x, test_y) = iris_data.load_data()

# Feature columns describe how to use the input.
my_feature_columns = []
for key in train_x.keys():
    my_feature_columns.append(tf.feature_column.numeric_column(key=key))

train_spec = tf.estimator.TrainSpec(input_fn=lambda:iris_data.train_input_fn(train_x, train_y,
                                                 10), max_steps=100)
eval_spec = tf.estimator.EvalSpec(input_fn=lambda:iris_data.eval_input_fn(test_x, test_y,
                                                10), hooks=[MyHook()])
tf.estimator.train_and_evaluate(classifier, train_spec, eval_spec)

And it prints:

INFO:tensorflow:Saving checkpoints for 100 into /var/folders/***/model.ckpt.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2018-10-18-20:19:28
I'm here ! 1.9386581
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /var/folders/***/model.ckpt-100
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2018-10-18-20:19:28