Section 2 Exercise: Softmax clone doesn't work the same as tensorflow's softmax

[Raunak-Singh-Inventor]

I tried the softmax clone given in the course repo, and it doesn't give the same results as tf.keras.activations.softmax. The softmax clone's prediction probabilities also don't sum to 1. Here is a code example to recreate the issue:

import tensorflow as tf
tf.__version__

# Creating sample tensor 
tensor = tf.constant([[1, 2, 3, 6],
                     [2, 4, 5, 6],
                     [3, 8, 7, 6]])
print(f"input to softmax activation:{inputs[0, :]}")
print("--------------------------------------------------------------------------------------")
outputs = tf.keras.activations.softmax(inputs) # apply the softmax function on the input tensor
print(f"output of softmax activation:{outputs[0, :]}")
print("--------------------------------------------------------------------------------------")
print(f"sum: {tf.reduce_sum(outputs[0, :])}") 

# Replicate the functionality
def softmax_clone(x):
  e_x = tf.math.exp(x - tf.math.reduce_max(x))
  return e_x / tf.math.reduce_sum(e_x , axis = 0)

clone_outputs = softmax_clone(inputs)
print(f"output of softmax_clone activation:{clone_outputs[0, :]}")
print("--------------------------------------------------------------------------------------")
print(f"sum: {tf.reduce_sum(clone_outputs[0, :])}") 

[mrdbourke]

Hey @Raunak-Singh-Inventor,

I updated your code to reflect the correct solution based on here: https://www.tensorflow.org/api_docs/python/tf/nn/softmax

import tensorflow as tf
tf.__version__

# Creating sample tensor 
tensor = tf.constant([[1, 2, 3, 6],
                      [2, 4, 5, 6],
                      [3, 8, 7, 6]], dtype=tf.float32)
print(f"input to softmax activation:{tensor[0, :]}")
print("--------------------------------------------------------------------------------------")
outputs = tf.keras.activations.softmax(tensor, axis=-1) # apply the softmax function on the input tensor, axis=-1 is default
print(f"output of softmax activation:{outputs[0, :]}")
print("--------------------------------------------------------------------------------------")
print(f"sum: {tf.reduce_sum(outputs[0, :])}") 

# Replicate the functionality
def softmax_clone(x, axis=-1):
  return tf.exp(x) / tf.reduce_sum(tf.exp(x), axis=axis, keepdims=True) # <- updated this line

print(f"input to softmax activation:{tensor[0, :]}")
clone_outputs = softmax_clone(tensor)
print(f"output of softmax_clone activation:{clone_outputs[0, :]}")
print("--------------------------------------------------------------------------------------")
print(f"sum: {tf.reduce_sum(clone_outputs[0, :])}") 

Do you have a link to where the code you posted came from? I can go back and fix it.

[Raunak-Singh-Inventor]

@mrdbourke I know its late, but I got the code from here: https://github.com/mrdbourke/tensorflow-deep-learning/blob/main/extras/solutions/%F0%9F%9B%A0_02_neural_network_classification_in_tensorflow.ipynb

Also, thanks for the helpful solution. Great Course 👍 😃

[mrdbourke]

thank you :) glad you're enjoying it!