Unable to reproduce the result

[singhharmeet]

While using the imageNetFeatures script, I ran into following errors:-

1. the mean_file available in data folder is throwing the error during classifier stage
   [ValueError: number of channels incompatible with mean]
   We rectified it using the mean file provided through caffe for imagenet (Not sure if it's correct way to handle it)
2. We also removed axis from squeeze function while adding fc8 and fc7 features into mat.

After making the script work we tried running compute_openmax.py with fooling_images data provided on the main page. Surprisingly probabilty for fooling image was around 90% .
Please suggest some solution to reproduce the result specified in the thesis paper.

Error statements-

1. Error while using axis in squeeze:

File "imageNet_Features.py", line 302, in <module> main(sys.argv) File "imageNet_Features.py", line 299, in main extractFeatures(args) File "imageNet_Features.py", line 126, in extractFeatures compute_features(imgname,args) File "imageNet_Features.py", line 176, in compute_features feature_dict['fc7'] = sp.asarray(classifier.blobs['fc7'].data.squeeze(axis=(2,3))) ValueError: 'axis' entry 2 is out of bounds [-2, 2)

2. Error while using given mean file:

Traceback (most recent call last): File "imageNet_Features.py", line 302, in <module> main(sys.argv) File "imageNet_Features.py", line 299, in main extractFeatures(args) File "imageNet_Features.py", line 126, in extractFeatures compute_features(imgname,args) File "imageNet_Features.py", line 151, in compute_features input_scale=args.input_scale, channel_swap=channel_swap) File "/home/ubuntu/deep-learning/caffe/python/caffe/classifier.py", line 37, in __init__ self.transformer.set_mean(in_, mean) File "/home/ubuntu/deep-learning/caffe/python/caffe/io.py", line 250, in set_mean raise ValueError('Mean channels incompatible with input.') ValueError: Mean channels incompatible with input.

[sadimanna]

Did you succeed in reproducing the results I am also facing the same problems. However, I used VGG-13 net instead of the one used in the paper.

[sansinghsanjay]

Facing same issue. I wrote a CNN in tensorflow and trained it on MNIST. In test phase, Softmax and OpenMax giving same prediction, always [0 to 9] for images from the MNIST data and also for random images (outside from MNIST). Please let me know if I am missing something or if there is an alternate solution for this.

[sadimanna]

@sansinghsanjay

0 is the output label for the unknown images, after openmax the output labels of the known images will be [1-10].

If that is not the issue, then I guess the issue is that in Algorithm 2, the author has said
w_s(i) = 1 - (alpha-i)/alpha * exp((||x-loc||/scale)^shape)

So, when you call the w_score function, you get the output of the
Weibull CDF [1 -exp((||x-loc||/scale)^shape) ].

To get the exponential part, you need to use (1- w_score(distance)), and then multiply it with the alpha weights, and then subtract it from 1 to get the final output. Then proceed as usual.

Also, don't forget to use thresholding.

loc is the location parameter, scale is the scale parameter, shape is the shape parameter

[agaz1985]

@sadimanna I agree with you, .w_score returns the Weibull CDF, so modified_fc8_score should be something like:
modified_fc8_score = channel_scores[categoryid] * ( 1 - (1 - wscore)*ranked_alpha[categoryid] )

@agaz1985 Actually I don't get the conception. Scores, I think, should be multiplied by 1-alpha_rankCDF rather than alpha_rankCDF(which is the probability of being outlier) to modify scores. It is because the newly added label should represent the addition of all probabilities of being outlier. As for simple example, if the probability of being outlier is zero(which means that this class should not be rejected),however, zero is multiplied by scores so that its score becomes zero. Thus, I think line in paper is mistake not line in code. Would you help me about this?

[yasserboutaleb]

Dear @agaz1985 ,

Please did you know why they loop over the "categoryid" to compute modified_fc8_score = channel_scores[categoryid] * ( 1 - wscore*ranked_alpha[categoryid] ) ?
Yet, the "channel_scores" is the feature vector of the "fc8" layer for a specific channel, and the Len(channel_scores) is not equal to NCHANNELS.

Thank you.

[liuxz-cs]

@sadimanna I agree with you, .w_score returns the Weibull CDF, so modified_fc8_score should be something like: modified_fc8_score = channel_scores[categoryid] * ( 1 - (1 - wscore)*ranked_alpha[categoryid] )

@agaz1985 So do you solve this problem and have the correct answer or results ?

I think that the Weibull CDF w_score represents the probability of being outlier, and I wonder why the algorithm2 multiple the ranked_alpha when calculating w_s(i)(x). Why not directly use 1 - w_score to represents the probability of being inlier and modify the score? Actually, 1 - (1 - wscore)*ranked_alpha[categoryid] is not the value of the Weibull CDF?

[agaz1985]

I'm a bit out of context since it was long time ago, and I'm no longer working on this, so I cannot really answer without going back to where I left 7y ago.

[liuxz-cs]

I'm a bit out of context since it was long time ago, and I'm no longer working on this, so I cannot really answer without going back to where I left 7y ago.

Okay, thanks a lot.