Error in tutorial 3

[jcpeterson]

It's the saved output. The tail of the error is:
NameError: name 'attack_examples' is not defined

[tianweiy]

Hi, do you get this error locally? I pull the master branch and didn't encounter the error. I think Matt may have some change to the evaluation file that causes the error message. If you don't get any error, we will find out the cause and update the notebook tomorrow.

[jcpeterson]

All cells ran for me except for the last: callable(xentropy_eval) since xentropy_eval is not defined.

I'm not clear on what the reported loss is. The first part of the notebook says "Average successful loss value". What is meant by "successful" here? Shouldn't it just be loss for the whole batch?

[revbucket]

Ah, sorry, that cell should've been deleted in my last commit to that file: I originally was going to use CrossEntropy as the evaluation metric I'd monkeypatch in the "Custom Evaluation Techniques" section of the tutorial. That cell should've been deleted and, upon the most recent commit, is.

As far as average successful loss, I've also changed the wording to be more clear. In general, the definition we use is:
An adversarial attack $Atk$ is successful on image $x \in X$ against classifier $f_\theta$ if the index of the highest-valued logit of $f_\theta(Atk(x))$ is not the same as the index of the highest-valued logit of $f_\theta(x)$. i.e., the attack induces a change in top-1 label.

Evaluation of attacks and defenses is nuanced, in that there's three classes of things that matter:

1. How accurate is our classifier on clean examples?
2. Against a given classifier, what percentage of clean examples is our attack successful on?
3. What is the quality of the generated attacks which are successful? This corresponds to the 'average successful loss'

If the attack is not successful against a classifier, I argue that we don't particularly care what the quality of the generated (unsuccessful) adversarial example is, so by default we only support loss functions against the successful images. If you'd like to care about batch-loss, you can either monkey patch it in, submit a pull request with that added in, or ask me and I'll get around to it in a bit (opening a separate issue is probably best so I don't forget about it).