What is a Robust Neural Network?
When I started my data science journey, I read all about “robust neural networks”. It was everywhere. But no one ever explained what a “robust” neural net was! I’m here to tell you what it is and how to know you have one.
First, let’s discuss what robust is. Something that is robust or has robustness will be strong, healthy, and able to withstand or overcome adverse conditions or rigorous testing. A spider web is a great example of something seen in nature that is very strong and able to withstand unfavorable situations, like rain or very gusty winds. If it is the word robust that is throwing you off, feel free to replace it with any of its synonyms: durable, tough, sturdy or stable.
Now that you know a little bit about what robust means, it’s time to decide if your algorithm meets the criteria. When trying to figure out if a machine learning algorithm is robust or not, ask yourself:
- Is the training error consistent with the testing error?
- Is the performance stable after adding some noise to the dataset?
Consistent Training and Testing Errors
First, let’s review what training and testing errors are.
training error: the calculated result of running a trained model on training data. Since the model was trained using this same data, the results may or may not be accurate to other data.
testing error: the calculated result of running a trained model on unseen testing data. The model has not seen this data before and is assumed to be similar to real-world data.
The error itself is dependent upon what you are measuring. Errors can be accuracy, recall, AUC, R-squared, or any other measure that best suits your data.
If a training error — in this case, let’s say we are using accuracy — returns a high rate, but our testing error returns a much lower rate, we can safely assume our model has learned the training data, but cannot apply what it has learned to the unseen testing data.
If you create or tune a model that generates similar training and testing errors, then celebrate! But that doesn’t mean your model is robust quite yet. It is important that the model can get similar results when run multiple times.
Adding in K-Fold cross validation will give you the confidence that your model’s results can hold up over many iterations. With K-Fold cross validation, the data is shuffled randomly and split into K groups or folds (K=5 and K=10 are the most commonly used values). Then, each group is further split into training and testing sets. From here, the models are fit with the training data as usual and evaluated with the testing set. The errors are returned in an array and can be averaged to see the overall error.
Having a model that can produce consistent training and testing errors over multiple runs means the model is not overfitting the data and can accurately predict new data that is thrown at it. That’s proof of a robust machine learning model!
Stable Performance with Noise Added
Just like the last criteria, there are some terms we need to understand here before we move on.
noise: unwanted data items, features, or records that do not help explain the relationship between the feature and the target.
When data is collected, the result tends to be messy. Any inaccuracies or errors can be referred to as “noise.” Noise can be problematic in machine learning because the noise will trick the algorithms into thinking an incorrect pattern exists or block the algorithm from detecting the correct patterns.
It may sound like noise is a bad thing, so you may be wondering why would we purposely add it in?
A model that is trained on clean data may find its performance drops when it is exposed to data that contains noise. Many real-world scenarios and data may have some aspects of noise.
Adding noise into your data can help the algorithm generalize better and will build a model that is resistant to unforeseen errors. Having a strong model that can overcome adverse reactions is a robust one!
If you have a model that has consistent training and testing errors and can tolerate noise, you can confidently say you have a robust algorithm!
Thanks for reading!
Say hello on LinkedIn @ linkedin.com/in/cassie-nutter/
