Best Practices for Applying Deep Learning to Novel Applications

2/27/17 Best Practices for Applying Deep Learning to Novel Applications Leslie N. Smith Navy Center for Applied Research in Artificial Intelligence U.S. Naval Research Laboratory, Code 5514 Washington, DC 20375 leslie.smith@nrl.navy.mil

ABSTRACT This report is targeted to groups who are subject matter experts in their application but deep learning novices. It contains practical advice for those interested in testing the use of deep neural networks on applications that are novel for deep learning. We suggest making your project more manageable by dividing it into phases. For each phase this report contains numerous recommendations and insights to assist novice practitioners.
Introduction Although my focus is on deep learning (DL) research, I am finding that more and more frequently I am being asked to help groups without much DL experience who want to try deep learning on their novel (for DL) application. The motivation for this NRL report derives from noticing that much of my advice and guidance is similar for all such groups. Hence, this report discusses the aspects of applying DL that are more universally relevant. While there are several useful sources of advice on best practices for machine learning [1-5], there are differences relevant to DL that this report addresses. Still, I recommend the reader read and become familiar with these references as they contain numerous gems. In addition, there are many sources on best practices on the topic of software engineering and agile methodologies that I assume the reader is already familiar with (e.g., [6, 7]). The closest reference to the material in this report can be found in Chapter 11 of “Deep Learning” [8] on “Practical Methodology” but here I discuss a number of factors and insights not covered in this textbook. You can see below that a deep learning application project is divided into phases. However, in practice you are likely to find it helpful to return to an earlier phase. For example, while finding an analogy in phase 3, you might discover new metrics that you hadn’t considered in phase 1. All of these best practices implicitly include iteratively returning to a phase and continuous improvement as the project proceeds. Phase 1: Getting prepared In this report I assume you are (or have access to) a subject matter expert for your application. You should be familiar with the literature and research for solving the associated problem and know the state-of-the-art solutions and performance levels. I recommend you consider here at the beginning if a deep learning solution is a worthwhile effort. You must consider the performance level of the state-of- the-art and if it is high, whether it is worthwhile to put in the efforts outlined in this report for an incremental improvement. Don’t jump into deep learning only because it seems like the latest and

2 Version 1.0

2/27/17 greatest methodology. You should also consider if you have the computer resources since each job to train a deep network will likely take days or weeks. I have made ample use of DoD’s HPC systems in my own research. In addition, you should consider if machine learning is appropriate at all – remember training a deep network requires lots of labeled data, as described in phase 2. The first step is quantitatively defining what success looks like. What will you see if this is successful, whether it is done by human or machine? This helps define your evaluation metrics. Which metrics are important? Which are less important? You need to specify all quantitative values that play a role in the success of this project and determine how to weigh each of them. You also need to define objectives for your metrics; is your goal surpass human level performance? Your objectives will strongly influence the course of the project. Knowing quantitatively what human performance is on this task should guide your objectives; how does the state-of-the-art compare to human performance? Also, knowing how a human solves this task will provide valuable information on how the machine might solve it. Some of these metrics can also lead to the design of the loss function, which is instrumental in guiding the training of the networks. Don’t feel obligated to only use softmax/cross entropy/log loss just because that is the most common loss function, although you should probably start with it. Your evaluation metrics are by definition the quantities that are important for your application. Be willing to test these metrics as weighted components of the loss function to guide the training (see phase 6). Although you are likely considering deep learning because of its power, consider how to make the network’s “job” as easy as possible. This is anti-intuitive because it is the power of deep networks that likely motivates you to try it out. However, the easier the job that the networks must perform, the eas

…(Full text truncated)…