To implement our idea, we devise a general encoder-decoder network amenable to several restoration tasks. In this way, the system itself can discover which sub-tasks deserve more or less attention. Our approach relies on a feedback mechanism that, at each epoch of training, lets the system guide its own learning towards the right proportion of sub-tasks per difficulty level. During training, we aim to jointly train the deep neural network restoration model to accommodate all N sub-tasks. We divide each restoration task into N sub-tasks of increasing difficulty. This includes i) a solution that simply pools training instances from across difficulty levels, ii) a solution that focuses on hard examples, and iii) a curriculum learning solution that intelligently orders the training samples from easy to hard.īased on our findings, we introduce a new on-demand learning solution for training all-rounder deep networks for image restoration tasks. To solve the fixation problem, in this work we explore ways to let a deep learning system take control and guide its own training. For the pixel interpolation, deblurring and denoising results, we attempt analogous trials, i.e., training for 80% missing pixels, a single width blur kernel or a single level of noise, respectively, then observe poor performance by the fixated models on examples having different corruption levels. However, if we remove a block that is slightly shifted away from the central region, or remove a much smaller block, the model fails to inpaint satisfactorily. During testing, the resulting model can inpaint the central block of the same size at the same location very well. Specifically, for the image inpainting task, we train a model to inpaint a large central missing block of size 32 x 32. To get these results, we followed the current literature to train deep networks to target a certain degree of corruption for four applications. Just how bad is the fixatation problem in image restoration tasks? The above figure helps illustrate. As a result, existing methods risk training fixated models: models that perform well only at a particular level of difficulty. Unfortunately, these are strong assumptions that are difficult if not impossible to meet in practice.

The implicit assumption is that at test time, either i) corruption will be limited to that same difficulty, or ii) some other process will estimate the difficulty level before passing the image to the appropriate, separately trained restoration system. In particular, existing systems self-generate training instances with a manually fixed hyper-parameter that controls the degree of corruption-a fixed inpainting size, a fixed percentage of corrupted pixels, or a fixed level of Gaussian white noise, etc. However, the norm in existing deep learning methods is to train a model that succeeds at restoring images exhibiting a particular level of corruption difficulty.

Intuitively, the more missing pixels or the more severe the blur, the more difficult the restoration task. For example, as shown in the figure above, an inpainter may face images with varying sizes of missing content, and a deblurring system may encounter varying levels of blurriness. Image corruption exists in various degrees of severity, and so in real-world applications the difficulty of restoring images will also vary significantly. On four restoration tasks-image inpainting, pixel interpolation, image deblurring, and image denoising-and three diverse datasets, our approach consistently outperforms both the status quo training procedure and curriculum learning alternatives. The main idea is to exploit a feedback mechanism to self-generate training instances where they are needed most, thereby learning models that can generalize across difficulty levels. Then, we propose an on-demand learning algorithm for training image restoration models with deep convolutional neural networks. First, we examine the weakness of conventional 'fixated' models and demonstrate that training general models to handle arbitrary levels of corruption is indeed non-trivial. While machine learning approaches to image restoration offer great promise, current methods risk training models fixated on performing well only for image corruption of a particular level of difficulty-such as a certain level of noise or blur. On-Demand Learning for Deep Image Restoration Ruohan Gao Kristen Grauman University of Texas at Austin In ICCV 2017 Abstract

Inpaint photo restoration