Until now, we published two papers as Conditional Mutual Information-Based Generalization Bound for Meta Learning.
The other paper titled as "A Unified View on PAC-Bayes Bounds for Meta-Learning".
Two other papers about 1.Meta excess risk and 2. f-CMI for meta learning may be prepared for submission in the future.
To increase our theoretical understanding of DNNs, we have studied meta learning. Meta-learning formalizes the goal of why DNNs have good performance to problems such as transfer learning. In fact, the learning process is done based on the set of assumptions known as inductive bias. In many machine learning problems (including DNNs), finding methods for automatically learning the inductive bias is desirable. Meta learning formalizes this goal by observing data from a number of inherently related tasks. Then, it uses the gained experience and knowledge to learn appropriate bias which can be fine-tuned to perform well on new tasks. Thus, the meta learner speeds up the learning of a new, previously unseen task.
For example, in DNNs, learning the initial weight and the learning rate of the training algorithm is in the scope of meta learning. As mentioned, the goal is extracting knowledge from several observed tasks referred to as meta-training set, and using the knowledge to improve performance on a novel task. The meta-learner generalizes well if after observing sufficiently training tasks, it infers a hyperparameter which contains good solutions to novel tasks. The good solution means that meta-generalization loss, which is defined as the average loss incurred by the hyperparameter when used on a new task, is minimized.
However, since both data and task distributions are unknown, the meta-generalization loss cannot be optimized. Instead, the meta-learner evaluates the empirical meta-training loss for the hyperparameter based on the meta-training set. Meta-generalization gap is defined as the difference between the meta-generalization loss and the meta-training loss. If the meta-generalization gap is small, it means that the meta-training loss is a good estimation of the meta-generalization loss.
Thus, bounding the meta-generalization gap is a key technique to understanding how the prior knowledge acquired from previous tasks may improve the performance of learning an unseen task.
