We are seeing that there is an Enterprise-wide adoption of transfer learning techniques while using Machine learning and Deep learning. And here is the reason why…
Traditionally, all the Machine learning algorithms assume learning to happen from scratch for every new learning problem. The assumption is that no previous learning will be leveraged. Transfer learning is distinct from other approaches in learning cycle time and improving performance of the models. In cases where the domains for the learning problems relate, there will be some learnings from the past that can be acquired and used. Some common examples include:
The knowledge of French could help students learn Spanish
The knowledge in mathematics could help students learn Physics
The knowledge of driving a car could help drivers learn to drive the truck
The knowledge of recognizing cats could help computers to recognize tigers
In the Machine learning context, this refers to identifying and applying the knowledge accumulated from previous tasks to new tasks from a related domain. The key here is the ability to identify the commonality between the domains. Reinforcement learning and classification and regression problems apply transfer learning. The transfer learning process flow is as shown here:
In Transfer learning, end users are offered the option of training a selected model by using historical data. The results from the algorithm are compared against real-time results. Once the outcomes show a fair amount of accuracy with acceptable error margins, the end user can use the same algorithm for analyzing real-time data. This capability works best between preproduction and production environments or between the edge and the data center environments. Evolution of more complex use cases will require maturity and advanced skills on both the vendor and end-user side
Some advantages of Transfer learning when applied as a part of insights generation are:
As there inputs from the source model giving a head way to the learner there is a potential for gain in development time.
When the so called ignorant learner (a learner who learns in silos) is faced with knowledge from a source model, the baseline performance might improve due to potential knowledge transfer resulting in overall improved baseline performance. Thus resulting in overall improved final performance of the model.
I fing the below representation extremely valuable to understand the benefits of transfer learning, very clearly training data performance is alleviated with when transfer learning is applied.
Source: Transfer Learning, Lisa Torrey and Jude Shavlik
Transfer learning types
Based on the definition of transfer learning, we summarize the relationship between traditional machine learning and various transfer learning settings, where we categorize transfer learning under three sub-settings, inductive transfer learning, transductive transfer learning and unsupervised transfer learning, based on different situations between the source and target domains and tasks. The mind map below shows the different types of transfer learning the scope and base definition for each type.
Availability of labelled data varies for each type of transfer learning. the table below shows the details of the labelled data and tasks.
Training Models is a complex process
One of the biggest advantages of Transfer learning is that it helps overcome the need for massive amounts of training data required to train models as it facilitates applying existing knowledge. Enterprises are now developing multi-purpose machine learning models that help optimize the model training time. [embed]https://www.datadriveninvestor.com/2019/02/08/machine-learning-in-finance/[/embed]
For machine learning models to be have higher accuracy of prediction, there should be enough training data available or even in some cases annotating training data is time-consuming and can turn out to be heavily manual task. Transfer learning can still complete the task using available data qualities or even decrease resources required to annotate data.
Transfer learning in Neural networks
We see many applications of Transfer learning used within the neural network for natural language processing and image recognition. In recent times application of transfer learning with Robotic Process Automation (RPA) workflows.
Neural Networks are known to be effective with featrizing the problem at a fine grained level and hence fuzzy problems like vision, machine translation, speech to text, and modern NLP.
The use case for training neural networks with transfer learning is extensively used in intelligent document processing using unstructured text. Natural language Processing and text analytics platforms involve defining vocabularies, taxonomies, parts of speech, and exhaustive attribute lists to classify and extract information from text. With Neutral Networks, there is a specific advantage whne used in specific business domains like Financial services and healthcare, specific and custom data dictionary that is built can extract required data attributes from the financial or patient record documents.
Usecase: Invoice data from unstructured images (unlabeled)
This is a domain specific usecase where multiple invoices from vendors are processed and the extracted data is ingested into an Accounting system. The invoices recieved are of different formats and in some cases might contain hand-written content. A few invoice samples are given below:
Sample 1:
Sample 2
Some important domain specific data tags that are required to be extracted are “Quarter to date”, “Year to date”, “Total Amount”, “Invoice Date” etc…
Transfer learning is employed in this case to label the unstructured data extracted from the Invoice. The domain specific bag of words is key and reusable. Both Vision & NLP processing steps use transfer learning process here.
Conclusion
These different transfer learning methodologies are critical to reducing the amount of example data organizations need to train deep neural networks. When paired with deep learning compute, neural networks are suitable for most enterprise process automation tasks including complicated text analytics use cases. At present, transfer learning may be the most productive means of addressing the training data issues inhibiting machine learning adoption.
Important References
TensorFlow Tutorial: Transfer learning with a pretrained ConvNet
R Code: Using Pre-Trained Models
IMP: This blog is an excerpt from one of my publications on Machine learning updated to include current advancements in Transfer learning. Feedback and comments from the readers is welcome
[embed]https://upscri.be/b2a0d6/[/embed]