Towards a computationally efficient model of working memory : a case study on vocabulary acquisition.

Abstract

Seamlessly integrating today's advanced technologies into our life requires intelligent systems capable of handling various types of inputs such as images and audio; which calls for multi-modal learning. While creating a unique delineation of a concept based on different modalities such as visual and phonological representation is a seemly effortless task for humans, deep learning models often struggle to successfully implement a unified structure capable of handling multiple modalities due to their specialized structures that process a specific type of data. Challenges in multi-modal learning tasks include forming a homogeneous representation of the different modalities, translating signals from one modality to another, identifying overlapping information across inputs, fusing these inputs into one form, and employing the different inputs in learning a unique model. This dissertation presents a biologically inspired multi-modal deep learning stochastic based computational model of working memory adhering to Baddeley's mutli-component model. The model is composed of three loops playing the role of phonological short term processing store, visual short term analysis store and central executive. The loops employed are inspired by their biological counterpart at the structural and operational levels. Three training algorithms to learn the network's connections are proposed: an iterative Bayesian solution, a stochastic spike timing dependent reinforcement learning strategy and a two stage algorithm with an unsupervised phase and a reinforcement learning phase. The proposed network and algorithms are tested on publicly available datasets. The phonological loop is tested on a collection of English word recordings with the proposed network achieving an average recognition rate of 92.45percent. The visual loop tested on the Amsterdam Library of Object Images achieved an average accuracy of 95.3percent. The central executive is tested on the 1-2-AX task with a 97.5percent accuracy reported in our results. The multi-modal network attain