Humanitarian visual and semantic computing : crisis-related image tweets classification -

Abstract

When the public and first responders are flooding the internet with often annotated images and texts during natural disasters, rescue teams are overwhelmed to prioritize often scarce resources. Given that most of the efforts in such humanitarian situations rely heavily on human labor and input, we propose in this research a novel approach that leverages social media and uses machine learning and computer vision to help automate humanitarian intervention. After all, “an ANNOTATED image is worth a thousand words”. Our framework relies analyzing visually and semantically twitter data. We merge low-level visual features that extract color, shape and texture with semantic attributes extracted from annotated pictures posted on Twitter in disaster times. These visual and textual features are trained and tested on a home-grown dataset solely gathered from Twitter. The best accuracy obtained after crowdsource labeling the data was when low-level visual features and semantic attributes were applied to the different classification scenarios of support vector machines (SVM), Neural Networks and Ensemble Learning-SVM with 5-Fold cross-validation. Since the data is organically unsupervised, we proposed a structural neuronal modification to the cortical algorithms, a deep learning algorithm inspired by the human visual cortex and tested it using the previously proposed visual and semantic features. As expected, the proposed CA showed better performance than regular CA, which motivates follow on research.