Data collection from the web and social media sites

Abstract

As social media is becoming the trend now where more than half the world's population is on social networking sites, and given the wealth of information that can be found on these sites as well as the web, a crucial need has emerged for businesses to have insights into these data sources. To answer this need, we propose an automated information gathering web application that accepts a query and gathers the relevant information from Twitter tweets, YouTube comments, Facebook wall posts, and Google search engine returned web pages'. The gathered data is then displayed and stored in a structured format in a database. One edge of this data collection project is that it is tailored to the Arabic Language where both the query and the returned data are in Arabic. However, the system works for English queries as well. The other edge in the project is the use of Vaadin, a java framework for building modern web applications that look great and perform well. The main benefit of this application is that it constitutes an integral module for any project that requires data gathering from the web and social media sites. To name a few, such projects include product and market analysis and opinion mining.