Web session navigation behavior for bot detection -

Abstract

Web robots are everywhere in today's web technology. These bots range from robots associated with viruses known as malicious to spiders also known as search engine bots. The latter, attempt to crawl the Internet harvesting information from websites for different purposes whereas no one can claim control over how and when this rich information is going to be used. While artificial intelligence keeps improving, robots become very smart too. Bots are likely to increase in quality and quantity as the world-wide-web develops and evolves. This is becoming a real threat to today's businesses and social life. What we're more likely to see in the future are smarter bots which can do anything at any time. This naturally urges contemporary researchers and experts in cyber security to invest in every possible direction to try protect the web environment. Detecting bots, whether malicious or search engine bots is an important goal for most website admins. In this thesis, we propose a novel machine learning bot detection approach based on web session navigation behavior. While machine learning has been used before for bot detection, most existing approaches rely on general hypotheses based on statistical analysis over multiple websites and are thus easy to counter. In our work, we build a website-specific hypothesis or classifier based on the actual navigation data of the website. The advantages of our approach is that it can be generally used to detect any type of bot attacks and is difficult to counter unless website-specific bots are designed as well. Our classifier uses a Two-Class Boosted Decision Tree classification model and can be periodically re-trained to learn new hypotheses as bots evolve. We tested our approach on two real-world websites and achieved an accuracy of around 83 percent, outperforming state-of-the-art machine-learning bot detection approaches by almost 14 percent. In summary, we are after a solution where each website can learn, generate and tune its own defensive mechanism and can co-exist with other defensive