Using Machine Learning To Detect Anomalies

I’m going to start blogging more about detection of protocol/app anomalies, detection of lateral movement and/or data exfiltration, and more. For many years I have been watching users and applications furrow their way across networks and I’m gonna start data-dumping that info here :)

But…first…I manage a web server for a friend. It occurred to me that machine-learning could be useful in alerting when an attack is under way. I took the following steps

1) Get as much data as possible for this device. For Apache, this just meant gathering all the log files.

2) Parse the data and, for each session, look at the path taken as the user or bot perused the server (Note: outside of my initial scope, but timestamps are useful here to weed out a user versus a machine).

3) So, an average session will look like R1->R2->R3->RX where each “R” is a request. So R1 could be index.html, R2 could be “Contact Us”, R3 could be “contact_form.php”, etc. I started using Markov to build a model; however, instead, I took each set of 2 and initialized those values…e.g. S={R1->R2,R2->R3,R3->RX}. For the next session I might have S={R1->R5,R5->R3,etc.}. At the end of all the parsing, I have a big set of all state transitions possible for each R. So, given RX, there are a finite number of R states that RX can transition to.

4) For each of the R states, I now re-parse the log file and find the number of transitions. This is a matrix that shows the number of observed transitions from RN to every other R state. So, for instance, let’s say that R1 goes to 3 possible states : R4 (27% of time), R11 (3% of time) , and R12 (70% of time). Then the R1 row of our matrix looks like [0, 0, 0, .27, 0, 0, 0, 0, 0, 0, .03, .7]

5) There were some special cases that I had to account for (any page transitioning to the main page, any page transitioning to itself, etc.). Once I accounted for these, I ran my program against the log files and created LOW, MEDIUM, and HIGH alerts. I didn’t use a true standard deviation and I ignored the LOW and MEDIUM stuff…I just wanted the hits where the number for that transition was extremely low or 0. From our example above, this would be a transition like R1->R2=0. I didn’t really expect great results and figured that I would have to do a lot more tweaking…well, this wasn’t the case. I actually got really, really good data on my first run. Example:

732 total state transitions tracked
HIGH RISK GET /componentes3.7/fckeditor/editor/fckeditor.html->GET /affiliate/affiliate53/fckeditor/editor/fckeditor.html

HIGH RISK GET /portfolio/aui/FCKeditor/editor/fckeditor.html->GET /componentes3.7/fckeditor/editor/fckeditor.html

HIGH RISK GET /wp-content/uploads/wpfouot.php->POST /wp-content/plugins/Login-wall-etgFB/login_wall.php

etc.

So, I can use really basic machine learning to find my attackers in my web logs. I then parse out the attackers’ IP addresses and can throw them into a firewall ruleset. In the future, I would like to automate this and find when my server is under attack, send a message to my firewall which drops in a route rule which spins all of the attackers traffic to my honey net :)

Speaking of honeypots, You can also honeypot certain pages. For instance, I could create bogus files or directories based on what I see attackers going after (like the report from above) and drop canary tokens in there to (see Canary Tools). I can embed honeypot links within HTML comments and see where bots (or humans) are taking links from commented code and trying them out. I can put links in my robots.txt file and see who goes after them…there are so many ways to do this…and, at the end of the day, I can either run these attackers off my network or into a fake network…it’s just TONS and TONS of fun :)

!Dmitry
dmitry.chan@gmail.com

    SecuriTeam Secure Disclosure

    SecuriTeam Secure Disclosure (SSD) helps researchers turn their vulnerability discovery skills into a highly paid career. Contact SSD to get the most for your hard work.

SSD Advisory – Acunetix WVS XSS, Memory Exhaustion and DoS

Vulnerability Description
Three security vulnerabilities have been discovered in Acunetix WVS, these vulnerabilities allow a site owner that knows that his site will scanned by Acunetix (with permission or without) to target the user of the Acunetix and to cause the product to crash, exhaust memory of the scanner or to trigger a cross site scripting attack against the user during the configuration step and during the user’s reading of the final report.

All these vulnerabilities do not pose a harm greater than being an annoyance, beside the XSS which could be leveraged to preform cause more harm if it is combined with some social engineering aspects.

    SecuriTeam Secure Disclosure

    SecuriTeam Secure Disclosure (SSD) helps researchers turn their vulnerability discovery skills into a highly paid career. Contact SSD to get the most for your hard work.

SSD Advisory – Xerox DocuShare Multiple Vulnerabilities

Introduction
DocuShare is a content management system developed by Xerox Corporation. DocuShare makes use of open standards and allows for managing content, integrating it with other business systems, and developing customized and packaged software applications.

Multiple vulnerabilities have been found in Xerox DocuShare:
 

  • DSUtilityLib.HelperObj.4 Activex Control ShowHelp Method lstrcatW() Call Stack Buffer Overflow Vulnerability
  • DSUtilityLib.HelperObj.4 ActiveX Control GetResourceString Method _vswprintf() Call Stack Buffer Overflow Vulnerability
  • DSUtilityLib.HelperObj.4 Activex Control ProfileInt Property wsprintfW() Call Stack Buffer Overflow Vulnerability
  • DSITEMENUMLib.ItemObj.4 Activex Control Basetype Property Stack Buffer Overflow Vulnerability
  • DsSearch.SearchConsole.1 ActiveX Control RestrictGlobalScope Method wcscpy() Call Stack Buffer Overflow Vulnerability
  • DSUtilityLib.HelperObj.4 Activex Control RunCommand Method CreateProcessW() Call Command Execution Vulnerability
    SecuriTeam Secure Disclosure

    SecuriTeam Secure Disclosure (SSD) helps researchers turn their vulnerability discovery skills into a highly paid career. Contact SSD to get the most for your hard work.

SSD Advisory – EMC RecoverPoint for Virtual Machines (VMs) Restriction Bypass

Vulnerability Description
RecoverPoint’s virtual appliance can be accessible via SSH with the default credentials of boxmgmt:boxmgmt; during testing, no password change option was found. Using these credentials, it’s possible to escape the management interface via command injection to drop into a shell and further take advantage of sudo privileged operations to read arbitrary files as root. It also may also be possible to execute arbitrary os commands as root, but this was not confirmed.

    SecuriTeam Secure Disclosure

    SecuriTeam Secure Disclosure (SSD) helps researchers turn their vulnerability discovery skills into a highly paid career. Contact SSD to get the most for your hard work.

SSD Advisory – Infinite Automation Systems Mango Cross Site Scripting and Arbitrary File Upload

Introduction
Infinite Automation Systems is headquartered in Lafayette, Colorado.

The affected product, Mango Automation, is a centralized web-based SCADA/HMI and data acquisition software. According to Infinite Automation Systems, Mango Automation is deployed across several sectors including Commercial Facilities, Critical Manufacturing, Food and Agriculture, and Energy. Infinite Automation Systems estimates that these products are used worldwide.

Vulnerable Versions
Mango Automation version 2.5.0 through Version 2.6.0 beta (builds prior to 430)

Vulnerability Description
Improper verification of uploaded image files allows arbitrary files to be uploaded, which may allow for the execution of malicious JSP script files. In addition, the application does not verify HTTP requests, causing it to be vulnerable to a cross site scripting vulnerability.

    SecuriTeam Secure Disclosure

    SecuriTeam Secure Disclosure (SSD) helps researchers turn their vulnerability discovery skills into a highly paid career. Contact SSD to get the most for your hard work.

SSD Advisory – eBay Arbitrary Invoice Disclosure

Vulnerability Description
A vulnerability in the way invoices are handled by eBay allows users that sell items on eBay to view other’s reseller’s invoices. Though access to the invoice is somewhat arbitrary, there is no easy way to find a specific invoice of a specific seller, it is possible to harvest a large amount of invoice and gather sensitive information from them. This information includes (though not in all invoices):

    SecuriTeam Secure Disclosure

    SecuriTeam Secure Disclosure (SSD) helps researchers turn their vulnerability discovery skills into a highly paid career. Contact SSD to get the most for your hard work.

SSD Advisory – Media Wiki SVG XSS

Introduction
MediaWiki is a free software open source wiki package written in PHP, originally for use on Wikipedia. It is now also used by several other projects of the non-profit Wikimedia Foundation and by many other wikis, including this website, the home of MediaWiki.

Vulnerable Version
Media Wiki version 1.24.1

Vendor Response
The vulnerability has been addressed in Media Wiki version 1.24.2.

Vulnerability Details
A vulnerability in the way Media Wiki handles SVG files allow attackers to cause it to display arbitrary javascript code to users that are presented with an embedded SVG file. The vulnerability is triggered through the use of an encoded ENTITY that doesn’t get properly filtered out for malicious content.

    SecuriTeam Secure Disclosure

    SecuriTeam Secure Disclosure (SSD) helps researchers turn their vulnerability discovery skills into a highly paid career. Contact SSD to get the most for your hard work.

SSD Advisory – Remote Command Execution in Proliant iLO Intelligent Provisioning

Vulnerability Description
iLO is an embedded operating system available within HP Proliant and Integrity servers. IP is a feature within iLO that provides local and remote access for provisioning purposes. It was discovered that hidden requests were being made to server during a normal client session. Exploring this obfuscated functionality revealed the ability to execute arbitrary commands as root on the system.

    SecuriTeam Secure Disclosure

    SecuriTeam Secure Disclosure (SSD) helps researchers turn their vulnerability discovery skills into a highly paid career. Contact SSD to get the most for your hard work.