chad

NeXpose automation with Ruby

Discussion created by chad on Jun 24, 2011
Latest reply on Jan 27, 2012 by dpinto

We've had many customers and community users ask us "How do I automate NeXpose to do a certain task" or "How do I get data out of NeXpose into other systems?".  We've built NeXpose to be easy to automate and integrate, and I thought I'd do a quick article to introduce our users to NeXpose automation with Ruby.

 

As you may already know, Metasploit is written in Ruby. In order to integrate Metasploit with NeXpose, HD and his team had to develop a NeXpose API Ruby wrapper, and of course all of the community contributors to Metasploit are already comfortable with Ruby. In addition, Rapid7's Professional Services and solutions architecture teams tend to use Ruby when building custom solutions and integrations for NeXpose. I also work with quite a few customers who are already using Ruby as their language of choice for enterrpise security automation -- Puppet, Chef, and RightScale are all very Ruby-centric and this makes it very easy for people to integrate NeXpose and Metasploit into their environments.

 

So, needless to say that we've been doing a lot of Ruby work lately.  We've decided to release a lot of our Ruby NeXpose work under an open source license (3-clause BSD) so that others can take advantage of it. Recently we've been making a real effort to make the NeXpose Ruby API wrapper re-usable outside of Metasploit, and this has resulted in the NeXpose Ruby gem.  This gem allows you to easily manipulate NeXpose via its web-based API without having to figure out all our XML message formats.

 

We've also built and open-sourced an asynchronous NeXpose scan management layer in Ruby above the NeXpose API layer. The nexpose_scan_manager Ruby gem handles launching and asynchronous polling of NeXpose scans  via the NeXpose API.  This gem is built on top of the Ruby EventMachine library, and it provides a nice, clean way to abstract yourself away from polling and monitoring of scan results.

 

With the nexpose_scan_manager gem, you can easily write your own load-aware scan batching/queueing/load balancing frameworks. Let's say you want to scan a huge address space and you'd like to split that load across N parallel NeXpose scans on N different engines, but never run more than 3 scans at once per engine. The gem makes handles all of the threading, queuing, and asynchronous polling for you.  See the github repo for more information, but here's a very brief example of how easy it is to use the framework to get callbacks on your scans:

 

# Create a NeXpose API Connection

nexpose_connection = NeXpose::Connection.new host, username, password, port

 

# Initialize the scan manager

# Poll NeXpose every 5 seconds for the scan status

poll_time = 5

scan_manager = ScanManager.new nexpose_connection, false, poll__time

 

# Add your observing class to the scan manager that defines an update method

# scan_data contains the scan ID, status, and message if any

class ScanObserver

    def update scan_data, notifier

       <do whatever with scan_data>

    end

end

 

scan_observer = ScanObserver.new

scan_manager.add_observer scan_observer

 

 

# Example B: START SCANS CONDITIONALLY (RESTRICT CONCURRENT SCANS)

 

# Define the condition

 

conditional_scan =

    {

        :site_id   => <site ID>,

        :max_scans => 5,   # Don't start the scan until there are fewer than 5 scans running

        :devices   => <array of devices>, # IPs

        :listeners => listeners # Can be null or an observer as defined in A

    }

 

As a reference example of things you can do with the scan manager, we've written an example script which allows you to scan a NeXpose asset group.

 

Finally, because many people are interested in getting data out of NeXpose and into other systems for processing, we have also open sourced a reference implementation of a flexible, Ruby-based NeXpose CSV exporter.  This exporter (which builds on the other two libraries) allows you to export NeXpose data in CSV, choosing from multiple different fields including vulnerability information (title, CVSS, CVE, other references, description, solution, etc.) and affected host information (IP, FQDN, OS, affected TCP/UDP port(s), MAC address, etc.).

 

We'd love to get your feedback on the Ruby libraries. Please try them out, post here with your questions, send us patches, and create new utilities and integrations and share them with the community!

 

Message was edited by: Michael Burstein  Fixed some broken links.

Outcomes