Skip navigation
All Places > Metasploit > Blog > Authors luke_imhoff

Metasploit

2 Posts authored by: luke_imhoff Employee

Introduction

As Metasploit adopts community best practices related to testing and code structure, we've started breaking up our new (and part of our old) work into separate projects that can developed and tested independently.  These smaller projects take the form of Ruby gems and Rails::Engines.  As we've made more and more gems, we've gotten a repeatable process for making new gems, but we thought our process might we useful for other developers in the community that are looking for a newer guide on how to setup ruby gems from scratch and get the most out of all the free tools available to open source projects.

 

Skeleton

Github

Your gem will be open source, so you're going to want to host your source control server on one of the free source control services.  We use github.com and I'd recommend it because other services are likely to have Github integration before support for the other source control service like Gitorious or Bitbucket.

Account Setup

If you don't have a Github account, set one up by signing up at https://github.com.

Repository Creation

Once logged into your Github account, when you go to https://github.com, you'll see two columns, the left hand column is an activity stream while the right column lists repository. Click the big green, "+ New Repository" button to add a repository.

Repository Naming

What your name your repository is up to you, but you should keep rubygems naming conventions in mind:

Other Options

You can leave the rest of the options with their default values and click "Create repository".  After the repository is created, Github will display options for upload your code, but we haven't created any code yet, so let's do that.

RVM Installation

When developing Ruby projects on OSX (or Linux) we recommend using rvm (Ruby Version Manager), it's what we use.  Go to https://rvm.io and follow the installation instructions.

Local Repository Creation

The point of RVM is you can have a different version of ruby and gems that you depend on for each project you work on.  We'll use the ability to setup the tools to create your gem's repository.  In the below steps, replace <GEM_NAME> with the name of the repository you created on Github and <GITHUB_USERNAME> with your username on Github.

  1. Create a gemset for your gem: rvm use --create ruby-2.1@<GEM_NAME>
  2. Create the parent directory for your local git repository

    1. cd ~
    2. mkdir git
    3. mkdir <GITHUB_USERNAME>
    4. cd <GITHUB_USERNAME>
  3. Create the gem skeleton using bundle: bundle gem <GEM_NAME>
  4. cd  <GEM_NAME>
  5. bundle gem won't create a commit in the git repository, so do that now: git commit -m "bundle gem <GEM_NAME>"

Now, look back at Github in your browser and you'll see the instructions for pushing your local git repository to the remote repository on Github.  I've copied the instructions here for easier use:

git remote add origin git@github.com:<GITHUB_USERNAME>/<GEM_NAME>.git
git push -u origin master

Once the push has finished, refresh your browser and you'll see the gem skeleton in your Github repository.

Tool Integration

With the push to Github, you've now successfully started an open source gem project, but instead of stopping there, let's go on to add a common set of tools to take advantage of the project being open source.

Local Tools

We'll start with setting up some local tools that will help with development on our own machine.

RSpec

Metasploit uses RSpec for a unit test framework.  RSpec can set itself up on installation, but must also be declared in the gemspec file.

  1. Add the rspec as a development dependency to <GEM_NAME.gemspec>:
    Gem::Specification.new do |spec|
        # ...
        spec.add_development_dependency 'rspec', '~> 3.1'
        # ...
    end
    
  2. Install the new gems: bundle install
  3. Use rspec to set itself up: rspec --init

rspec --init will choose some good defaults and include comments on other options you can enable in spec/spec_helper.rb, but for Metasploit we want to make sure we're using the rspec 3 features as well as possible, so we turn on some additional options and our spec/spec_helper.rb looks like this:

$LOAD_PATH.unshift File.expand_path('../../lib', __FILE__)

require 'GEM_NAME'

RSpec.configure do |config|
  config.expose_dsl_globally = false

  # These two settings work together to allow you to limit a spec run
  # to individual examples or groups you care about by tagging them with
  # `:focus` metadata. When nothing is tagged with `:focus`, all examples
  # get run.
  config.filter_run :focus
  config.run_all_when_everything_filtered = true

  # allow more verbose output when running an individual spec file.
  if config.files_to_run.one?
    # RSpec filters the backtrace by default so as not to be so noisy.
    # This causes the full backtrace to be printed when running a single
    # spec file (e.g. to troubleshoot a particular spec failure).
    config.full_backtrace = true
  end

  # Print the 10 slowest examples and example groups at the
  # end of the spec run, to help surface which specs are running
  # particularly slow.
  config.profile_examples = 10

  # Run specs in random order to surface order dependencies. If you find an
  # order dependency and want to debug it, you can fix the order by providing
  # the seed, which is printed after each run.
  #     --seed 1234
  config.order = :random

  # Seed global randomization in this process using the `--seed` CLI option.
  # Setting this allows you to use `--seed` to deterministically reproduce
  # test failures related to randomization by passing the same `--seed` value
  # as the one that triggered the failure.
  Kernel.srand config.seed

  config.expect_with :rspec do |expectations|
    # Enable only the newer, non-monkey-patching expect syntax.
    expectations.syntax = :expect
  end

  # rspec-mocks config goes here. You can use an alternate test double
  # library (such as bogus or mocha) by changing the `mock_with` option here.
  config.mock_with :rspec do |mocks|
    # Enable only the newer, non-monkey-patching expect syntax.
    # For more details, see:
    #   - http://teaisaweso.me/blog/2013/05/27/rspecs-new-message-expectation-syntax/
    mocks.syntax = :expect

    mocks.patch_marshal_to_support_partial_doubles = false

    # Prevents you from mocking or stubbing a method that does not exist on
    # a real object.
    mocks.verify_partial_doubles = true
  end
end

To finish the RSpec setup, we want it runnable using rake.  We can do this by adding the following to Rakefile:

require 'rspec/core/rake_task'

RSpec::Core::RakeTask.new(:spec)

task :default => :spec

You can test that your specs by calling rspec, rake spec, or just rake now, but no tests are defined, so rspec isn't testing anything for us.  To get some benefit from rspec we'll need to add spec file.  Usually, we'd test the gem versioning at this point.

  1. mkdir spec/<GEM_NAME>
  2. edit spec/<GEM_NAME>/version_spec.rb
  3. Add the following code to spec/<GEM_NAME>/version_spec.rb<:
    require 'spec_helper'
    
    RSpec.describe GEM_NAME do
      context 'CONSTANTS' do
        context 'VERSION' do
          subject(:version) {
            described_class::VERSION
          }
    
          it { is_expected.to be_a String }
          it { is_expected.to match_regex(/\d+.\d+.\d+(-[a-zA-Z0-9]+)*/) }
        end
      end
    end
    
  4. Run your new spec: rake spec

SimpleCov

We can use RSpec to test our code, but how do we know our tests are testing all the gem code?  We need to use a tool that measure test coverage, SimpleCov.

  1. Add simplecov as a development dependency to <GEM_NAME>.gemspec:
    Gem::Specification.new do |spec|
        # ...
        spec.add_development_dependency 'simplecov'
        # ...
    end
    
  2. Install simplecov: bundle install
  3. Require simplecov before any other file in spec/spec_helper.rb
    # require before anything else so coverage is shown for all project files
    require 'simplecov'
    
  4. Metasploit uses a SimpleCov configuration file, .simplecov, that works both with rake spec and Rubymine, our Ruby IDE of choice:
    # RM_INFO is set when using Rubymine.  In Rubymine, starting SimpleCov is
    # controlled by running with coverage, so don't explicitly start coverage (and
    # therefore generate a report) when in Rubymine.  This _will_ generate a report
    # whenever `rake spec` is run.
    unless ENV['RM_INFO']
      SimpleCov.start
    end
     
    SimpleCov.configure do
      # ignore this file
      add_filter '.simplecov'
     
      # Rake tasks aren't tested with rspec
      add_filter 'Rakefile'
      add_filter 'lib/tasks'
     
      #
      # Changed Files in Git Group
      # @see http://fredwu.me/post/35625566267/simplecov-test-coverage-for-changed-files-only
      #
     
      untracked = `git ls-files --exclude-standard --others`
      unstaged = `git diff --name-only`
      staged = `git diff --name-only --cached`
      all = untracked + unstaged + staged
      changed_filenames = all.split("\n")
     
      add_group 'Changed' do |source_file|
        changed_filenames.detect { |changed_filename|
          source_file.filename.end_with?(changed_filename)
        }
      end
     
      add_group 'Libraries', 'lib'
     
      #
      # Specs are reported on to ensure that all examples are being run and all
      # lets, befores, afters, etc are being used.
      #
     
      add_group 'Specs', 'spec'
    end
    
  5. Now, when you run rake spec, you'll generate a coverage report.
  6. Open the coverage report open coverage/index.html

YARD

By default, Ruby comes with a documentation format called RDoc, but for Metasploit, we use YARD because it allows for more structured documentation with the addition of `@tags` that would be familiar to anyone that's used Doxygen or Javadoc and the ability to write extensions as Metasploit has done with yard-metasploit-erd, which allows use to include ERDs (Entity-Relationship Diagram in our documentation to make working with the database easier.

At Rapid7, we've just released metasploit-yard, so we can share a common rake yard task.  The READMEexplains how to set it up:

  1. Add metasploit-yard as a development dependency to <GEM_NAME>.gemspec:
    Gem::Specification.new do |spec|
        # ...
        spec.add_development_dependency 'metasploit-yard', '~> 1.0'
        # ...
    end
    
  2. Install metasploit-yard: bundle install
  3. Add the following to your Rakefile to load yard.rake from metasploit-yard:
    # Use find_all_by_name instead of find_by_name as find_all_by_name will return pre-release versions
    gem_specification = Gem::Specification.find_all_by_name('metasploit-yard').first
    
    Dir[File.join(gem_specification.gem_dir, 'lib', 'tasks', '**', '*.rake')].each do |rake|
      load rake
    end
    
  4. Run rake yard to see what you need to document.

Remote Tools

With RSpec, SimpleCov, and YARD setup, we can run tests and generate documentation locally, but doing everything locally is not enough, we need a second system, that isn't our develop machine to run these steps too to ensure that they aren't passing just because of some fluke of our local machine.

Travis CI

To test remotely, Metasploit uses Travis CI.  Travis CI is free for open source software and integrates well with github, as Travis CI is able to automatically test that Pull Requests pass our specs.  This is a big help on metasploit-framework where we're up to 3785 pull requests and 9200 builds.  Without Travis CI we'd either have had to run those all by hand, go through an error-prone and time-consuming manual process, or roll a Jenkins build pipeline to handle all of that, which would be a significant maintenance overhead.

On Travis CI, we can build against multiple implementations and versions of Ruby.  Metasploit tries to build all versions of MRI Ruby and the latest stable versions of JRuby and Rubinius (rbx in the .travis.yml).  We need to setup a Travis-CI account and then push a .travis.yml to Github:

  1. Go to https://travis-ci.org/
  2. Click "Sign in with Github" in the top-right corner
  3. On the left sidebar, click the "+" button to add a new repository for Travis CI to build.
  4. Select a repository to build by changing the slider on the right hand side of the table row from "Off" to "On"
  5. Add a .travis.ymlto the root of your local git repository:
    env:
      matrix:
        - RAKE_TASK=spec
    language: ruby
    matrix:
      # documentation coverage is not influenced by ruby implementation, so only run once
      include:
        - rvm: 2.1
          env: RAKE_TASK=yard
    rvm:
      - '1.9.3'
      - '2.0'
      - '2.1'
      - 'ruby-head'
      - 'jruby-19mode'
      - 'rbx-2.2'
    script: "bundle exec rake $RAKE_TASK"
    
  6. Add a Travis CI badge to your README.md so users can quickly see if your project's build is failing when visiting the repository on Github: [![Build Status](https://travis-ci.org/<GITHUB_USER>/<GEM_NAME>.svg?branch=master)](https://travis-ci.org/<GITHUB_USER>/<GEM_NAME>). When done correctly it will look like this:
  7. Commit your change: git commit -am "Travis CI"
  8. Push your changes: git push
  9. Watch the build on Travis CI

CodeClimate

CodeClimate is a automated code quality analyzer. Metasploit uses it for both our open source and private repositories; however, CodeClimate can't cope with the sure volume of code in metasploit-framework: metasploit-framework itself is a DoS attack on static analysis tools.  Like Travis CI, CodeClimate has a badge, which shows the Code Quality using a 4.0 GPA scale. CodeClimate also monitors code coverage, so we'll be adding a coverage report on top of SimpleCov.

  1. Go to https://codeclimate.com and sign up for an account
  2. On your dashboard click the "Add Open Source Repo" button
  3. Enter your repo name: <GITHUB_USER>/&ltGEM_NAME>
  4. Click "Import Repo from Github"
  5. On the repository page, click the "Setting" link.
  6. Click the "Test Coverage" button in the left sidebar.
  7. Copy the CODECLIMATE_REPO_TOKEN
  8. You'll want to encrypt the CODECLIMATE_REPO_TOKEN on travis-ci to prevent forks from affecting the original repo's coverage, so install the travis gem: gem install travis
  9. Encrypt the token: travis encrypt CODECLIMATE_REPO_TOKEN=<YOUR_TOKEN_HERE>
  10. Put the encrypted value (remember to substitute your value and not to use the example value below) in your .travis.yml:
    env:
      global:
        secure: "G0LDGrupZ+RAFzoPwd6bjfrWfwoU/V9RTswQXIUNmi640rW/CP86a8F9hQcAXdUwy7Ag1cwmlEEv2JDDbLopQdNvkh8B7M3PSIpmJWlBsj+UPMMcCz04oYiQB3G6xgYb4xeuY6cNqxiY2rTJMHlfUfTlSJf1Z0dh6hAS3alHpGw="
    
  11. Add the CodeClimate badge to the README: [![Code Climate](https://codeclimate.com/github/<GITHUB_USER>/<GEM_NAME>.png)](https://codeclimate.com/github/<GITHUB_USER>/<GEM_NAME>)
  12. Add the codeclimate-test-reporteras a development dependency:
    Gem::Specification.new do |spec|
      # ...
      spec.add_development_dependency 'codeclimate-test-reporter'
      # ...
    end
    
  13. bundle install
  14. Add following after require 'simplecov'in spec/spec_helper.rb
    # ...
    
    require 'codeclimate-test-reporter'
    
    if ENV['TRAVIS'] == 'true'
      formatters = []
    
      # don't use `CodeClimate::TestReporter.start` as it will overwrite some .simplecov settings
      if CodeClimate::TestReporter.run?
        formatters << CodeClimate::TestReporter::Formatter
      end
    
      SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter[
        *formatters
      ]
    end
    
    # ...
    
  15. git commit -am "CodeClimate"
  16. git push

Coveralls

Although CodeClimate has a coverage reporter, it only reports if there is coverage issues, to view the actual coverage reports online, as outputed locally by simplecov, Metasploit uses Coveralls.

  1. Go to https://coveralls.io
  2. Click "Sign up free with Github" in the top-right corner
  3. Click the "Add Repo" button on the Your Repositories screen
  4. Change the slider from "Off" to "On" next to your repository
  5. Add the Coveralls badge to your README: [![Coverage Status](https://img.shields.io/coveralls/<GITHUB_USER>/<GEM_NAME>.svg)](https://coveralls.io/r/<GITHUB_USER>/<GEM_NAME>), which will look like this: Coverage Status
  6. Add coveralls as a development dependency:
    Gem::Specification.new do |spec|
      # ...
      spec.add_development_dependency 'coveralls'
      # ...
    end
    
  7. bundle install
  8. Change the SimpleCov.formatters in spec/spec_helper.rb to include Coveralls::SimpleCov::Formatter:
    # ...
    
    require 'codeclimate-test-reporter'
    require 'coveralls'
    
    if ENV['TRAVIS'] == 'true'
      formatters = []
    
      # don't use `CodeClimate::TestReporter.start` as it will overwrite some .simplecov settings
      if CodeClimate::TestReporter.run?
        formatters << CodeClimate::TestReporter::Formatter
      end
    
      formatters << Coveralls::SimpleCov::Formatter
    
      SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter[
        *formatters
      ]
    end
    
    # ...
    
  9. Add a .coveralls.yml file
    service_name: travis-ci
    
  10. git commit -am "Coveralls"
  11. git push

Gemnasium

Metasploit's various gems depend on many other gems from the community.  We need to keep track of new versions of those gems in case there are vulnerabilities both so we can update our usage of those gems and so we can make Metasploit Framework modules to exploit those vulnerability.  To monitor our gems, we use Gemnasium.  Gemnasium has sent us alerts about Rails vulnerabilities that have led to Rapid7's rapid vulnerability mitigation response with new updates across our Portfolio multiple times in the past.

  1. Go to https://gemnasium.com
  2. Click "Sign up"
  3. Click "Sign up with Github"
  4. Click "Authorize application" on Github
  5. Click "+ Add New Project"
  6. Click "Hosted on Github"
  7. Click "Allow Access" in the "Public Project" column so Gemnasium can automatically configure the Github hooks
  8. Click "Authorize application" on Github to authorize the expanded permissions for Gemnasium
  9. Click your username to expand the list of your repositories
  10. Check the repositories to monitor
  11. Click Submit
  12. Add the Gemnasium badge to the README: [![Dependency Status](https://gemnasium.com/<GITHUB_USER>/<GEM_NAME>.svg)](https://gemnasium.com/<GITHUB_USER>/<GEM_NAME>)
  13. git commit -am "Gemnasium"
  14. git push

Rubygems

Although you haven't released your gem yet, once you do, you'll want users that visit your github repository to be able to find the latest released gem, which you can do with a Rubygems version badge

  1. Add to your README: [![Gem Version](https://badge.fury.io/rb/<GEM_NAME>.svg)](http://badge.fury.io/rb/<GEM_NAME>)
  2. git commit -am "Gemnasium"
  3. git push

Inch CI

Inch CI checks our documentation as a second review of the checking Metasploit already does with metasploit-yard.

  1. Go to http://inch-ci.org
  2. Enter the URL to your github repository: https://github.com/<GITHUB_USER>/<GEM_NAME>
  3. In the top-right corner click the "doc" badge
  4. Select "SVG" for the image format
  5. Copy the "Markdown" badge
  6. Add it to your README
  7. git commit -am "Inch CI"
  8. git push

Pull Review

PullReview is another static analysis tool like CodeClimate.  Metasploit uses both because PullReview has a more detailed reviews and better handling of branches than CodeClimate, but CodeClimate is cheaper for our team for private repositories.  Just like CodeClimate, PullReview can't handle analysing metasploit-framework, but instead of lagging behind and reporting odd or out-of-date analysis, PullReview just won't return an analysis. Metasploit has had discussion with PullReview and they were nice enough to send us an offline analysis.

  1. Go to https://www.pullreview.com
  2. Enter your github username and sign up
  3. Click "Authorize application" on Github
  4. Click "Authorize access to my repositories on Github"
  5. Click "Authorize application" on Github to authorize PullReview to post reviews and setup hooks
  6. Click the "+ Review" button next to repository you want reviewed
  7. Add the badge to your README: [![PullReview stats](https://www.pullreview.com/github/<GITHUB_USER>/<GEM_NAME>/badges/master.svg?)](https://www.pullreview.com/github/<GITHUB_USER>/<GEM_NAME>/reviews/master), which should look like this: , which is a bit cooler than the CodeClimate badge.

Conclusion

So, that's everything.  Don't be intimidated by the length of this article: it takes about a day or less for me to set this all up on new projects.  If there are open source tools you think Metasploit should be using (such a static analysis tool that can handle metasploit framework and our odd Metasploit module format) let us know in the comments below.

In Metasploit 4.10, we converted Metasploit Framework (and prosvc in Metasploit Commercial Editions) to be a full-fledged Rails::Application.  You may be wondering why Metasploit Framework and prosvc, should be Rails applications when they aren't serving up web pages.  It all has to do with not reinventing the wheel and very useful parts of Rails, Rails::Railtie and Rails::Engine.

Rails 3.0 infrastructure

Since Rails 3.0, Rails has been broken into multiple gems that didn't require each other and could be used or not used.

  • actionmailer
  • actionpack
  • activerecord
  • activeresource
  • activesupport

Rails::Railtie

To tie all these gems together, and allow other gems too be used in their place, a gem, railties, sits at the bottom of rails and supplies the infrastructure to connect different Rails::Railties together.  For our purposes we care about two features of Rails::Railtie

  1. Allow rake tasks defined in other gems to be called.
  2. Allow initializers to be defined in a set order.

Rails::Engine

On top of Rails::Railtie is built Rails::Engine, which is where a lot of the conventions that are thought of as Rails Conventions reside:

  1. standardized paths like lib and app/models.
  2. Automatic code loading (when we follow standard naming conventions that make directories correspond to namespace and Classes be files).

Rails::Application

Finally, on top of Rails::Engine sits a single Rails::Application, which is in charge of running all the initializers from the Rails::Railties and Rails::Engines. Without a Rails::Application the initializers won't automatically run and functionality from Rails ecosystem gems is lost.

Working around not having a Rails::Application before 4.10

In Metasploit Framework 4.9 we had to work-around not having access to (Railtie 1) by manually loading the yard rake task from metasploit_data_models.  We used the yard.rake defined in metasploit_data_models so we didn't have to duplicate code.  Eliminating duplicate code is a long term goal of my work on Metasploit Framework as a smaller code base is easier to maintain (because it's faster to test and easier to developers to understand) and we don't run the risk of fixing bugs in one copy of the code, but not the others.

Additionally, we had to port (i.e copy and slightly change which mean we won't be able to just grep for duplicates) portions of activerecord's databases.rake to support all the rake db* commands used to test against the database with rspec for Metasploit Framework 4.7.

We couldn't take advantage of (Railtie 2) because initializers are only run by Rails::Application#initialize! and Metasploit Framework didn't have a Metasploit::Framework::Application, which meant we couldn't take advantage of Rails community helpers like factory_girl_rails to automatically load factories in tests or rspec-rails to manage transactions for tests.  Not having transactions has led to lot of hard to track down bugs with database records from one test interfering with another test.

In Metasploit Commercial Editions 4.9, we were already using a Rails::Engine in MetasploitDataModels::Engine to automatically load the Mdm models in the UI, but since Metasploit Framework 4.9 and prosvc didn't support Rails::Engines, we had to duplicate the code logic .  Duplicate code means both code paths need to be tested and since this code path was only exercised in Metasploit Framework and prosvc, any changes to loaded code needed to be tested all the way up the stack through Metasploit Framework and prosvc. All of these complexities added up to slower development on metasploit_data_models.

Impetus for adopting Rails::Application

As you can see, there was a lot of duplicate code cruft building up to support the non-Rails::Application environments in Metasploit Framework and prosvc, which came to a breaking point when we decided to make metasploit-credential.

metasploit-credential sits on top of metasploit_data_models and defines credentials models in the Metasploit::Credential namespace.  These models aren't part of metasploit_data_modelsfor the following reason:

  1. Better understandability because developers don't have to know which parts of metasploit_data_models to ignore when dealing with credentials.
  2. Faster tests because metasploit-credential can be tested independently of metasploit_data_models.
  3. Cleaner API and versioning because metasploit-credential will only change if there's a change to credential's API and not any change in database API as would be the case with metasploit_data_models containing the credentials code.

Due to ActiveRecord associations being declared on each side of the relationship, metasploit-credential would need to be able to monkey patch inverse associations onto metasploit_data_models like Mdm::Service to associate it back to Metasploit::Credential::Login.  In Ruby, monkey patching can be accomplished a number of ways

  1. Reopen the class
  2. class_eval
  3. extend a Module
  4. include a Module

(1) and (2) have a downside that it's very hard to keep track of the monkey patches since Ruby doesn't keep track where a Class is defined (or redefined).  (The closest it can do is give the source location of a method.) So, (1) and (2) aren't good solutions as they drive anyone that works on the code later crazy.  (3) and (4) are good, but the problem is how to ensure the extend and include calls happen dynamically when the class to be monkey patched is loaded.

It turned out we already had a specific solution in Metasploit Commercial Editions for loading modules from pro/ui/lib/mdm to add Metasploit Commercial Editions monkey patches to Mdm models, but the problem was that it depended on running Rails initializer, which wouldn't work in metasploit-framework and only worked in prosvc because it manually loaded the initializer and ran it as a script at the appropriate point in the prosvc startup.  So, we were left with either implementing both an elegant Rails solution using its initializers and a port to work for Metasploit Framework and prosvc to support metasploit-credential extensions to metasploit_data_models or making Metasploit Framework and prosvc be Rails::Applications so they could run initializers like Metasploit Commercial Editions' UI.  I chose making Rails::Applications.

Converting to Rails::Applications

Converting to Rails::Applications allowed to me to do one of the best things you can do in a commit: delete more code than you add.

We were able to completely remove lib/tasks/database.rake and lib/tasks/rails.rake, which means we no longer have to watch out for changes to those in upstream activerecord.  We're using them from active_record/railtie now!  Other rake tasks are now defined by *-rails gems, like rake spec from rspec-rails, which also gave us transactional fixtures, which means that specs will automatically start and rollback database transactions after each example.  factory_girl_rails automatically loads the factories from metasploit-credential, and metasploit-model, metasploit_data_modelsyard.rake is automatically loaded from metapsloit_data_models just because it's under lib/tasks!

Having Metasploit::Framework::Application also gave all these other side benefits: rails console can be used to access the Mdm and Metasploit::Credential models without having to boot msfconsole and then running irb.  You can access the SQL prompt for the database directly with rails db.

What does all this have to do with Kali?

So, when a Rails::Application boots, it needs to opt-in to requiring the railties from the core gems, including active_record/railtie.  While 4.10 was in development, the Metasploit Applications team developed with active_record/railtie always being loaded as it ensured that the database was connected and ActiveRecord::Base.logger was setup, which made triage and debugging easier; however, we knew eventually that requiring active_record/railtie would have to be optional based on whether the db Gemfile group was installed and whether the -n (disable database) flag was passed to msfconsoleSince I was handling users that didn't install database support at all or turned it off with -n, I thought we were covered for the 4.10 release and this is the state of the boot process that shipped.

It turned out that although Kali documents to start postgresql and setup a ~/.msf4/database.yml, many users hadn't found that documentation and so were starting msfconsole without a database.yml and/or without postgresql running, but without the -n flag.  Not having a database.yml led to the ENOENT error everyone was seeing since Metasploit::Framework::Application tried to read the non-existent database.yml when the database wasn't disabled.  James "egypt" Lee and I worked to fix this and egypt came up with checking if the file exists before trying to load active_record/railtie.  This fixed the issue of no database.yml, but not having a database.yml and postgresqlnot started.  Egypt and I discussed various ways around this, but it came down to adding a lot of error detection and making a direct connection using the low-level PGConn from the pg gem so we could test the database.yml could connect to the database OR never requiring active_record/railtie. We went with not requiring active_record/railtie.  Now, in 4.10.1, the database connection is just made by msfconsole how it had been in 4.9. We lost ActiveRecord::Base.logger setup in rails console and msfconsole, but those were new additions for Metasploit Framework 4.10 anyway, so it was an acceptable loss to get Kali's msfconsole to boot under excepted configuration scenarios and led to a more robust boot process.  If we want to restore the logger for the next release we can look into moving the error handling out of msfconsole into a place that the Rails boot can use it.

Goodies for the future

To leave on a brighter note, here some cool features of having access to Rails.application we hope to use in the future or that anyone in the community can use.

Since a Rails::Application can iterate the Rails::Engines plugged into it, we made migrations automatically load from any Rails::Engine we add to the metasploit ecosystem. Meterpreter extensions are also copied from Rails::Engines .  Finally module paths from Rails::Engines are automatically added to framework.modules

That's right, you make a gem with a Rails::Engine and put it in your Gemfile.local and metasploit-framework will treat it just like the normal metasploit-framework/modules.  We hope to use this ability in the future to break up the modules into target-specific gems so as the number of modules grow users can choose which loadouts they want for a given engagement.  This will make it easier to keep running metasploit-framework on small form factor devices with limited space.

Filter Blog

By date: By tag: