In Metasploit 4.10, we converted Metasploit Framework (and prosvc in Metasploit Commercial Editions) to be a full-fledged
Rails::Application. You may be wondering why Metasploit Framework and prosvc, should be Rails applications when they aren't serving up web pages. It all has to do with not reinventing the wheel and very useful parts of Rails,
Rails 3.0 infrastructure
Since Rails 3.0, Rails has been broken into multiple gems that didn't require each other and could be used or not used.
To tie all these gems together, and allow other gems too be used in their place, a gem, railties, sits at the bottom of rails and supplies the infrastructure to connect different
Rails::Railties together. For our purposes we care about two features of
raketasks defined in other gems to be called.
- Allow initializers to be defined in a set order.
On top of
Rails::Railtie is built
Rails::Engine, which is where a lot of the conventions that are thought of as Rails Conventions reside:
- standardized paths like
- Automatic code loading (when we follow standard naming conventions that make directories correspond to namespace and
Classes be files).
Finally, on top of
Rails::Engine sits a single
Rails::Application, which is in charge of running all the initializers from the
Rails::Engines. Without a
Rails::Application the initializers won't automatically run and functionality from Rails ecosystem gems is lost.
Working around not having a Rails::Application before 4.10
In Metasploit Framework 4.9 we had to work-around not having access to (
Railtie 1) by manually loading the
yard rake task from
metasploit_data_models. We used the
yard.rake defined in metasploit_data_models so we didn't have to duplicate code. Eliminating duplicate code is a long term goal of my work on Metasploit Framework as a smaller code base is easier to maintain (because it's faster to test and easier to developers to understand) and we don't run the risk of fixing bugs in one copy of the code, but not the others.
Additionally, we had to port (i.e copy and slightly change which mean we won't be able to just grep for duplicates) portions of
activerecord's databases.rake to support all the
rake db* commands used to test against the database with rspec for Metasploit Framework 4.7.
We couldn't take advantage of (
Railtie 2) because initializers are only run by
Rails::Application#initialize! and Metasploit Framework didn't have a
Metasploit::Framework::Application, which meant we couldn't take advantage of Rails community helpers like
factory_girl_rails to automatically load factories in tests or
rspec-rails to manage transactions for tests. Not having transactions has led to lot of hard to track down bugs with database records from one test interfering with another test.
In Metasploit Commercial Editions 4.9, we were already using a
MetasploitDataModels::Engine to automatically load the
Mdm models in the UI, but since Metasploit Framework 4.9 and prosvc didn't support
Rails::Engines, we had to duplicate the code logic . Duplicate code means both code paths need to be tested and since this code path was only exercised in Metasploit Framework and prosvc, any changes to loaded code needed to be tested all the way up the stack through Metasploit Framework and prosvc. All of these complexities added up to slower development on metasploit_data_models.
Impetus for adopting
As you can see, there was a lot of duplicate code cruft building up to support the non-Rails::Application environments in Metasploit Framework and prosvc, which came to a breaking point when we decided to make
metasploit-credential sits on top of
metasploit_data_models and defines credentials models in the
Metasploit::Credential namespace. These models aren't part of
metasploit_data_modelsfor the following reason:
- Better understandability because developers don't have to know which parts of
metasploit_data_modelsto ignore when dealing with credentials.
- Faster tests because
metasploit-credentialcan be tested independently of
- Cleaner API and versioning because
metasploit-credentialwill only change if there's a change to credential's API and not any change in database API as would be the case with
metasploit_data_modelscontaining the credentials code.
ActiveRecord associations being declared on each side of the relationship,
metasploit-credential would need to be able to monkey patch inverse associations onto
Mdm::Service to associate it back to
Metasploit::Credential::Login. In Ruby, monkey patching can be accomplished a number of ways
- Reopen the class
(1) and (2) have a downside that it's very hard to keep track of the monkey patches since Ruby doesn't keep track where a Class is defined (or redefined). (The closest it can do is give the source location of a method.) So, (1) and (2) aren't good solutions as they drive anyone that works on the code later crazy. (3) and (4) are good, but the problem is how to ensure the
include calls happen dynamically when the class to be monkey patched is loaded.
It turned out we already had a specific solution in Metasploit Commercial Editions for loading modules from
pro/ui/lib/mdm to add Metasploit Commercial Editions monkey patches to
Mdm models, but the problem was that it depended on running Rails initializer, which wouldn't work in metasploit-framework and only worked in prosvc because it manually loaded the initializer and ran it as a script at the appropriate point in the prosvc startup. So, we were left with either implementing both an elegant Rails solution using its initializers and a port to work for Metasploit Framework and prosvc to support
metasploit-credential extensions to
metasploit_data_models or making Metasploit Framework and prosvc be
Rails::Applications so they could run initializers like Metasploit Commercial Editions' UI. I chose making
Converting to Rails::Applications
Rails::Applications allowed to me to do one of the best things you can do in a commit: delete more code than you add.
We were able to completely remove
lib/tasks/rails.rake, which means we no longer have to watch out for changes to those in upstream
activerecord. We're using them from
active_record/railtie now! Other rake tasks are now defined by
*-rails gems, like
rake spec from
rspec-rails, which also gave us transactional fixtures, which means that specs will automatically start and rollback database transactions after each example.
factory_girl_rails automatically loads the factories from
yard.rake is automatically loaded from
metapsloit_data_models just because it's under
Metasploit::Framework::Application also gave all these other side benefits:
rails console can be used to access the
Metasploit::Credential models without having to boot
msfconsole and then running
irb. You can access the SQL prompt for the database directly with
What does all this have to do with Kali?
So, when a
Rails::Application boots, it needs to opt-in to requiring the railties from the core gems, including
active_record/railtie. While 4.10 was in development, the Metasploit Applications team developed with
active_record/railtie always being loaded as it ensured that the database was connected and
ActiveRecord::Base.logger was setup, which made triage and debugging easier; however, we knew eventually that requiring
active_record/railtie would have to be optional based on whether the db Gemfile group was installed and whether the
-n (disable database) flag was passed to
msfconsole. Since I was handling users that didn't install database support at all or turned it off with
-n, I thought we were covered for the 4.10 release and this is the state of the boot process that shipped.
It turned out that although Kali documents to start
postgresql and setup a
~/.msf4/database.yml, many users hadn't found that documentation and so were starting
msfconsole without a
database.yml and/or without
postgresql running, but without the
-n flag. Not having a
database.yml led to the
ENOENT error everyone was seeing since
Metasploit::Framework::Application tried to read the non-existent
database.yml when the database wasn't disabled. James "egypt" Lee and I worked to fix this and egypt came up with checking if the file exists before trying to load
active_record/railtie. This fixed the issue of no
database.yml, but not having a
postgresqlnot started. Egypt and I discussed various ways around this, but it came down to adding a lot of error detection and making a direct connection using the low-level
PGConn from the
pg gem so we could test the
database.yml could connect to the database OR never requiring
active_record/railtie. We went with not requiring
active_record/railtie. Now, in 4.10.1, the database connection is just made by
msfconsole how it had been in 4.9. We lost
ActiveRecord::Base.logger setup in
rails console and
msfconsole, but those were new additions for Metasploit Framework 4.10 anyway, so it was an acceptable loss to get Kali's msfconsole to boot under excepted configuration scenarios and led to a more robust boot process. If we want to restore the logger for the next release we can look into moving the error handling out of
msfconsole into a place that the Rails boot can use it.
Goodies for the future
To leave on a brighter note, here some cool features of having access to
Rails.application we hope to use in the future or that anyone in the community can use.
Rails::Application can iterate the
Rails::Engines plugged into it, we made migrations automatically load from any
Rails::Engine we add to the
metasploit ecosystem. Meterpreter extensions are also copied from
Rails::Engines . Finally module paths from
Rails::Engines are automatically added to
That's right, you make a gem with a
Rails::Engine and put it in your
Gemfile.local and metasploit-framework will treat it just like the normal metasploit-framework/modules. We hope to use this ability in the future to break up the modules into target-specific gems so as the number of modules grow users can choose which loadouts they want for a given engagement. This will make it easier to keep running metasploit-framework on small form factor devices with limited space.