Skip navigation
All Places > Services > Blog > Tags incident-response
1 2 3 Previous Next


34 Posts tagged with the incident-response tag

Are you ready for an incident? Are you confident that your team knows the procedures, and that the procedures are actually useful? An incident response tabletop exercise is an excellent way to answer these questions. Below, I’ve outlined some steps to help ensure success for your scenario-based threat simulations.


First, identify your audience. This will help inform which type of exercise you’ll want to run. Will it be an executive exercise or technical in nature? It does not make sense to invite your entire C-Suite to a technical exercise, just like it would not go over well to have your technical incident responders drive an exercise that focuses on executive oversight and compliance. This does not mean that there cannot be overlap (some exercises can combine both executive and technical aspects), but the exercise must be managed closely to ensure it’s a good use of time for everyone. You can also involve counsel at this point. Legal counsel provides invaluable guidance and advice for navigating an increasingly complex regulatory environment.


Now that your scope and audience have been set, it is time to define your scenario. This is where many exercises go off the rails. You must set a realistic scenario that truly exercises your organization. Remember, this is a time when you will be pulling together many people who have cleared off their schedules for a few hours. Make it worth their time. Use the maturity of your organization’s incident response (IR) capabilities and the threats to your business to help guide the selection of a scenario for the exercise. For instance, a defense contractor will not have much need to practice a case of adware infection on a handful of machines, and a restaurant will not greatly benefit from preparing for a nation-state threat. You have to find the sweet spot to ensure a successful exercise. It should not be out your team’s reach, yet it also shouldn’t be a softball. And if you intend to conduct multiple exercises over time (which we highly recommend), you will want to keep the audience engaged and ensure they do not dread the effort.


With the audience set and the scenario defined, you can move into scripting the exercise itself. While it is good to set an outline for the time you have everyone together, leave enough flexibility to improvise when needed. This phase of your planning should not involve every potential exercise participant. Limit this to a handful of trusted agents. This is not a case in which more is better. Having all participants help write the test only ensures that the results will be artificially inflated and the assessment will be inaccurate. Unwavering candor is necessary to help the organization truly know where it stands in its preparedness. You would much rather discover deficiencies during practice than during a live event.


Now that you have fully prepared, the steps that remain are executing the exercise and reporting the results. You should not be afraid to call out areas for improvement in your program. Narrow your assessment down to specific facets of incident response; we like to look at clients’ incident response plans, their adherence to those plans, coordination among IR teams, communications (internal and external), and technical analysis. As mentioned before, it is helpful to go over the results with your legal counsel and seek advice for how to proceed with improvements.


At the end of the day, you may not be able to address every finding. As with any aspect of security, your decisions on what to address and how to go about it should be risk-based. We wish you luck in assessing your program, and our experts are happy to help when needed.

You can learn more about the role tabletop exercises play in incident response by watching this Whiteboard Wednesday, and be sure to keep an eye out for more posts around the role legal counsel plays in IR. If you are interested in partnering with Rapid7 to help you develop a robust incident response plan at your organization, check out our incident response services.


This is part 2 of a 2-part blog series on how Incident Response is changing. Here's part one.


The changing threat landscape forced an evolution in incident detection & response (IDR) that encompasses changes in tools, process, and people. While in 2005 we could get away with basic detection and a “pave and re-image” approach, 2016 sees us needing complex detection methodologies enabled by powerful software and hardware to enable experts to drive the IDR lifecycle.


One of my go-to analogies to help people understand the need to evolve cyber IDR programs is to point to the evolution of banks (yes, the paper money kind!). in the early days, you just needed a bandana to hide your identity and a six-shooter to be handed the cash. Banks could reasonably protect money using basic safes, little physical protection, and a tough sheriff in town. Today, the serious bank robber is an expert in electronics, locks, and deception. Banks have evolved to create layers upon layers of physical security, monitoring, alarms, and SWAT teams to help protect the valuables. That is exactly what happened between 2005 and 2016 in the IDR space, but we need to continue to evolve.


When we look at the areas of importance for modern IDR programs, we speak of: preparation, detection, validation, response, containment, and recovery. Let’s now examine these based on our 2005-to-2016 timeline.


Breach preparation in 2005 vs. 2016

In 2005, very little was done to proactively secure the attack surface as the focus was mostly on availability of systems and speeding business processes. Very few organizations proactively conducted breach response exercises, and even fewer thought that a cyber attack against them was a concern.

In 2016, the best IDR programs have:

  • Implemented defense in depth principles in their network infrastructure
  • Cataloged, organized, and restricted their data following least privilege principles
  • Actively managed their exposure through attack surface management
  • Rehearse technical, coordination, and communication aspects of breach detection and response


Incident detection and validation in 2005 vs. 2016

The nature of the changes in incident detection and validation come from the changes in the number of organizations processing data of value to attackers, the expanding number of determine attackers, and the motivations behind the breaches. In 2005, most organizations didn’t have to worry much about being the victim of a targeted breach using unknown vulnerabilities, malware, and tools.

In 2016, you don’t even need to have valuable data to experience a targeted breach, you just need to promote ideologies that are different than the bad guys’. In 2016, the best IDR programs have:

  • Technology to detect threats across the entire ecosystem from the endpoint (including mobile devices) to the cloud based services and applications outside of the network walls
  • Technology to validate machine driven threat detection on the endpoint and the network
  • Threat detection methodologies that include tailored and timely threat intelligence, behavior analytics, and data analytics
  • Subject matter expertise that covers attacker methodology, malware analysis, endpoint analysis, network analysis, data visualization, and automation
  • Defined and rehearsed processes for threat detection, validation, and escalation
  • Metrics to measure and improve performance


Breach response in 2005 vs. 2016

One of the best aspects of breach response is that there is no typical day at the office. Attackers are constantly learning new techniques, creating new tools, devising new ways of persisting in victim environments. As such, incident response is in a constant state of change as investigative techniques adapt to attacker techniques.

In 2016, the best IDR programs have:

  • Technology that enables detailed analysis of endpoint, network, and logs
  • Subject matter expertise that covers forensics analysis, incident management, incident coordination, and incident communications (in addition to the expertise from the incident detection and validation section)
  • A breach response plan that covers all aspects of response activities ranging from technical analysis to restoring normal business operating processes


Breach containment and recovery in 2005 vs. 2016

The last, and least prepared for, aspect of IDR is how quickly can you contain a breach and restore normal business operating processes. In 2005, computers were a critical component of business, but today, they’re a critical component of our entire lives. In 2005, containing a threat most often required removing a system from the network and recovery was difficult due to storage capacities and older technology.

In 2016, containing a threat can be done remotely, help desk teams have system imagine processes and data restoration processes that can return a user’s machine in a day. In 2016, the best IDR programs have:

  • Technology to contain threats at the endpoint, on the network, and in the identity management system
  • System imaging and data backup/restore processes for servers and workstations
  • A tested and rehearsed disaster recovery program


The IDR industry is no different from any other industry (including law enforcement) aimed at thwarting criminals. We evolve based on the threat that we are facing. Today that threat has a massive footprint, cutting edge technology, and the ability to adapt to any challenge they encounter. As such, our approach to IDR must implement programs that are flexible and adaptive to keep up.

While everyone in the security world is seemingly at RSA Conference, my mind has been searching through the past. It actually started a few weeks ago, when Gartner’s Anton Chuvakin asked for examples of how today’s Incident Detection & Response (IDR) is different from 2005. My short comment to his post started to explore the topic of change over the past decade of IR, but I kept thinking about it more and it struck me that there was much more to discuss. The fact is, most security programs are still stuck in 2005 and that many people outside of the industry still think about threats as if it was 2005! In order to start getting away from IDR of the last decade, we must first examine differences in the threat landscape.


Landscape: Threats in 2005 vs. 2016

Threats really haven’t evolved all that much, that’s not much of a surprise. Attackers are still motivated by financial gain and destructive actions. They target organizations or people who have data that can be sold and organizations with views contrary to theirs. The big difference between 2005 and 2016 is the volume of breaches and amount of stolen data.


Landscape: Data in 2005 vs. 2016

The changes that catapulted the evolution of threats and the volume of breaches is the amount of data and its value. In 2005, there was a lot of sensitive data in computers, but nothing like what we see now. Further, in 2005, the black market primarily demanded financial data, but today intellectual property, health data, personal information, and computing resources can generate significant income for attackers.


Landscape: Computers in 2005 vs. 2016

The last aspect of the changing threat landscape is the attack surface itself; the computing devices. We’ve exponentially increased the number of connected devices and computing power since 2005. Creating malware has become easier, vulnerability data is open source, and testing against common threat prevention and detection technology is offered as a public service.


In the 2nd part of this post, which I'll post tomorrow, I'll explore how this changing threat landscape has evolved IDR practices.

This post is the final in a series examining the roles of search and analytics in the incident-detection-to-response lifecycle. To read the previous six, click one, two, three, four, five, or six.


As a final discussion topic around search and analytics, it is really important to talk about the different teams involved, who all have different questions to ask of the same overall data. Roles and titles can vary greatly, but for simplicity, I’ll refer to the three primary groups as ‘IT Ops’ [for anyone looking to maintain servers and software required for the employees and partners of the organization], ‘DevOps’ [for anyone focused on the availability and quality of servers and software for external customers], and ‘security’ [for the individuals looking to prevent and detect malicious activities]. Let’s talk about the order in which these teams generally develop in an organization and how that leads to politics between teams and the duplication of efforts, using the now-classic Lethal Weapon series.roger-murtaugh.jpg


IT operations monitored logs first, but not last

For our purposes here, I’m going to consider the IT Ops team the Roger Murtaugh of the company because they were likely there first, so they were around to acquire the necessary tools and lay out “the way things are done”. When the first email servers were stood up and various infrastructure deployed, IT Ops were manually checking the health of each system and receiving floods of automated emails with complex error codes when the slightest problem inevitably occurred. This was never an easy job, but the gradual addition of ticketing systems, helplines, email distribution lists, and pagers to their processes made it possible to handle the day-to-day needs and off-hours emergencies. Centralized log management solutions were then brought in to simplify the entire monitoring and troubleshooting process, and the first automated trend charts around CPU usage, errors, and bandwidth made drastic improvements on initial notifications. Troubleshooting could finally be done from a single place, so IT Ops was happy with the way things were working.


lethal-weapon-frightened-riggs.jpegIf your organization has custom-developed software, as many do today, at least one DevOps team has gotten sick of similarly SSHing into production systems to verify a new release is functional or track down a bug. For the IT Ops team, realizing this new team is looking to implement a log management solution was probably a similar shock to when Murtaugh first learned he had to work with that loose cannon, Riggs, who did things his own way. Given that change can be scary, providing high-level permissions to access your prized tools and letting another team do things “their way” both feel inherently risky. Even once these teams come to an agreement to either share a solution or work independently, they are typically minimizing very specific risks to the business with little thought to the risks with which the security team must obsess.


You're mistaken if you've assumed no one else needs your logs

One of my favorite cognitive biases to consider, because we can all rarely avoid it, is the "focusing illusion" best summarized as "nothing in life is as important as you think it is while you are thinking about it". This is relevant here because when you are paid, trained, and thanked for using the tools and data at your disposal for very clear purposes, you are unlikely to realize just how much value is in there for others with very different questions or that sometimes those other questions could be more important to the company than your own. Whether or not Murtaugh and Riggs in your organization have become partners, they most likely see security as a pest [like Leo Getz] or like Internal Affairs [Riggs's future wife, Lorna Cole] because the effort to reduce the risk of attacks and data leaks often requires very different efforts than keeping applications functional and available for customers, be they internal or external. Meanwhile, in reality, just as all of these characters had the same goal [of stopping serious criminals], the security team has the same overarching goal as IT Ops and DevOps: eliminating reasons the rest of the company would be unable to accomplish its goals [and yes, I remember that Leo Getz was originally a money-launderer]. If you're in IT Ops or DevOps, your security team wants you to know that they would really like access to your logs. It would help them reduce the risks they worry about, even if you don't completely understand why.Leo-getz.jpg


It’s healthier not to differentiate between “logs” and “security logs”

Microsoft may have biased us all by labeling some logs as "security logs" and others for different purposes, but the easiest example of non-security logs with value to security are DHCP logs; if your security team doesn't have access to DHCP lease grants and releases, it can be very difficult to find out which machine is operating at any given IP address in an incident investigation. The same goes for machine data the DevOps team considers only relevant to them - if you can see who made the last change to the production servers before they went down, you can also see whose account might have been stolen and used to change settings preventing outside access to production systems. If you never ask the security team if your logs would be useful, you are likely to make the wrong assumptions. Besides, they should absolutely have the security tools capable of ignoring anything they don't need.


Getting a single solution for all parties to use can save you a lot

Though the security team should have the tools that help them find what they care about in your logs, far too many organizations have three different tools for three different teams when there could be a single one. You don't need pivot tables or fancy finance visualizations to realize that buying a log management solution for IT Ops, a separate search solution for DevOps, and also a SIEM for the security team is a very costly way to solve different problems with the same data. Even if you forget about the three different purchases, failing to take advantage of discounted tiers of pricing, you can see that three different groups have to manage three different solutions focused on search and analytics which only differ by the questions they ask and use cases to which they apply. There may have been technology challenges preventing this before, but you should demand better now. Ask for real-time search and analytics flexible enough to satisfy Murtaugh's, Riggs's, Getz's, and Cole's problems. You all want to analyze your data to remove anything which could possibly hinder your organization's progress.

If you are a part of the IT Ops or DevOps teams, you can start a free trial of Logentries here and search your data within seconds.
If you are on the security side of things, check out an InsightIDR demo.

This post is the penultimate in a series examining the roles of search and analytics in the incident-detection-to-response lifecycle. To read the first five, click here, here, here, here, and here.


shovel-pan.JPGAs the amount of data within our organizations continues to grow at a nearly exponential rate, we are forced to look for more efficient ways to process it all for security use cases. If we want to keep up as these problems and systems scale, we need to learn from industries much older than computing. For example, gold miners have graduated from just a shovel and pan to automating as much of the process as possible, reducing the manual labor to the areas machines just cannot handle. From the beginning of the data analysis process, there are four pieces of your security data processing which can be automated and a fifth you need to have human eyes and minds to handle.


Automate the collection from all potentially valuable data deposits

pale-rider-large-nugget.jpegIf you were a fan of either Pale Rider in the 80s or White Fang in the 90s, you can probably recall the way gold was mined for centuries. Once someone decided they had a potential gold deposit, pick axes and shovels were popular tools to free the gold particles from the surrounding rock. Using blunt instruments in very specific places is quite similar to the original process to debug software issues and look for malicious activity by logging in to a specific system and searching through the logs. In recent years, over a dozen stages of crushing rock, pumping water, applying pressure, and various other methods of filtration have been introduced to reduce the need for luck when “digging for gold”. Thankfully, for every security team’s sanity, data prospecting from system to system was similarly seen as too laborious to scale around ten years ago and any relevant data is now centralized. You should demand a software solution which offers multiple options for data collection (read: not just a syslog feed) and allows you to monitor your data sources for inactivity or sudden failures in collection.


Automate the normalization of data

I won’t force the analogy into every section because reducing large chunks of rock down to just the valuable minerals doesn't involve the understanding of the rock and addition of other rock in the way more data must be used to help interpret data. In order to simplify the later processing of the raw data, you must address an area less frequently automated by off-the-shelf software solutions: normalization. Many solutions put the onus on the customer to configure connectors, match specific fields in their logs to the “normalized” data field, and maintain this parsing and translation as logs change over time and new devices are deployed. There is no reason this shouldn’t be automated for the vast majority of data sources common to organizations today. Your team shouldn’t have to deal with these manual tasks unless you are working to monitor your own internal systems through custom logs.


Automate the internal attribution of activity

After the data is normalized, a massive amount of time is generally spent figuring out the actions that were actually taken, and it shouldn't be so time-consuming. If your automation stops there, it takes a great deal of working knowledge just to take the normalized data correlated with other information by a single shared attribute: concurrency. Your team shouldn’t have to first identify a specific log line or other data point as being of interest, and then conduct a few more queries or “pivots” to other data before you can determine which user was responsible for the action and on which asset. This attribution should be automatically performed as the data is ingested. Why would you ever want to look at the data without knowing these details?


Automate the behavioral analysis

The last stage of automation you should demand of your monitoring software is the explanation of the behavior. What did the user actually do on the asset? You shouldn’t have to diligently decipher these details every time you investigate the same kind of activity in your machine data. Without the context of the amount of data transmitted externally or the destination organization, a lot of time can be spent simply to find out your code repository vendor now has a few new public IP addresses. And as long as the previous stage is automated, you can immediately see that a software developer transmitted this sudden increase in data from her primary machine. These details should all be obtainable at a glance. Critical thinking should be reserved for determination of intent and recognition of new, questionable behavior.


gold-panning.jpgClick here to learn more about User Behavior Analytics.


Ease the manual analysis with usability, visualizations, and flexible search

Once you have automated these first four stages of data analysis, the security team can spend a lot more of its time deciding whether the activity is malicious and what should be done about it. It's like the process of panning the loose surface sediment in hopes of leaving nothing but the gold and other high specific gravity materials for manual review. It doesn't scale to only perform these actions all day because you don’t know where to look in the data, but it is very effective as a complement to the automated analysis. With the four stages of automation, you can already have enough direction and context to know where you’re looking before taking this last action and planning remediation. By pairing the automation with data visualizations and rapid search capabilities, you can make this final stage as painless and quick as possible for your team to act with confidence.


If your team needs to use the extensive log, and other machine, data in your organization to effectively detect attackers as they laterally move from initial compromise to multiple endpoints and, eventually, the systems containing the most valuable data, you should not be forced to build every stage of the processing yourselves like in the old days.


If you want to learn Rapid7’s approach to automating the first 80%, check out an InsightIDR demo. If the last 20% of manual effort is your challenge, you can start a free trial of Logentries here and search your data within seconds.

This post is the fourth in a series examining the roles of search and analytics in the incident-detection-to-response lifecycle. To read the first three, click here, here, and here.


Nearly a year ago, I likened the incident handling process to continuous flow manufacturing to help focus on the two major bottlenecks: alert triage and incident analysis. At the time of these posts, most incident response teams were just starting to find value amid the buzz of “security analytics,” so I focused on how analytics could help those struggling with search alone. As more teams have brought the two capabilities together, the best practices for balancing them to smooth the process have started to become apparent, so here is what we've heard thus far.


Analytics are meant to replace slow and manual search—but not everywhere

ace-ventura-sliding-door.pngYou need to be able to use search as if you’re zooming in to find the supporting details for your initial conclusion. It would be great if we could solve every incident like Ace Ventura deducing that Roger Podacter was murdered by remembering a string of seemingly unrelated data points, but as Don Norman explained in The Design of Everyday Things, the limitations of the human working memory demand better designed tools to balance the trade-off between “knowledge in the head” and “knowledge in the world”. Since investigating an incident involves a great deal of information be gathered, “knowledge in the world” must be maximized through analytics and effective visualizations for the context they bring to minimize the necessary “knowledge in the head”.


Search cannot be your only tool for alert triage. Search results provide immediate "knowledge in the head", but little else. They are great for answering clear and pointed questions, but there is not enough context in search results alone. Receiving an alert is the beginning of the incident handling process and if the alert is nothing more than a log line, triage can be a gut feeling based only on previous experience with the specific alert triggered. Sometimes, seeing the log line in the context of the events immediately preceding and following it is enough to understand what occurred, but often, higher level questions need to be asked, such as “who did it?”, “on which asset?”, and “is this common for that person and system?” None of these questions immediately explain the root cause of the triggered alert, but they make it exponentially easier to triage. If you need to manually search your machine data for the answer to every one of these questions to triage every alert, you need to provide your team with a great deal more “knowledge in the world”. This should include manicured views containing the answer to most of these questions on display next to the alert, or a simple click away.


If you are analyzing dozens of incidents per day, every repetitive action is building the bottleneck. This is where analytics play a part in the incident handling process, as they only add value if the conclusion they make is easy to understand. Dozens to hundreds (and hopefully not thousands) of alerts per day require a progression of quick conclusions which others have made in the past before finding a very specific answer to the incident at hand. Easy-to-understand analytics are the answers to the many early questions you need to ask to before you understand the root cause of the alert and decide whether it was either malicious or an action which should be permitted in the future. If the analyst performing triage and the incident analyst (who may be the same person) have access to these analytics as soon as they receive the potential incident in their queue, it vastly reduces the amount of manual searching through data they must perform to close the incident faster than previously possible.


It hurts the flow when you cannot immediately access the data you need

clue-board-game.jpgIn the previous post of this series, I covered the need to collect data outside (and inside) the traditional perimeter. An adjacent problem is the situations in which the data is being collected but not readily made available to the IR team. Needing to wait for the data to be restored or indexed is a bit like if you were playing Clue with your family and every time someone said "It was Colonel Mustard, in the conservatory, with the candlestick", you all took out your phones and caught up on emails for thirty minutes as the player-to-the-left searched through filing cabinets for her cards disproving the suggestion.


Expensive tools and verbose data sources have led too many teams to delay investigation until the data is acquired for indexing. The first common cause of this challenge is caused by the realities of departmental budgets, which for the majority of teams to pick and choose the data they make unconditionally available to search. Sadly, this leads many teams to choose between firewall, DNS, or web proxy data to have immediately searchable for incident responders, storing the rest on the device itself or pushing it to cold storage. Then, when an incident is deemed severe enough, the team is waiting hours to restore (or forward) the data and get it indexed for search. This sounds crazy to anyone who hasn't had to stay under a budget, but these three devices are logging so much noise in a typical day that it becomes the only perceived sacrifice the team can make. Your team shouldn't have to worry about exceeding data thresholds or adding more members to your cluster just to have this key data constantly available.


Setting reminders to search the data once it's available severely challenges the "without interruption" aspect of continuous flow. The second guaranteed "data loading..." way to slow your team's response to an alert is using technology that takes up to twenty minutes to make it searchable. If receiving an email alert and copying it to your calendar to remind you to investigate in twenty to thirty minutes doesn't sound like the best use of your time, you are likely to be frustrated by the inability to search at the moment an incident has your attention. Unless you're the one incident responder who never has enough to do, you'll probably be busy looking into something else by that time, meaning an even longer gap before the initial alert is investigated, or worse, it gets forgotten altogether until you or someone else on the team reviews the open incidents hours later.


Everything from the query language to the search results needs to be designed for junior analysts

No matter the maturity of your team, the best search and analytics capabilities can offer little value if they don't account for a single truth: you have to hire and train entry-level analysts just as any technical team does. If every tool at your team's disposal is designed with a focus only on the type of data it obtains and not on the usability, your junior analysts have steep learning curves to climb before they are contributing to the efficiency of the entire process. This is not the new hires' faults. If you need to learn a complex query language, become and expert on each data type, or decipher the search result screens which perfectly depict "information overload" before you can contribute to the team, you need to get more comfortable blaming the software at your disposal. Going back to Don Norman, we have all become too accustomed to blaming the user when the design is at fault. Every software solution (for search or otherwise) your team uses needs to be designed for an entry-level analyst.


If searching through machine data is your current need, you can start a free trial of Logentries here and search your data within seconds. If you want to learn more about building your incident response capabilities, check out our incident response toolkit.

This post is the third in a series examining the roles of search and analytics in the incident-detection-to-response lifecycle. To read the first two, click here and here.


In the second blog of this series, I touched on the need for solutions more flexible than the traditional SIEM architecture focused primarily on receiving logs from the security appliances in your infrastructure. This wasn’t only a passing comment; in the past five years, the approach to compromising an organization and corresponding work to defend against it have changed significantly as new technologies have emerged for use by both sides.


Attackers get in by a variety of perimeter-blind routes, which means an expansion of the data sets security teams need to interpret

Whether you use the Verizon Data Breach Investigations Report as your source of truth or rely on anecdotal evidence from the details which gradually emerge, it is clear that web application attacks and stolen credentials comprise the majority of the successful entry points in recent years. This is especially frightening for companies using perimeter security devices as their only source of prevention and/or detection. Not only are they not watching the right entry points for most of today's attacks, but even if they possess the tools and knowledge to effectively analyze it, they likely have no access to the data which would help them detect these attacks.


The unfortunate reality for security teams today is that they cannot simply forget about perimeter devices and dedicate months of their lives to overhauling their entire infrastructure while learning the most advanced practices for mitigating these primary attack vectors. Using firewalls, intrusion detection/prevention systems, and two-factor authentication on VPNs should be considered a starting point for security teams. Removing these tools from the equation would mean initial compromise techniques currently with shrinking success rates would reclaim the top of the list. This gradual evolution of intrusion techniques forces the need for a continually expanding breadth of knowledge on the security team and more direct access to other teams' knowledge when an incident warrants it.


Tracking lateral movement is near-impossible if you only have data on centralized servers

blue-streak-border.jpgUnless you expose your servers with the most sensitive [read: monetizable] information to the internet, the attackers are not going to stop after the first compromise. Even in the most poorly secured environments, it would be rare to store the data most critical to the business in a web application on the network's edge. Once the initial compromise is successful, there are many ways to pivot to other hosts, once inside. Harvesting credentials for use in moving from system to system is a popular method of stealthy reconnaissance as more is discovered, and it can be done with polymorphic malware to evade known-hash detection or with any number of attacker-at-keyboard toolkits. This presents a massive challenge for investigations in the vast majority of organizations who are unable to monitor the wealth of information isolated on their endpoints.


As if it weren't a large enough challenge to monitor every endpoint on the network, and at your employees' homes, an attacker can also move laterally to one of your organization's cloud environments, whether it hosts employee data, customer data, or other valuable data like source code. Segmenting valuable data across distinct cloud services is a great way for modern enterprises to reduce the impact of a compromise, but they all need to be monitored and secured the same as you would any area of your physical network. Otherwise, analyzing an incident without activity from your managed clouds can feel like the scene at the end of Blue Streak when the police are powerless to follow Miles any further because he has crossed the border into Mexico. No portion of your environment, even virtual and hosted infrastructures, should be just beyond the reach of your security team.


You need to analyze machine data from inside and outside the traditional perimeter to effectively investigate today’s attacks

All of this seems like a daunting assignment because it is. If you don't have the benefit of a large team who understands the structured and unstructured data across your environment and the resources to automate the correlation and internal attribution across all of these data sources, your team is going to spend a significant number of hours duplicating manual efforts to analyze the incidents you detect. There will be incidents closed in minutes because of their familiar feel to a senior analyst and there will be incidents which take junior analysts days to effectively investigate, but there will be little time for other duties outside of alert triage or incident analysis.


disparate-search.jpgIf you have a different user interface for your perimeter devices, malware detection, endpoint detection, and cloud monitoring, you likely spend a lot of time switching between monitors and browser tabs to look for a string to tie events together. Too much of this has become a cascade of similar searches across unlike data sets. The ability to search will always be a necessity [as I wrote in the first post of this series], but siloed search is too manual and slow to stand alone as an investigative tool, especially if it means different search capabilities in different places. Search cannot be an afterthought. Each data source can provide valuable context to search results after analytics have been applied.


The paired solution needs to be a single place to pivot through relevant data from your entire environment. This was once dubbed a "single pane of glass" and today, it is at the center of what analytics solutions now aim to offer incident responders. If the data is structured and understood, speed analysis by automating the classification of data as user or system behavior. Detection and analysis solutions from other security vendors need to be a source of more context, rather than another console for independent analysis. If classification is currently a challenge, allow easy exploration through search. One place to send the breadth of data. One place to access a mix of automatically-drawn conclusions and conclusions requiring a human mind to interpret.


Click here to learn more about User Behavior Analytics.


If you want to learn more about building your incident response capabilities, check out our incident response toolkit. If searching through machine data is your current need, you can start a free trial of Logentries here and search your data within seconds.

This post is the second in a series examining the roles of search and analytics in the incident-detection-to-response lifecycle. To read the previous, click here.


Various security vendors have made very public declarations claiming everything from “SIEM is dead.” to asking if it has merely “lost its magic”. Whatever your stance on SIEM, what’s important to recognize is that while technologies may fail to solve a problem, this doesn’t make the problem any less serious or prevalent.


The debate over SIEM’s demise is a distraction

SIEM’s supposed magical life is unlikely to suddenly end for another decade because of the time it takes for the momentum for the industry’s largest investment to fade. Despite this impressive market life expectancy, former SIEM vendors are clamoring to pile on and replace the dead SIEM 1.0 with their SIEM 2.0, or even SIEM 3.0. But the only reason these statements have made such an impact is the wide-ranging expectations of those engaged. Teams properly equipped to take the blank slate that often qualifies as SIEM technology and build an effective incident handling process on top of it realize its value. The other 95% of the security teams who invest the majority of their annual budget in simply deploying a SIEM continue turning it into shelfware to dust off whenever a high-risk audit appears in the calendar.



Unlike Jumanji, where a single toy automagically made jungles and creatures appear, SIEM solutions generate magic not through the technology, but through the people customizing and using it. I’ll skip the obvious Harry Potter reference here to avoid insulting any SIEM engineers who don’t like fantasy films, but if you cannot afford to hire a team of these wizards, you are doomed to carry on with the same low ROI as the other organizations out there that receive hundreds of thousands of alerts per day and day-long efforts to construct all of the complex queries necessary to answer enough questions to close an incident investigation.


These important problems are not close to disintegrating

If you ask the wrong people, SIEM solved security in 2006. Realistically, though, it solved a lot of the leading challenges  security and IT faced at the time by putting the historical data in one place. Centralization of logs made it possible for IT, networking, and security teams to address their biggest problem: accessing the data from their most critical servers and networking devices in a single place to troubleshoot and identify any issues quickly. Never before could all teams who need to investigate outages, software errors, and security incidents and even those who needed to answer every auditors’ questions all go to a single place to do so within just a couple of days of digging through the data.


However, despite having done more to address these problems than any preceding products, the biggest reason SIEM’s magic death is being so heavily discussed is its failure to consistently solve the most challenging of them: all cyber compromised detection needs in a “single pane of glass”. Again, the wizards out there have to manage, with the help of data scientists joining the team, to build some very impressive SIEM-based detection cores for their incident response armies, but the vast majority of organizations bounce from an IPS management console to a SIEM solution for some more details before obtaining more details from an endpoint forensics solution and pushing them through an unstructured data search solution to answer the final questions necessary to respond appropriately to the incident in question.


SIEM is rapidly losing trust because of its inability to adapt and aforementioned dependence on internal experts

There is absolutely a wealth of valuable data in even the least-used SIEM solution deployed today, but these largely outdated software solutions were designed to operate in an environment significantly different from today’s. When organizations ran all third-party software and internally developed software on physical servers they either housed inside their offices or hosted in a rented space, there was an illusion of control because of the option to pull the physical power plug from the systems. However, thanks to companies like Google and Amazon, no successful company today still operates in that version of reality. Ten-year old solutions should not be expected to efficiently adapt to the modern need for elastic cloud computing and constant access to the corporate network via mobile devices.



Just as we learned from Days of Thunder, even the most effective driver (or incident responder) cannot perform with technology meant for a different environment. When Cole Trickle first moved from open-wheel vehicles to stock cars, he had to swallow his pride and let Harry teach him how to use the phenomenal car he’d built for him. Now, I’m no stock car racing expert [I’m not even a passing fan], but I do know I’m not insulting SIEM solutions with this analogy. You cannot just make some minor modifications to a stock car and compete in an open-wheel race, just as you cannot simply start piping data from your cloud and mobile management systems into your pre-existing SIEM server(s). You can invest in more hardware, hire a larger team, and even build a complex ETL pipeline to get your data into more modern data stores leveraging Hadoop [buzzword!], but the truth is, as you fall victim to this sunk cost fallacy, you’ll spend insanely large sums of money trying to realize the value you could get by other means for a fraction of the cost.


Modern flexible solutions built for the security problems are needed to address these challenges

If you want to get back to addressing the problems SIEM was meant to solve, but do so in the modern landscape, you need to switch to search and analytics solutions built solely to address these very problems of one place to search all data, detect and investigate attacks in this very environment in which we live today. You could even reduce the time your team regularly spends maintaining the hardware and custom software enough to focus more on the day-to-day questions they need to answer.


Are you building in-house software? That’s a facetious question. Everyone is. If you want to deploy a solution now designed to adapt as technology evolves, you need it to be immensely flexible to the machine data it ingests. If the goal is simple log management and search, you don’t need an overly complex solution bolted onto an existing SIEM. You need flexible machine data search.


If your goal is simplifying how your team automates the alerts from your many complex detection technologies so you can spend more and more time hunting for new threats, you need behavioral analysis designed for the continually growing data generated by the people and systems in your organization. If you need to analyze incidents across your endpoints and managed cloud environments, you need an analytics solution designed solely for today’s incident response teams.


If the problems described here are similar to yours, Rapid7 has a number of Incident Detection & Response solutions that you can read about here. They are continuing to adapt as technology and the attackers do.

This post is the first in a series examining the roles of search and analytics in the incident-detection-to-response lifecycle.


Strong data analytics have recently enabled security teams to simplify and speed incident detection and investigation, but at some point of every incident investigation, a search through machine data is nearly always necessary to answer a one-time question before the investigation can be closed. Whether your incident response team is just trying to combat the flood of alerts coming from a series of noisy detection appliances or seasoned specialists who have automated a sizable amount of data enrichment to focus on manually hunting the sophisticated, there are many reasons you need the ability to search through disparately sourced machine data for answers.


Attackers try for easy, then adapt and iterate

For those who mostly hear about massive data breaches via the major media outlets prior to the substantive details being publicized, hindsight bias brings a great deal of thoughts like "they should have known to watch their medical record database more closely" and "how could they have not watched the applications on POS devices better?", but this is drastically over-simplifying the attacks, themselves. Every attack plays out as the intruders adjust to the preventive controls in place. Stealing partner credentials was likely not the first technique attempted for initially gaining access to the Target network, but it was the first to work. Though the initial compromise typically occurs via either stolen credentials or web application exploit, the next step taken will be the same isolated trial-and-error process of trying some basics like harvesting more credentials and scanning for potential new hosts or a five year-old remote exploit, yet it is guided by what works in that one moment in time.



If you've watched Braveheart as many times as I have [and you probably haven't], you may see the similarities between its [historically inaccurate] battle tactics and a cyber-attack. Whether it was luring a large number of soldiers into a gully before revealing your army on the cliff above, pretending to have no plan for the English cavalry, or a gentlemen's agreement with the onrushing Irish, every skirmish was started and progressed in a different way. Being crafty doesn't always lead to success, but it leads to a lot more success than quitting after trying the easiest tactic and failing.


This unpredictable flow of every attempted compromise is why organizations so often don't know what data they should be collecting for investigative reasons. There will always be high value data sources in organizations which provide necessary context for detection and investigation, but they will have limitations when attackers adapt to what they successfully compromise. Analytics just won't help explore the data which may only be relevant to an incident investigation one time.


Every organization and corresponding environment is unique

identifying-your-blindspots.jpgIt is frequently underestimated just how organically an every network grows, but as evidenced by the recent report on the IRS feeling lost to upgrade its Windows systems, it is extremely challenging just to identify every endpoint and server with access to your network. Add to that just how easy it is for software developers to spin up a small cloud environment or a financial analyst to take work home on an iPad, and you start to see the attack surface expand rapidly.


If you are limiting the data you collect for monitoring and investigation to what is easily centralized or on the managed perimeter, your incident response team is forced to close a great deal of incoming alerts and follow-up investigations without the high level of confidence they would prefer. They simply don't have enough of the data they would need to know what happened.


Custom application logs and unstructured evidentiary data are needed for confident incident analysis

Of even greater concern than the large blind spots in the structured data is the unstructured data often unavailable to security teams. If your organization has a great deal of internally-developed applications, as most modern businesses do, and you're only monitoring the logs from your firewall, domain controller, and web proxy, you'll be blind to an attacker moving from server to server in your production environment, looking for some data to monetize. For those who think this is rare, our Logentries team estimates that thirty percent of the logs we process for our customers are completely custom. In these scenarios in which the relevant data to completing incident analysis could be unstructured and unique to your company, your security team needs the ability to search it quickly for data points which tie it to the broader investigation. Inflexible log management solutions are of no help here.


Understanding the root cause can take many forms

What incident response teams rarely say, but typically know, is that the major reason incident management is so difficult is that you're using machine data generated as a result of some actions and working to determine exactly what an actor, be it human or software, did to cause this resulting data generation. Then, you not only have to understand the behavior, but also have a high enough level of confidence to explain the actor's intentions to either take action (when malicious) or close the investigation (when benign). Every person and team has its own comfort level, but the process to reach it is different for every type of initial behavior and too often isn't reached because of insufficient data or the inability to properly explore it in a timely fashion.


If you really want to understand the root cause and close your investigations at a pace to avoid alert fatigue, you need the context around user and endpoint behavior you can only obtain from an analytics solution plus the flexibility and ease of machine data search solutions. You cannot limit yourself to one or the other and maintain that high level of confidence. If this sounds interesting to you, we do have a number of Incident Detection & Response solutions that you can read about here.

security_awareness_phishing11x17.jpgThe Internet is full of articles for how to tell if an email is phishing but there seems to be a lack of concise checklists how to prepare an organization against phishing attacks, so here you go.


Because phishing attacks humans and systems alike, the defense should also cover both aspects. None of the following steps is bullet proof, so layering your defenses is important – and having an incident response plan in case someone does get through.


Here are my recommendations on how to defend against phishing attacks:


1. Filter emails for phishing threats

It’s important that you filter your emails for malicious URLs and attachments to prevent phishing emails making it to your users in the first place. Sandboxing can detect a lot of the malware in emails, but make sure that you have a follow up plan in place if you’re deploying this technology in detection rather than blocking mode – otherwise the malware is still live on your systems. Use security analytics to filter out malicious URLs. Rapid7 UserInsight uses threat feeds to detect known malicious URLs and security analytics to alert on unknown ones. It also integrates with sandboxing solutions, such as FireEye NX Series and PaloAlto WildFire, to enable quick and easy incident investigation of malware alerts.


2. Update client-side operating systems, software, and plug-ins

Some phishing emails include URLs to exploit vulnerabilities in the browsers and its plug-ins, such as Flash and Java; others send file attachments that try to exploit applications like Adobe Acrobat or Microsoft Office. That’s why it’s important to patch vulnerabilities on your endpoints as well. Many organizations already have a vulnerability management program in place but only scan servers. Make sure you extend coverage to your endpoints and patch operating systems, software, and plug-ins. This not only protects you from phishing emails but also drive-by attacks. Rapid7 Nexpose can help you manage vulnerabilities on your endpoints, and much more.


3. Harden Your Clients

Lock down your clients as much as possible. This includes things like not making your users local administrators and deploying mitigation tools like Microsoft EMET (check out this Whiteboard Wednesday on EMET on how to deploy this free tool). Rapid7 Nexpose Ultimate includes Controls Effectiveness Testing, which helps you scan your clients and guides you through the steps to harden them against phishing and other attacks.


4. Block Internet-bound SMB and Kerberos traffic

One of our penetration testing team’s favorites is to use an SMB authentication attack. In this scenario, the attacker sets up an SMB service on the Internet and sends a phishing email with a URL or Word document that references an image through file:// rather than http://. This tricks the computer to authenticate with the domain credentials to the SMB service, providing the attacker with a user name and password hash. The hash can then be cracked or used in pass the hash attacks. To defend against SMB and Kerberos attacks, you should block TCP ports 88, 135, 139, 445 and UDP ports 88, 137, 138 for non-RFC 1918 IP addresses, both on the perimeter and the host-based firewalls. You’ll want to have a process in place to detect compromised credentials, for example Rapid7 UserInsight, which leads us to the next item on our checklist.


5. Detect malware on endpoints

Many phishing attacks involve malware that steal your data or passwords. You should have technology in place to detect malware on the endpoint. Regular anti-virus is great for catching commodity malware, which is likely the bulk of what you will see used against you. There are also many new endpoint detection vendors out there that have great alternative technologies. Rapid7 UserInsight uses its agentless endpoint monitor to collect process hashes from all machines on your network to highlight known malicious processes based on the output of 57 anti-virus scanners; it also looks for rare/unique unsigned processes that may indicate malware.


6. Detect compromised credentials and lateral movement

Even with all of these protections in place, your users may still fall prey to credential harvesting attacks. A common phishing attack is leading users to a fake Outlook Web Access page and asking them to enter their domain credentials to log on, but there are many variations. Once the attackers have the passwords, they can impersonate users. Rapid7 UserInsight can detect compromised credentials, both on your network and in cloud services, such as Office 365, and It detects lateral movement to other users, assets, or to the cloud, so you’ll be able to trace intruders even if they break out of the context of the originally compromised user.


7. Implement 2-factor authentication

Add 2-factor authentication (2FA) to any externally-facing system to stop attackers from using stolen passwords. While Rapid7 doesn’t offer a solution in this space, check out our partners Okta and Duo Security. All systems protected with Okta (Rapid7/Okto Integration Brief) or Duo Security can be monitored with Rapid7 UserInsight to help detect any attempts to use compromised credentials.


8. Enable SPF and DKIM

There are two standards that help determine if an email actually came from the sender domain it claims to detect email spoofing. The first one is the Sender Policy Framework (SPF), which adds an list to your DNS records that includes all servers that are authorized to send mail on your behalf. The second standard is DomainKeys Identified Mail (DKIM), which is a way for an email server to digitally sign all outgoing mail, proving that an email came from a specific domain and was not altered during transportation. Together, they raise the confidence in the authenticity of the sender and email content by the recipient. To help improve security hygiene, check that your systems have both SPF and DKIM enabled on your outgoing email. For incoming email, you should check if a the sender domain has SPF set up and the email came from an authorized server, and that DKIM signed emails have not been tampered with. While these protections are not bullet proof against targeted attacks that register look-alike domains, they can help filter out a lot of mass phishing.


9. Train your employees on security awareness

While even educated users won’t catch everything, they are worth investing in. Train your users about how to detect phishing emails and send them simulated phishing campaigns to test their knowledge. Use the carrot, not the stick: Offer prizes for those that detect phishing emails to create a positive security-aware culture – and extend the bounty from simulated to real phishing emails. Whenever you see new phishing emails targeting your company, alert your employees about them using sample screenshots of the emails with phishy features highlighted. Encourage your users to use secure browsers – I put Google Chrome (64-bit version) on the top of my list for security and usability. Here at Rapid7, we offer Security Awareness Trainings; you can also send phishing simulations with Rapid7 Metasploit Pro that track click-throughs so you can report on user awareness.


10. Have an incident response plan

Even if you put all of these protections in place, some phishing emails will get through, especially if they are targeted against your organization and tailored to the individual. It’s not whether these emails will get through but how well you are prepared to respond to intruders on the network. Rapid7 UserInsight enables you to detect compromised users and investigate intruders that entered the network through a phishing attack. This helps you shorten your time-to-detection and time-to-contain, reducing the impact of a phishing attack on your organization. In addition, Rapid7 offers incident response services and can help you develop an incident response program.


While these areas cover the most important counter-phishing measures, I’d love to hear if you’ve implemented anything else that you found to be effective - just post your experience in the comments section.


If you’re looking at defending against phishing attacks, you may also enjoy my related webcast "You've Been Phished: Detecting and Investigating Phishing Attacks”register now to save a seat to ask questions during the live session.

The recent webcast “Storming the Breach, Part 1: Initial Infection Vector”, with Incident Response experts Wade Woolwine and Mike Scutt sparked so many great questions from our live attendees that we didn't have time to get through all of them! Our presenters took the time to answer additional questions after the fact... so read on for the overflow Q&A on tips and tricks for investigating common breach scenarios:



Q. “What do you recommend for sandboxing the browser?”

A. I’m a huge fan of Sandboxie, which is free for personal use. Sandboxie allows users to launch individual applications within a comprehensive application sandbox, and provides granular control over the application’s access to the underlying operating system. 

In enterprise environments, we often recommend that clients implement Microsoft’s Enhanced Mitigation Experience Toolkit (EMET). IT and security personnel can implement EMET protection for any application via group policy, allowing the implementation of several anti-exploitation techniques at scale. EMET does a fantastic job of mitigating many browser-based exploits, malicious document exploits, and can be tailored to your environment.  We recommend enforcing EMET for web browsers, Flash, Acrobat, and Office/Productivity suites, as they are commonly-targeted platforms. 


Q. “A lot of this information assumes that you know when the malware was dropped in a reasonable window.  For a company that gets contracted to do investigations is there a good way to efficiently identify that window that the malware was dropped?”


A. This is a fantastic question, with a not-so-fantastic answer (though one I give often): it depends.  For this example I will briefly walk through a portion of my analysis methodology for Windows compromises. Note: each investigation is different, so I stress “incident response methodology” and critical thinking when training new analysts. For each incident you respond to, consider the following questions:


“What evidence do I have that indicates the system was compromised?” – Use this as your starting point.  If a user received a pop-up message when logging in, there is likely persistent malware on the system that a tool such as Sysinternal’s AutoRuns utility will help you identify the sample. If responding to a live system, look through volatile data to determine whether there are malicious processes running on the system, unusual DLLs loaded into processes, unusual open handles, and connections to odd or known bad domains/IPs.  Use this data to determine where the malware is on disk or in the registry, and use the associated timestamps from those sources to assist in building a timeline of activity. If the initial alert came from AV / SIEM / IPS alerts, use that time period to temporarily reduce the scope of data that you’re looking at, and focus on identifying newly created files (especially PE files), modified registry keys, and event log entries.


“What is the earliest evidence of compromise on the system?” – Once you’ve identified evidence to support the system being compromised, consider how the system was compromised.  At some point either an attacker or malicious executable code interacted with the system. Work back from your earliest piece of evidence, and timeline logs, file system activity, registry activity, etc, until you’ve identified a potential method of ingress. 

Ingress evidence often takes the form of an initial logon (for interactive attackers moving laterally / transferring files), Java or Flash artifacts from a user visiting a malicious site (or a legitimate site hosting malicious code), a user opening a malicious document, etc. 


These are a few steps that help me find my bearing when responding to a breach, and are not comprehensive.  There are a number of excellent Incident Response and OS internals books on the market that will provide a greater level of detail on Incident Response methodologies.


Focus on answering those questions when initially responding to a compromised system.  It is seldom enough to simply determine whether there is malware on a system, identifying the initial ingress method / root cause of the compromise is critical to ensuring appropriate remediation and mitigation strategies are deployed for the compromised system and all vulnerable systems on the network


Q. “On the web application side, does it help to use IIS IP filtering to prevent malicious attacks from the outside?”

A. IP filtering is useful as a temporary remediation strategy when dealing with a novice attacker, however I do not recommend relying solely on this as a defensive measure to prevent web application compromises.  Many attackers will use proxies such as TOR, other compromised systems, or larger infrastructure during an attack. I recommend following a “defense-in-depth” measure for protecting internet facing systems, to include placing a firewall tailored to your application (i.e. a Web Application Firewall), and an intrusion prevention system between your system and the internet.  The system itself should run a host-based intrusion prevention application, a file integrity monitor, a host-based firewall, and the application should be using the most restrictive permissions possible.  The system should be as isolated as possible from internal systems, and should be monitored carefully for anomalous behavior. Consider the following: if an attacker was attacking your application whilst using multiple TOR IPs and successfully compromised the application – is the application running on the system with elevated privileges? Can the attacker access critical files such as a database connector configuration file?  We frequently respond to web server breaches, and often the underlying applications are running either with elevated privileges or the application user has access to core configuration files.  Once an attacker obtains the configuration files it is trivial for them to connect to backend databases and dump the contents.


Q. “For web servers, do you see specific CMOs/hosting platforms more targeted than others?  If so, is targeting because of popularity or insecurity?  Ex: Wordpress or Joomla.”

A. In opportunistic attacks, absolutely.  Most of the common content management systems on the market today are heavily targeted by automated scanners or opportunistic attackers, as are most of the common plugins. In targeted attacks, we’ve found that it doesn’t matter whether the client is running a well-known CMS or something they’ve developed in-house, as the attackers will diligently reconnoiter the application and identify methods of exploitation.  My mitigation strategies for these attacks remains the same: follow defense-in-depth best practices, patch your CMS and plugins, and ensure that the web application user is running with least-privileges.

Q. “Given that we have to isolate individual machines to mitigate the risk of spreading infection, it doesn't seem feasible to install diagnostic tools or VM platforms on every machine in an office. Would it be more feasible to carry around mobile platforms, like a collection of diagnostic tools stored on USB drives, and are there any risks of the mobile diagnostic tool contracting and spreading that infection?”

A. In these situations, I recommend either using a live-cd with the utilities you need, or purchasing a USB device with a write-block switch.  In large environments, consider creating an isolated VLAN to place the compromised systems on, alongside an analysis machine booted from a Live-CD.  This allows you to connect to the remote machine, perform your data analysis and remediation, and maintain a clean image remotely.  Note: I do not suggest this technique for any work that may be used for litigation purposes. This method is not forensically sound in any way J.


Q. “You mentioned whitelist websites. How to effectively do when sites hosted around the world now. Also with akami and amazon being used to host, IP ranges constentaly change. Would proxy be best or blocked sites/regions via firewall?”


A. Whitelisting is typically only a component of the recommendations we have for mitigating browser based attacks. Specifically only allowing known good sites from the browsers in your enterprise will raise the barrier of entry for attackers. Additionally, we also recommend that customers run an intrusion detection and/or prevention system at the internet egress to alert on potential patterns in traffic that could provide some investigative leads.


Q. “How much of a concern do you have, during these investigations, is there much focus on collecting data in ways that can be used in litigation / prosecution? Or are you mostly concerned with finding breaches and mitigating, etc? Thanks much.”

A. This heavily depends on the client and the type of engagement.  In situations where we know that the data will be used for litigation / prosecution we maintain forensically sound images of the affected systems and follow internal chain of custody protocols.  In situations where it is unlikely that the data will be used outside of the investigation we attempt to avoid impacting the underlying systems as much as possible while collecting.


Q. “If we are using web-based email like Gmail and not Outlook, where else can we look for where the attacks are coming from or other logs inside the registry?”

A. Enterprise Google Apps clients have a wealth of logging for email.  In a situation where a system was suspected of being compromised via a Gmail email we’d correlate timeline data from the user’s browser histories, download histories, file system activity, and Gmail logging.  These data sources will typically provide enough data to determine whether the system was compromised via a malicious email opened within Gmail.


Q. “What should we investigate on a ransomware infection on a PC?”

A. Root cause.  The majority of the ransomware infections we run across are due to the browser rendering a page with an exploit kit that determines what vulnerable plugins and applications the browser is running.  Once you’ve determined when the malware dropped on the system, timeline the file system activity prior to determine whether there were artifacts from Java, Flash, etc. created just prior to the compromise.  Use that information to ensure that the rest of your systems are appropriately patched (and that your browsers are sandboxed / protected by EMET).  Additionally, correlate the timeline data with the user’s browser history to identify the malicious domain if possible, and sinkhole DNS resolution for the domain (at least temporarily) to prevent additional systems from being compromised.


Q. “Do you typically do full forensics for single PC issues or triage with tools like Sysinternals?”

A. This depends on why we’re investigating the system. IF we have indications that an attacker interacted with the system, we’ll typically perform a full forensic investigation to ensure we fully understand the scope of the attacker’s actions. If we’ve simply identified that the attacker attempted to log into the system and need to confirm whether the attempt was successful and whether they interacted with the system we’ll perform a lighter triage wherein we collect targeted data from a few relevant sources.


Q. “Do you have any tips for prioritizing how/where you hunt for malware on a machine that you suspect is infected?”

A. If I don’t have an initial indicator of compromise aside from “Thar be dragons malware!” I employ a few techniques (Windows system – Linux methodology is similar but the particulars vary considerably):

1) Check running processes, and their associated network connections. – here I’m looking for unusual process names, paths, arguments, ports, and remote IPs.  If I’m still struggling I’ll enumerate the process’s open handles, loaded DLLs, etc.

2) Review Prefetch  - if enabled, the Windows Prefetch can help quickly identify unusual executables that have run recently.  Review prefetch files for unusual target paths, unusual accessed files, and last execution time (for timelining).

3) Review persistent binaries – Use a tool such as Mandiant’s Redline or Sysinternals Autoruns to collect a list of persistent samples (items that startup on boot / login / scheduled time).  Look for unsual items by path, name, digital signature status, etc.

4) Review AppCompat /Shimcache – Review the application compatibility cache to get a rough timeline of executables run on the system.  Look for unusual filenames & Paths, as well as “last modified” time.  Note: there are intricacies and caveats involved in using this data.  I recommend reading Mandiant’s whitepaper prior to parsing this data to better understand its use:


Q. “What forensic tools would you recommend for a mobile IR kit? For the field”


1)    Computer repair toolkit

2)    Flash drives (several large + a read-only drive)

3)    Hard drives for imaging

4)    Hard drive imager (we’re big fans of the CRU Ditto – it has night vision mode!)

5)    Flashlight

6)    ESD evidence bags

7)    Small switch and patch cables

8)    Analysis rig with analysis VMs, Encase/FTK/Etc.


It sounds like a lot, but fits nicely into a standard laptop bag with careful organization. 


Q. “Are there any known limitations to discovering malicious activity via behavioral analysis software like CrowdStrike?”

A. There are always limitations when automating malicious activity detection. We always recommend a combination of people and technology for effective threat detection. Additionally, we recommend that our customers use a constant feedback mechanism to ensure that lessons learned through threat detection and incident response are fed back into the technology to reduce false-positives and increase analyst efficiencies.


Q. “Do you find many organizations have SIEM tools to help corrolate the information in the logs when responding to incidents?”

A. Yes, we often find customers using SIEMs. In almost all cases, the SIEM doesn’t ingest logs directly from the endpoint, but rather from select systems and devices (such as Windows domain controllers and critical network devices), which makes the SIEM less relevant during investigations of individual systems. Further, we typically recommend that customers invest in expanding their SIEM deployment to include endpoint logs to improve log based threat detection.


Q. “For crypto ransom ware what will be prevention step from antivirus concern or other....can we identify any encryption process is working during ransomware”

A. In certain situations, a key can be derived through malware analysis and memory analysis, but we typically recommend that customers be prepared for such events by implementing a good system and configuration backup strategy.

To see the full presentation on the expert's guide to investigating common breach scenarios: view the on-demand webinar now.

In the recent Rapid7 webcast, “Storming the Breach, Part 1: Initial Infection Vector”, Incident Response experts Wade Woolwine and Mike Scutt had a technical discussion on investigation methodologies for the 3 most common breach scenarios: spear phishing, browser exploitation, and web server compromise. Their discussion was packed with details and expert tips for investigating these scenarios, so it’s definitely worth the watch, but in the meantime, here are the top 3 takeaways from their discussion:


Time lining is Everything – During an investigation, building out a timeline is crucial.  By building out the chain of attack that was used, you will start to get a better understanding of what may be happening in your environment. The best way to get a good footing is to start from a pivot point, whether that be something like the date an email was sent, in the case of spear phishing, or any piece of data or time stamp that can give a rough idea of when a browser exploitation occurred. If you don’t have this initial pivot point, use some techniques to reduce noise and locate one, like finding malware and working your way back. Use whatever data you can find to piece together what happened right before, during, and after the malware was dropped.


Tools are your friend – There are many tools that can help during an investigation to make your job easier and faster, whether you’re analyzing malware or in the mitigation stage. (Specific tools that help during an investigation are recommended throughout the on-demand webinar.) There are also tools and systems that you can have in place to help prevent and detect attacks on your network. IPS/IDS systems are vital for helping to protect endpoints. Further, make sure to sandbox critical applications known to be targets of attacks, such as email clients, flash, java, adobe acrobat, web browsers, etc., so that if they’re infected by malware, it can’t entrench itself in your system beyond that application.


Limit Admin Access – You can save yourself a lot of headaches, time, money, and more, by limiting user access. Do not allow all users to have admin privileges. There is no reason for the average user to have local admin on their box, and you can easily ensure that users contact a help desk if they need to install additional software. Make sure your users only have the privileges they need to accomplish their job on a day to day basis. Attackers are getting really smart and constantly finding new and interesting ways of hiding themselves, so make sure you’re doing everything in your power to make your systems and users more difficult to attack.


View the on-demand webinar now to get the detailed picture of how experts begin to investigate a breach.


Register for the follow up of this technical discussion, "Storming the Breach, Part 2: Uncovering Attacker Tracks", by visiting the webcast fire pit at Rapid7’s free Security Summer Camp.

We've had a few conversations with our customers recently who have alerted us to extortion attempts against their organizations. Thankfully, none were successful.


This post is to detail the events that have transpired so that you can alert your organizations and increase your odds of not falling victim to this scam:

  •   Attackers will register a domain name similar to yours. For example, the attacker might register when is the legitimate domain
  •   Attackers will target the financial organization while impersonating a executive and requesting that funds be transfered to a bank account
  •   Please note that the attackers are very adept at convincingly carrying on email conversations with their targets


Should this approach fail, or even if it is successful, the attackers might then move to target your customers, vendors, or partners depending on how much information they can obtain through open source research. The sequence of events for the second scam are as follows:

  •   Attackers will use the spoofed domain name to email customer, vendor, or partner invoicing and accounts payable departments informing them that a disastrous event has caused a need to divert payments to another account


To mitigate these threats, we recommend the following:

  •   Implement a verification step in wire transfer processes, invoicing processes, and accounts payable processes to validate the authenticity of the requests
  •   Communicate with your vendors, suppliers, and customers to ensure they know who to contact should they have questions regarding invoices from your organization
  •   Re-emphasize the importance of validating email fields such as "FROM:" to identify spoofed domain names


As always, if you have any questions, please engage your account representative or contact us and we're happy to help.

- @wadew

wbwir.jpgToday, we launched a short Whiteboard Wednesday video aimed at providing a brief overview of how to effectively prepare for an incident. In this post, I'd like to expand on that a little bit by providing some additional concrete steps on how most organizations should be thinking about how preparedness can directly impact incident response program execution during a breach.


The first step is going to involve discovery and data collection. The goal: know what you're protecting and the resources you have at your disposal.

  • Understanding the business - since the end goal of any incident response investigation is to identify remediation activities to restore normal business operations, you must focus on understanding the business priorities. Should you work to remediate the non-critical development environment before the production environment? Probably not; Understanding the business will help you make better decisions.
  • Understanding assets, users, and data - not only are these items key property that you are charge with protecting and restoring, but they're also going to be the key tools and targets sought out by attackers during a breach. Understanding which key assets enable business processes will help focus monitoring and response efforts. Understanding the various user types will help determine incident scope. Understanding the data will help determine monitoring efforts and attacker motivations. The attackers always want privileged accounts on critical assets to reach the target data.
  • Understanding legal, regulatory, and policy drivers - most organizations conduct business in a regulated industry. Whether this is dictated by regulations such as PCI and HIPAA or whether state and federal disclosure laws. Every incident response team must take this into account when building the program. One cannot consider that a breach response has been successful if and the end of the investigation fines are levied against the business.
  • Understanding technology, people, and processes - as responders, we're dependent on the technology at our disposal, the skills we've acquired throughout our careers, and a repeatable process for applying said skills. Technology and skills must be right-sized for the task at hand; do you need an expert reverse engineer or expensive forensics technology to respond to a commodity virus outbreak? Probably not. Should you have the right response processes defined that will enable an effective breach investigation? Probably so.
  • Understanding communication needs - each organization is going to differ with the level of communication needed during a breach. Some Executive teams and Boards are thirsty for every detail, others simply want to know when the issue has been remediated.
  • Identify the key contributors - while each breach response will be unique and require different contributors, you can begin to establish the core team members. Obviously, you'll require the technical team to drive analysis, the coordination team to manage and ensure that evidence is recorded, and probably representatives from Legal, HR, and Corporate Communications.


The next step is going to be documenting key processes and training the incident response team members.

  • IR invocation process - how will you decide when to invoke the IR team? Who will be responsible for sounding the klaxon horn?
  • Threat response processes - how will the technical analysts conduct analysis for identified threats? This is not a "how-to" guide, but rather a high level process for ensuring that evidence is recorded and communicated appropriately. You always want to rely on the creativity, skills, and instincts of your technical team to drive analysis.
  • Threat and incident prioritization process - how will you ensure you deal with the gravest threats first? How will you prioritize response to multiple incidents? These questions can be answered by creating a prioritization process that takes into account the severity and criticality of assets, users, and data involved with the breach.
  • Remediation prioritization - since the key to a successful incident response is restoring business operations, how will you determine which assets, users, and data will be restored first? Here again, taking into account business priorities, assets, users, and data will help make decisions that will enable the business to return to steady state quickly.
  • Cleanup and lessons learned - there is never enough emphasis on deriving lessons learned from incidents. Ensuring that a post-mortem process is available to capture critical intelligence and recommendations to prevent breaches in the future will ensure that your threat detection and incident response programs are consistently maturing.


The final step, rehearsal. To quote an old saying: "practice makes perfect". Performing threat simulation exercises will allow the incident response team to rehearse response processes, expand familiarity with evidence, and grow the understanding of the attack lifecycle in a safe, stress free environment. There is a reason the top law enforcement groups routinely train for various scenarios: the data shows that training and rehearsal increase brain and muscle memory for critical tasks that must be executed without failure in the heat of the moment.


Now, if you're sitting there after reading this article wondering how in the world you're going to get all this done, don't worry, Rapid7 is here to help. As part of our recent announcement regarding general availability of incident response services, the Strategic Services organization has some programs to help:

  • Threat Detection and Incident Response Program Development where our experts help your organization get better at detecting and responding to breaches.
  • Threat Simulation Services where our experts create a threat scenario, tailored to your business, and host a tabletop threat simulation, or go the extra step and perform a staged breach of your environment.
  • Incident Response Retainer where your experts are here to help investigate a breach.


We also hosted a webcast that covers incident preparation and response with myself and Mike Scutt: view it on-demand now!


Engage us on Twitter: @wadew @omgAPT