Skip navigation
All Places > Information Security > Blog

(A Story by Rapid7 Labs)

 

Merry HaXmas to you! Each year we mark the 12 Days of HaXmas with 12 blog posts on hacking-related topics and roundups from the year. This year, we’re highlighting some of the “gifts” we want to give back to the community. And while these gifts may not come wrapped with a bow, we hope you enjoy them.

 

Happy Holi-data from Rapid7 Labs!

It’s been a big year for the Rapid7 elves Labs team. Our nigh 200-node strong Heisenberg Cloud honeypot network has enabled us to bring posts & reports such as The Attacker’s Dictionary, Cross-Cloud Adversary Analytics and Mirai botnet tracking to the community, while Project Sonar fueled deep dives into National Exposure as well as ClamAV, fuel tanks and tasty, tasty EXTRABACON.

 

Our final gift of the year is the greatest gift of all: DATA! We’ve sanitized an extract of our November, 2016 cowrie honeypot data from Heisenberg Cloud. While not the complete data set, it should be good for hours of fun over the holiday break. You can e-mail research [at] rapid7 [dot] com if you have any questions or leave a note here in the comments.

 

While you’re waiting for that to download, please enjoy our little Haxmas tale…

divider.png

Once upon a Haxmas eve… CISO Scrooge sat sullen in his office. His demeanor was sour as he reviewed the day’s news reports and sifted through his inbox, but his study was soon interrupted by a cheery minion’s “Merry HaXmas, CISO!”.

 

CISO Scrooge replied, “Bah! Humbug!”

 

The minion was taken aback. “HaXmas a humbug, CISO?! You surely don’t mean it!”

 

“I do, indeed…” grumbled Scrooge. “What is there to be merry about? Every day attackers are compromising sites, stealing credentials and bypassing defenses. It’s almost impossible to keep up. What’s more, the business units and app teams here don’t seem to care a bit about security. So, I say it again ‘Merry HaXmas?’ - HUMBUG!”

 

Scrooge’s minion knew better than argue and quickly fled to the comforting glow of the pew-pew maps in the security operations center.

 

As CISO Scrooge returned to his RSS feeds his office lights dimmed and a message popped up on his laptop, accompanied by a disturbing “clank” noise (very disturbing indeed since he had the volume completely muted). No matter how many times he dismissed the popup it returned, clanking all the louder. He finally relented and read the message: “Scrooge, it is required of every CISO that the defender spirit within them should stand firm with resolve in the face of their adversaries. Your spirit is weary and your minions are discouraged. If this continues, all your security plans will be for naught and attackers will run rampant through your defenses. All will be lost.”

 

Scrooge barely finished uttering, “Hrmph. Nothing but a resourceful security vendor with a crafty marketing message. My ad blocker must be misconfigured and that bulb must have burned out.”

 

“I AM NO MISCONFIGURATION!” appeared in the message stream, followed by, “Today, you will be visited by three cyber-spirits. Expect their arrivals on the top of each hour. This is your only chance to escape your fate.” Then, the popup disappeared and the office lighting returned to normal. Scrooge went back to his briefing and tried to put the whole thing out of his mind.

 

The Ghost of HaXmas Past

CISO Scrooge had long finished sifting through news and had moved on to reviewing the first draft of their PCI DSS ROC[i]. His eyes grew heavy as he combed through the tome until he was startled with a bright green light and the appearance of a slender man in a tan plaid 1970’s business suit holding an IBM 3270 keyboard.

 

“Are you the cyber-spirit, sir, whose coming was foretold to me?”, asked Scrooge.

 

“I am!”, replied the spirit. “I am the Ghost of Haxmas Past! Come, walk with me!”

 

As Scrooge stood up they were seemingly transported to a room full of mainframe computers with workers staring drone-like into green-screen terminals.

 

“Now, this was security, spirit!” exclaimed Scrooge. “No internet…No modems…Granular RACF[ii] access control…” (Scrooge was beaming almost as bright as the spirit!)

 

“So you had been successful securing your data from attackers?”, asked the spirit.

 

“Well, yes, but this is when we had control! We had the power to give or deny anyone access to critical resources with a mere series of arcane commands.” As soon as he said this, CISO Scrooge noticed the spirit moving away and motioning him to follow. When he caught up, the scene changed to cubicle-lined floor filled with desktop PCs.

 

“What about now, were these systems secure?”, inquired the spirit.

 

“Well, yes. It wasn’t as easy as it was with the mainframe, but as our business tools changed and we started interacting with clients and suppliers on the internet we found solutions that helped us protect our systems and networks and give us visibility into the new attacks that were emerging.”, remarked CISO Scrooge. “It wasn’t easy. In fact, it was much harder than the mainframe, but the business was thriving: growing, diversifying and moving into new markets. If we had stayed in a mainframe mindset we’d have gone out of business.”

 

The spirit replied, “So, as the business evolved, so did the security challenges, but you had resources to protect your data?”

 

“Well, yes. But, these were just PCs. No laptops or mobile phones. We still had control!”, noted Scrooge.

 

“That may be,” noted the spirit, “but if we continued our journey, would this not be the pattern? Technology and business practices change, but there have always been solutions to security problems coming at the same pace?” CISO Scrooge had to admit that as he looked back in his mind, there had always been ways to identify and mitigate threats as they emerged. They may not have always been 100% successful, but the benefits of the “new” to the business were far more substantial than the possible issues that came with it.

 

The Ghost of Haxmas Present

As CISO Scrooge pondered the spirit’s words he realized he was back at his desk, his screen having locked due to the required inactivity timeout.  He gruffed a bit (he couldn’t understand the 15-minute timeout when at your desk as much as you can’t) and fumbled 3 attempts at his overly-complex password to unlock the screen before he was logged back in. His PCI DSS ROC was minimized and his browser was on a MeTube video (despite the site being blocked on the proxy server). He knew he had no choice but to click “play”. As he did, it seemed to be a live video of the Mooncents coffee shop down the street buzzing with activity. He was seamlessly transported from remote viewer to being right in the shop, next to a young woman in bespoke, authentic, urban attire, sipping a double ristretto venti half-soy nonfat decaf organic chocolate brownie iced vanilla double-shot gingerbread frappuccino. Amongst the patrons were people on laptops, tablets and phones, many of them conducting business for CISO’s company.

 

“Dude. I am the spirit of Haxmas Present”, she said, softly, as her gaze fixated upon a shadowy figure in the corner. CISO Scrooge turned his own gaze in that direction and noticed a hoodie-clad figure with a sticker-laden laptop. Next to the laptop was a device that looked like a wireless access point and Scrooge could just barely hear the figure chuckling to himself as his fingers danced across the keyboard.

 

“Is that person doing what I think he’s doing?”, Scrooge asked the spirit.

 

“Indeed,” she replied. “He’s setup a fake Mooncents access point and is intercepting all the comms of everyone connected to it.”

 

Scrooges’ eyes got wide as he exclaimed “This is what I mean! These people are just like sheep being led to the shearer. They have no idea what’s happening to them! It’s too easy for attackers to do whatever they want!” As he paused for a breath, the spirit gestured to a woman who just sat down in the corner and opened her laptop, prompting Scrooge to go look at her screen. The woman did work at CISO’s company and she was in Mooncents on her company device, but — much to the surprise of Scrooge — as soon as she entered her credentials, she immediately fired up the VPN Scrooge’s team had setup, ensuring that her communications would not be spied upon. The woman never once left her laptop alone and seemed to be very aware of what she needed to do to stay as safe as possible.

 

“Do you see what is happening?”, asked the spirit? “Where and how people work today are not as fixed as it was in the past. You have evolved your corporate defenses to the point that attackers need to go to lengths like this or trick users through phishing to get what they desire.”

 

“Technology I can secure. But how do I secure people?!”, sighed Scrooge.

 

“Did not this woman do what she needed to keep her and your company’s data safe?”, asked the spirit.

 

“Well, yes. But it’s so much more work!”, noted Scrooge. “I can’t install security on users, I have to make them aware of the threats and then make it as easy as possible for them to work securely no matter where they are!”[iii]

 

As soon as he said this, he realized that this was just the next stage in the evolution of the defenses he and his team had been putting into place. The business-growing power inherent in this new mobility and the solid capabilities of his existing defenses forced attackers to behave differently and he understood that he and his team probably needed to as well.

 

The spirit gave a wry, ironic grin at seeing Scrooge’s internal revelation. She handed him an infographic titled “Ignorance & Want” that showcased why it was important to make sure employees were well-informed and to also stay in tune with how users want to work and make sure his company’s IT offerings were as easy-to-use and functional as all the shiny “cloud” apps.

 

The Ghost of Haxmas Future

As Scrooge took hold of the infographic the world around him changed. A dark dystopian scene faded into view. Buildings were in shambles and people were moving in zombie-like fashion in the streets. A third, cloaked spirit appeared next to him and pointed towards a disheveled figure hulking over a fire in a barrel. An “eyes” emoji appeared on the OLED screen where the spirit’s face should have been. CISO Scrooge didn’t even need to move closer to see that it was a future him struggling to keep warm to survive in this horrible wasteland.

 

“Isn’t this a bit much?”, inquired Scrooge. The spirit shrugged and a “whatever” emoji appeared on the screen. Scrooge continued, “I think I’ve got the message. Business processes will keep evolving and moving faster and will never be tethered and isolated again. I need to stay positive and constantly evolve — relying on psychology, education as well as technology — to address the new methods attackers will be adopting. If I don’t, it’s ‘game over’.”

 

The spirit’s screen flashed a “thumbs up” emoji and CISO Scrooge found himself back at his desk, infographic strangely still in hand with his Haxmas spirt fully renewed. He vowed to keep Haxmas all the year through from now on.

 


[i] Payment Card Industry Data Security Standard Report on Compliance

[ii] http://www-03.ibm.com/systems/z/os/zos/features/racf/

[iii] Scrooge eventually also realized he could make use of modern tools such as Insight IDR to combine security & threat event data with user behavior analysis to handle the cases where attackers do successfully breach users.

Merry HaXmas to you! Each year we mark the 12 Days of HaXmas with 12 blog posts on hacking-related topics and roundups from the year. This year, we’re highlighting some of the “gifts” we want to give back to the community. And while these gifts may not come wrapped with a bow, we hope you enjoy them.

sam-snowman.jpg

 

Sam the snowman taught me everything I know about reindeer [disclaimer: not actually true], so it only seemed logical that we bring him back to explain the journey of machine learning. Wait, what? You don’t see the correlation between reindeer and machine learning? Think about it, that movie had everything: Yukon Cornelius, the Bumble, and of course, Rudolph himself. And thus, in sticking with the theme of HaXmas 2016, this post is all about the gifts of early SIEM technology, “big data”, and a scientific process.

 

SIEM and statistical models – Rudolph doesn’t get to play the reindeer games

donner-worst-dad.jpg

Just as Rudolph had conformist Donner’s gift of a fake black nose promising to cover his glowing monstrosity [and it truly was impressive to craft this perfect deception technology with hooves], information security had the gift of early SIEM technology promising to analyze every event against known bad activity to spot malware. The banking industry had just seen significant innovation in the use of statistical analysis [a sizable portion of what we now call “analytics”] for understanding the normal in both online banking and payment card activities. Tracking new actions and comparing them to what is typical for the individual takes a great deal of computing power and early returns in replicating fraud prevention’s success were not good.

 

SIEM had a great deal working against it when everyone suddenly expected a solution designed solely for log centralization to easily start empowering more complex pattern recognition and anomaly detection. After having witnessed, as consumers, the fraud alerts that can come from anomaly detection, executives starting expecting the same from their team of SIEM analysts. Except, there were problems:

  • the events within an organization vary a great deal more than the login, transfer, and purchase activities of the banking world,
  • the fraud detection technology was solely dedicated to monitoring events in other, larger systems, and
  • SIEM couldn’t handle both data aggregation and analyzing hundreds of different types of events against established norms.

After all, my favorite lesson from data scientists is that “counting is hard”. Keeping track of the occurrence of every type of event for every individual takes a lot of computing power and understanding of each type of event. After attempting to define alerts for transfer size thresholds, port usage, and time-of-day logins, no one understood that services like Skype using unpredictable ports and the most privileged users regularly logging in late to resolve issues would cause a bevy of false positives. This forced most incident response teams to banish advanced statistical analysis to the island of misfit toys, like an elf who wants to be a dentist.

 

“Big Data” – Yukon Cornelius rescues machine learning from the Island of Misfit Toys

There is probably no better support group friend for the bizarre hero, Yukon Cornelius, than “big data” technologies. Just as NoSQL databases, like Mongo, to map-reduce technologies, like Hadoop, were marketed as the solution to every, conceivable challenge, Yukon proudly announced his heroism to the world. Yukon carried a gun he never used, even when fighting Bumble, and “big data” technology is varied and each kind needs to be weighed against less expensive options for each problem. When jumping into a solution technology-first, most teams attempting to harness “big data” technology came away with new hardware clusters and mixed results; insufficient data and security experts miscast as data experts still prevent returns on machine learning from happening today. yukon.jpg

 

However, with the right tools and the right training, data scientists and software engineers have used “big data” to rescue machine learning [or statistical analysis, for the old school among you] from its unfit-for-here status. Thanks to “big data”, all of the original goals of statistical analysis in SIEMs are now achievable. This may have led to hundreds of security software companies marketing themselves as “the machine learning silver bullet”, but you just have to decide when to use the gun and when to use the pick axe. If you can cut through the hype to decide when the analytics are right and for which problems machine learning is valuable, you can be a reason that both Hermey, the dentist, and Rudolph, the HaXmas hero, reach their goal.

 

Visibility - only Santa Claus could get it from a glowing red nose

red-nose-reality.jpg

But just as Rudolph’s red nose didn’t make Santa’s sleigh fly faster, machine learning is not a magic wand you wave at your SIEM to make eggnog pour out. That extra foggy Christmas Eve couldn’t have been foggy all over the globe [or it was more like the ridiculous Day After Tomorrow], but Santa knows how to defy physics to reach the entire planet in a single night, so we can give him the benefit of the doubt and assume he knew when and where he needed a glowing red ball to shine brighter than the world’s best LED headlights. I know that I’ve held a pretty powerful Maglite in the fog and still couldn’t see a thing, so I wouldn’t be able to get around Mt. Washington armed with a glowing reindeer nose.

 

Similarly, you can’t just hand a machine learning toolkit to any security professional and expect them to start finding the patterns they should care about across those hundreds of data types mentioned above. It takes the right tools, an understanding of the data science process, and enough security domain expertise to apply machine learning to the attacker behaviors well hidden within our chaotically normal environments. Basic anomaly detection and the baselining of users and assets against their peers should be embedded in modern SIEM and EDR solutions to reveal important context and unusual behavior. It’s the more focused problems and data sets that demand the kind of pattern recognition within the characteristics of a website or PHP file only the deliberate development of machine learning algorithms can properly address.

Merry HaXmas to you! Each year we mark the 12 Days of HaXmas with 12 blog posts on hacking-related topics and roundups from the year. This year, we’re highlighting some of the “gifts” we want to give back to the community. And while these gifts may not come wrapped with a bow, we hope you enjoy them.

 

As we close out the end of the year, I find it important to reflect on the IoT vulnerability research conducted during 2016 and what we learned from it. There were several exciting IoT vulnerability research projects conducted by Rapid7 employees in 2016, which covered everything from lighting automation solutions to medical devices. Through this research, we want to give the "gift" of more information about IoT security to the community. In the spirit of the celebrating the holidays, let's recap and celebrate each of these projects and some of the more interesting findings.

 

Comcast XFINITY Home Security System

 

xfinity-armed-everything-is-cool.png

2016 opened with a research project on the Comcast XFINITY Home Security System which was published in January 2016. Phil Bosco, a member of the Rapid7’s Global Services pen test team, targeted his XFINITY home security systems for evaluation. During testing, Phil discovered that by creating a failure condition in the 2.4 GHz radio frequency band used by the Zigbee communication protocol, the Comcast XFINITY Home Security System would fail open, with the base station failing to recognize or alert on a communications failure with the component sensors. This interesting finding showed us that if communication to the systems sensors is lost, the system would fail to recognize that lost communication. Additionally, this failure also prevented the system from properly alarming when the sensor detected a condition such as a open door or window. This vulnerability allowed anyone capable of interrupting the 2.4 GHz Zigbee communication between sensor and base station to effectively silence the system. Comcast has since fixed this issue.

 

Osram Sylvania Lightify Automated Lighting

2.png

Since automated lighting has become very popular I decided to examine the OSRAM Sylvania LIGHTIFY automated lighting solution. This research project consisted of looking at both the Home and Pro (enterprise) versions. This project ended up revealing a total of nine issues, four in the Home version and five in the Pro. The Pro version had the most interesting of these results which included identifying issues consisting of persistent Cross Site Scripting (XSS) and Weak default WPA2 pre-shared keys (PSKs). The XSS vulnerabilities we found had two entry points with the most entertaining one being an out of band injection using WiFi Service Set Identifier (SSID) to deliver the XSS attack into the Pro web management interface. A good explanation of this type of attack delivery method is explained in a Whiteboard Wednesday video I recorded. Next, the finding that I would consider the most worrisome is the WPA PSK issue. Default passwords have been a scourge of IoT recently. Although, in this case the default password are different across every Pro device produces, closer examination of the WPA PSK revealed they were easily cracked. So how did this happen? Well, in this case the PSK was only eight characters long, which is considered very short for a PSK, and it only used characters that were hexadecimal lowercase (abcdef0123456789) which makes the number of combinations or key space much easier to brute force  and can allow a malicious actor to capture a authentication handshake and brute force it offline in only a few hours.

 

Bluetooth Low Energy (BLE) Trackers

 

Screen Shot 2016-12-16 at 2.50.37 PM.png

You ever get curious about those little Bluetooth low energy (BLE) tracker dongles you can hang on your key chain that helps you locate your keys if you misplace them? So did I, but my interest went a little further then finding my lost keys. I was interested in how they worked and what, if any, security issues could be associated to their use or misuse. I purchased several different brands and started testing their ecosystem, yes ecosystem, that is all of the services that make an IoT solution function, which often includes the hardware, mobile applications and cloud APIs. One of the most fascinating aspects of these devices is the crowd GPS concept. How does that work? Let’s say you attach one of the devices to your bicycle and it gets stolen. Every time that bicycle passes within close proximity to another user of that specific product their cell phone will detect your dongle on the bicycle and send the GPS location to the cloud allowing you to identify its location. Kind of neat and I expect it works well if you have an area with a good saturation of users, but if you live in a rural area there's less chance of that working as well. During this project we identified several interesting vulnerabilities related to the tracking identifiers and GPS tracking. For example, we found that the devices' tracking ID was easy identified and in a couple cases was directly related to the BLE physical address. Combining that with some cloud API vulnerabilities, we were able to track a user using the GPS of their device. Additionally, in couple cases we were able to poison the GPS data for other devices. With weak BLE pairing, we were also able to gain access to a couple of the devices and rename them and set off their location alarms which drained the small batteries for the devices.

 

Animas OneTouch Ping Insulin Pump

 

insulin-pump.jpg

Rapid7’s Jay Radcliffe, a member of the Global Services team and security researcher at Rapid7, found several fascinating vulnerabilities while testing the Animas OneTouch Ping insulin pump. Jay is renowned for his medical device research, which has a personal aspect to it as he is diabetic. In the case of the Animas OneTouch, Jay found and reported three vulnerabilities which include cleartext communication, weak pairing between remote and pump, and replay attack vulnerability. During this research project it was determined that these vulnerabilities could be used to potentially dispense insulin remotely, which could impact the safety and health of the user. Jay worked closely with the manufacturer to help create effective mitigations for these vulnerabilities, which can be used to reduce or eliminate the risk. Throughout the project, there was positive collaboration between Jay, Rapid7 and Johnson & Johnson and patients were notified prior to disclosure.

 

 

Conclusion:

Stepping back and taking a holistic look at all of the vulnerabilities found during these research projects, we can notice a pattern of common issues including:

 

  • Lack of communication encryption
  • Poor device pairing
  • Confidential data (such as passwords) stored within mobile applications
  • Vulnerability to replay attacks
  • Weak default passwords
  • Cloud API and device management web services vulnerable to common web vulnerabilities

 

These findings are not a surprise and appear to be issues we commonly encounter when examining an IoT product's ecosystem. What is the solution to resolving these issues then? First, IoT manufactures can easily apply some basic testing across these areas to quickly identify and fix vulnerabilities on products prior to going to market. Second, we as end users can change default passwords, such as the standard login password and WiFi WPA PSK, to protect our devices from many forms of compromise.

 

It is also important to note that these IoT research projects are just a few examples of the dedication that Rapid7 and its employees have in regards to securing the world of IoT.  These research projects allow us to continually expand our knowledge around IoT security and vulnerabilities. By working closely with vendors to identify and mitigate issues, we can continue to help those vendors in expanding their working knowledge of security, which will flow into future products. Our work also allows us to share this knowledge with consumers so they can make better choices and mitigate common risks that are often found within IoT products.

Merry HaXmas to you! Each year we mark the 12 Days of HaXmas with 12 blog posts on hacking-related topics and roundups from the year. This year, we’re highlighting some of the “gifts” we want to give back to the community. And while these gifts may not come wrapped with a bow, we hope you enjoy them.

 

“May you have all the data you need to answer your questions – and may half of the values be corrupted!”

- Ancient Yiddish curse

 

This year, Christmas (and therefore Haxmas) overlaps with the Jewish festival of Chanukah. The festival commemorates the recapture of the Second Temple. As part of the resulting cleaning and sanctification process, candles were required to burn continuously for seven days – but there was only enough oil for one. Thanks to the intervention of God, that one day of oil burned for eight days, and the resulting eight-day festival was proclaimed.

 

Unfortunately despite God getting involved in everything from the edibility of owls (it’s in Deuteronomy, look it up) to insufficient stocks of oil, there’s no record of divine intervention to solve for insufficient usable data, and that’s what we’re here to talk about. So pull up a chair, grab yourself a plate of latkes and let’s talk about how you can make data-driven security solutions easier for everyone.

 

Data-Driven Security

 

As security problems have grown more complex and widespread, people have attempted to use the growing field of data science to diagnose issues, both on the macro level (industry-wide trends and patterns found in the DBIR) and the micro (responding to individual breaches). The result is Data-Driven Security, covered expertly by our Chief Data Scientist, Bob Rudis, in the book of the same name.

 

Here at Rapid7, our Data Science team has been working on everything from systems to detect web shells before they run (see our blog post about webshells) to internal projects that improve our customers’ experience. As a result, we’ve seen a lot of data sources from a lot of places, and have some thoughts on what you can do to make data scientists’ problem-solving easier before you even know you have an issue.

 

Make sure data is available

 

Chanukah has just kicked off and you want two things: to eat fritters and apply data science to a bug, breach or feature request. Luckily you’ve got a lot of fritters and a lot of data – but how much data do you actually have?

 

People tend to assume that data science is all about the number of observations. If you’ve got a lot of them, you can do a lot; only a few and you can only do a little. Broadly-speaking, that’s true, but the span of time data covers and the format it takes are also vitally important. Seasonal effects are a well-studied phenomenon in human behavior and, by extension, in data (which one way or another, tends to relate to how humans behave). What people do and how much they do it shifts between seasons, between months, even between days of the week. This means that the length of time your data covers can make the difference between a robust answer and an uncertain one – if you’ve only got Chanukah’s worth, we can draw patterns in broad strokes but we can’t eliminate the impact seasonal changes might have.

 

The problem with this is that storing a lot of data over a long period of time is hard, potentially expensive and a risk in and of itself – it’s not great for user or customer privacy, and in the event of a breach it’s another juicy thing the attacker can carry off. As a result, people tend to aggregate their raw data, which is fine if you know the questions you’re going to want answering.

 

If you don’t, though, the same thing that protects aggregated data from malicious attackers will stymie data scientists: it’s very hard, if you’re doing it right, to reverse-engineer aggregation, and so researchers are limited to whatever fields or formats you thought were useful at the time, which may not be the ones they actually need.

 

One solution to both problems is to keep data over a long span of time, in its raw format, but sample: keep 1 random row out of 1,000, or 1 out of 10,000, or an even higher ratio. That way data scientists can still work with it and avoid seasonality problems, but it becomes a lot harder for attackers to reconstruct the behavior of individual users. It’s still not perfect, but it’s a nice middle ground.

 

Make sure data is clean

 

It’s the fourth day of Chanukah, you’ve implemented that nice sampled data store, and you even managed to clean up the sufganiyot the dog pulled off the table and joyously trod into the carpet at its excitement to get human food. You’re ready to get to work, you call the data scientists in, and they watch as this elegant sampled datastore collapses into a pile of mud because 3 months ago someone put a tab in a field that shouldn’t have a tab and now everything has to be manually reconstructed.

 

If you want data to be reliable, it has to be complete and it has to be clean. By complete we mean that if a particular field only has a meaningful value 1/3rd of the time, for whatever reason, it’s going to be difficult to rely on it (particularly in a machine learning context, say). By clean, we mean that there shouldn’t be unexpected values, particularly the sort of unexpected value that breaks data formatting or reading.

 

In both cases the answer is data validity checks. Just as engineers have tests – tasks that run every so often to ensure changes haven’t unexpectedly broken code – data storage systems and their associated code need validity checks, which run against a new row every so often and make sure that they have all their values, they’re properly formatted, and those values are about what they should be.

 

Make sure data is documented

 

It’s the last day of Chanukah, you’ve got a sampled data store with decent data, the dreidel has rolled under the couch and you can’t get it out, and you just really really want to take your problem and your data and smush them together into a solution. The data scientists read in the data, nothing breaks this time… and are promptly stumped by columns with names like “Date Of Mandatory Optional Delivery Return (DO NOT DELETE, IMPORTANT)” or simply “f”. You can't expect their bodies to harbour that kind of thing.

 

Every time you build a new data store and get those validity checks set up, you should also be setting up documentation. Where it lives will vary from company to company, but it should exist somewhere and set out what each field means (“the date/time the request was sent”), an example of what sort of value it contains (“2016-12-10 13:42:45”) and any caveats (“The collectors were misconfigured from 2016-12-03 to 2016-12-04, so any timestamps then are one hour off”). That way, data scientists can hit the ground running, rather than spending half their time working out what the data even means.

 

So, as you prepare for Chanukah and 2017, you should be preparing for data science, too. Make sure your data is (respectfully) collected, make sure your data is clean, and make sure your data is documented. Then you can sit back and eat latke in peace.

Kirk Hayes

The Twelve Pains of Infosec

Posted by Kirk Hayes Employee Dec 24, 2016

One of my favorite Christmas carols is the 12 Days of Christmas. Back in the 90’s, a satire of the song came out in the form of the 12 Pains of Christmas, which had me rolling on the floor in laughter, and still does. Now that I am in information security, I decided it is time for a new satire, maybe this will start a new tradition, and so I am presenting, the 12 Pains of Infosec.

musical-notes.jpg The first thing in infosec that's such a pain to me is your password policy

 

The second thing in infosec that's such a pain to me is default credentials, and your password policy

 

The third thing in infosec that's such a pain to me is falling for phishing, default credentials, and your password policy

 

The fourth thing in infosec that's such a pain to me is patch management, falling for phishing, default credentials, and your password policy

 

The fifth thing in infosec that's such a pain to me is Windows XP, patch management, falling for phishing, default credentials, and your password policy

 

The sixth thing in infosec that's such a pain to me is Lack of input filtering, Windows XP, patch management, falling for phishing, default credentials, and your password policy

 

The seventh thing in infosec that's such a pain to me is no monitoring, lack of input filtering, Windows XP, patch management, falling for phishing, default credentials, and your password policy

 

The eighth thing in infosec that's such a pain to me is users as local admins, no monitoring, Lack of input filtering, Windows XP, patch management, falling for phishing, default credentials, and your password policy

 

The ninth thing in infosec that's such a pain to me is lack of management support, users as local admins, no monitoring, lack of input filtering, Windows XP, patch management, falling for phishing, default credentials, and your password policy

 

The tenth thing in infosec that's such a pain to me is testing for compliance, lack of management support, users as local admins, no monitoring, lack of input filtering, Windows XP, patch management, falling for phishing, default credentials, and your password policy

 

The eleventh thing in infosec that's such a pain to me is no asset management, testing for compliance, lack of management support, users as local admins, no monitoring, lack of input filtering, Windows XP, patch management, falling for phishing, default credentials, and your password policy

 

The twelfth thing in infosec that's such a pain to me is trust in antivirus, no asset management, testing for compliance, lack of management support, users as local admins, no monitoring, Lack of input filtering, Windows XP, patch management, falling for phishing, default credentials, and your password policy

 

musical-notes.jpg

The first thing in infosec that's such a pain to me is your password policy

When I go into organizations for penetration tests, one of the easiest ways to get in is through password guessing. Most organizations use a password policy of 8 characters, complexity turned on, and change every 90 days. So, what do the users do? They come up with a simple to remember password that will never repeat. Yes, I am talking about the infamous Winter16 or variations of season and year. If they aren’t using that password, then chances are it is something just as simple. Instead, set a longer password requirement (12 characters or more) and blacklist common words, such as password, seasons, months, and company name.

 

The second thing in infosec that's such a pain to me is default credentials

The next most common finding I see is the failure to change default credentials. It is such a simple mistake, but one that can cost your organization a ton! This doesn’t just go for web apps like JBOSS and Tomcat, but also for embedded devices. Printers and other embedded devices are a great target since the default credentials aren’t usually changed. They often hold credentials to other systems to help gain access or simply can be used as a pivot point to attack other systems.

 

The third thing in infosec that's such a pain to me is falling for phishing

Malicious actors go for the weakest link. Often this is the users. Sending a carefully crafted phishing email is almost 100% successful. In fact, even many security professionals fall victim to phishing emails. So, what can we do about it? Well, we must train our employees regularly to spot phishing attempts as well as encourage and reward them for alerting security on phishing attempts. Once reported, add the malicious URL to your blacklist, and redirect to a phishing education page. And for goodness sake, Security Department, please disable the links and remove any tags in the email before forwarding off as "education".

 

The fourth thing in infosec that's such a pain to me is patch management

There are so many systems out there. It can be hard to patch them all, but having a good patch management process is essential. Ensuring our systems are up to date with the latest patches will help protect those systems from known attacks. It is not just operating system patches that need to be applied, also for any software you have installed.

 

The fifth thing in infosec that's such a pain to me is Windows XP

Oh Windows XP, how I love and hate thee. Even though Windows XP support went the way of the dodo back in 2014, over 2.5 years later I still see it being used in corporate environments. While I called out Windows XP, it is not uncommon to see Windows 2000, Windows Server 2003, and other unsupported operating system software. While some of the issues with these operating systems have been mitigated, such as MS08_067, many places have not applied the patches or taken the mitigation tactics. That is not to mention what unknown security vulnerabilities that exist and will never be patched. Your best bet is to upgrade to a supported operating system. If you cannot for some reason (required software will not run on newer operating systems), segregate the network to prevent access to the unsupported systems.

 

The sixth thing in infosec that's such a pain to me is lack of input filtering

We all know and love the OWASP Top 10. Three of the top 10 is included in this pain. Cross-Site Scripting (XSS), SQL Injection (SQLi), HTML Injection, Command Injection, and HTML Redirects are all vulnerabilities that can be solved fully, or at least partially in the case with XSS, with input filtering. Web applications that perform input filtering will remove bad characters, allow only good characters, and perform the input filtering not at the client-side, but at the server-side. Then using output encoding/filtering, XSS is remediated as well.

 

The seventh thing in infosec that's such a pain to me is no monitoring

In 1974, Muhammad Ali said “His hands can't hit what his eyes can't see” referring to his upcoming fight with George Foreman. This quote bodes true in Infosec as well. You cannot defend what you cannot see. If you are not performing monitoring in your network, and centralized monitoring so you can collaborate the logs, you will miss attacks. As Dr. Eric Cole says “Prevention is ideal, but detection is critical.” This is also critical to REVIEW the logs, meaning you will need good people that know what they are looking for, not just good monitoring software.

 

The eighth thing in infosec that's such a pain to me is users as local admins

Though for years we have been suggesting to segregate user privileges, yet almost every penetration test I perform I find this to be an issue. Limiting use of accounts to only what is needed to do their job is very hard, but essential to secure your environment. This means not giving local administrator privileges to all users but also using separate accounts for managing the domain, limiting the number of privileged accounts, and monitoring the use of these accounts and group memberships.

 

The ninth thing in infosec that's such a pain to me is lack of management support

How often do I run into people who want to make a change or changes in their network, yet they do not get the support needed from upper management? A LOT! Sometimes an eye-opening penetration test works wonders.

 

The tenth thing in infosec that's such a pain to me is testing for compliance

I get it, certain industries require specific guidelines to show adequate security is in place, but when you test only for compliance sake, you are doing a disservice to your organization. When you attempt to limit the scope of the testing or firewall off the systems during the test, you are pulling a blinder over your eyes, and cannot hope to secure your data. Instead, use the need for testing to meet compliance a way to get more management support and enact real change in the organization.

 

The eleventh thing in infosec that's such a pain to me is no asset management

You can’t protect what you don’t know about. In this regard, employ an asset management system. Know what devices you have and where they are located. Know what software you have, and what patch levels they are at. I can’t tell you how many times I have exploited a system and my customer says “What is that? I don’t think that is ours”, only to find out that it was their system, they just didn’t have good asset management in place.

 

The twelfth thing in infosec that's such a pain to me is trust in antivirus

A few years ago, I read that antivirus software was only about 30% effective, leading to headlines about the death of antivirus, yet it still is around. Does that mean it is effective in stopping infections on your computer? No. I am often asked “What is the best antivirus I should get for my computer?” My answer is usually to grab a free antivirus like Microsoft Security Essentials, but be careful where you surf on the internet and what you click on. Antivirus will catch the known threats, so I believe it still has some merit, especially on home computers for relatives who do not know better, but the best protection is being careful. Turn on “click to play” for Flash and Java (if you can’t remove Java). Be careful what you click on. Turn on an ad blocker. There is no single “silver bullet” in security that is going to save you. It is a layering of technologies and awareness that will.

 

I hope you enjoyed the song, and who knows, maybe someone will record a video singing it! (not me!) Whatever holiday you celebrate this season, have a great one. Here’s to a more secure 2017 so I don’t have to write a new song next year. Maybe “I’m dreaming of a secure IoT” would be appropriate.

Security Information and Event Management (SIEM) is security’s Schrödinger’s cat. While half of today’s organizations have purchased SIEM tools, it’s unknown if the tech is useful to the security team… or if its heart is even beating or deployed. In response to this pain, people, mostly marketers, love to shout that SIEM is dead, and analysts are proposing new frameworks with SIEM 2.0/3.0, Security Analytics, User & Entity Behavior Analytics, and most recently Security Operations & Analytics Platform Architecture (SOAPA).

 

However, SIEM solutions have also evolved from clunky beasts to solutions that can provide value without requiring multiple dedicated resources. While some really want SIEM dead, the truth is it still describes the vision we all share: reliably find insights from security data and detect intruders early in the attack chain. What’s happened in this battle of survival of the fittest is that certain approaches and models simply weren’t working for security teams and the market.

 

What exactly has SIEM lost in this sweaty regimen of product development exercise? Three key areas have been tightened and toned to make the tech something you actually want to use.

 

No More Hordes of Alerts without User Behavior Context

User Behavior Analytics. You’ll find this phrase at every SIEM vendor’s booth, and occasionally in their technology as well. Why? This entire market segment explosion spawned from two major pain-points in legacy SIEM tech: (1) too many false-positive, non-contextual alerts, and a (2) failure to detect stealthy, non-malware attacks, such as the use of stolen credentials and lateral movement.

 

By tying every action on the network to the users and assets behind them, security teams spend less time retracing user activity to validate and triage alerts, and can detect stealthy, malicious behavior earlier in the attack chain. Applying UBA to SIEM data results in higher quality alerts and faster investigations, as teams are spending less time retracing IPs to users and running tedious log searches.

 

Lateral-Movement-Domain-Creds.PNG

 

Detections now Cover Endpoints Without Heavy Lifting

Endpoint Detection and Response. This is another super-hot technology of 2016, and while not every breach originates from the endpoint, endpoints are often an attacker’s starting point and provide crucial information during investigations. There are plenty of notable behaviors that if detected, are reliable signs of “investigate-me” behavior.

 

A couple examples:

  • Log Deletion
  • First Time Admin Action (or detection of privilege exploit)
  • Lateral Movement

 

Any SIEM that doesn’t offer built-in endpoint detection and visibility, or at the very least, automated ways to consume endpoint data (and not just anti-virus scans!), leaves gaps in coverage and across the attack chain. Without endpoint data, it’s very challenging to have visibility into traveling and remote workers or detect an attacker before critical assets are breached. It can also complicate and slow incident investigations, as endpoint data is critical for a complete story. The below highlights a standard investigation workflow along with the relevant data sources to consult at each step.

 

Speeding-Up-Investigations-Rapid7.PNG

 

Incident investigations are hard. They require both incident response expertise (how many breaches have you been a part of?) and also data manipulation skills to get the information you need. If you can’t search for endpoint data from within your SIEM, that slows down the process and may force you to physically access the endpoint to dig deeper.

Leading SIEMs today now offer a combination of Agents or an Endpoint Scan to ingest this data, detect local activity, and have it available for investigations. We do all of this and supplement our Endpoint detections with Deception Technology, which includes decoy Honey Credentials that are automatically injected into memory to better detect pass-the-hash and credential attacks.

 

Drop the Fear, Uncertainty, and Doubt About Data Consumption

There are a lot of things that excite me, for example, the technological singularity, autonomous driving, loading my mind onto a Westworld host. You know what isn’t part of that vision? Missing and incomplete data. Today’s SIEM solutions derive their value from centralizing and analyzing everything. If customers need to weigh the value of inputting one data set against another, that results in a fractured, frustrating experience. Fortunately, this too is now a problem of the past.

 

There are a couple of factors behind these winds of change. Memory capacity continues to expand close to a Moore’s Law pace, which is fantastic, as our log storage needs are heavier than ever before.

 

Evolution-of-Memory-Storage.jpeg

 

Vendors now are offering mature cloud architectures that can securely store and retain log data to meet any compliance need, along with faster search and correlation activity than most on-premise deployments can dream about. The final shift, and one that’s currently underway today, is with vendor pricing. Today’s models revolve around Events per Second and Data Volume Indexed. But, what’s the point of considering endpoint, cloud, and log data if the inevitable data volume balloon means the org can’t afford to do so?

 

We’ve already tackled this challenge and customers have been pleased with it. Over the next few years, new and legacy vendors alike will also shed existing models to also reflect the demand for sensible data pricing that finally arms incident investigators with the data and context they need.

 

There’s a lot of pain with existing SIEM technology – we’ve experienced it ourselves, from customers, and every analyst firm we’ve spoken with. However, that doesn’t mean the goal isn’t worthy or the technology has continually failed to adapt. Can you think of other ways SIEM vendors have collectively changed their approach over the years? Share it in the comments! If you’re struggling with an existing deployment and are looking to augment or replace, check out our webcast, “Demanding More From Your SIEM”, for recommendations and our approach to the SIEM you’ve always wanted.

What does 2017 hold for cybersecurity? Our mystics have drawn cards, checked crystal balls, and cast runes to peer into the future. See what the signs have in store for you in the new year.

 

aquarius-water-container-symbol.pngSage Corey Thomas, Rapid7

Gazing into the future of 2017, I believe we will continue to see market consolidation of security vendors. With a focus on increasing productivity, organizations will move further from disparate, point-solutions that solve just one problem to solutions that can be leveraged throughout the IT environment. This will drive security and IT vendors to integrate, consolidate, and better collaborate. It will become increasingly clear that IT and security professionals want to manage fewer solutions that are easy to use. I also expect to see the skills gap start to right itself. Security has reached a state of accessibility, by necessity. In most cases, you don’t need an advanced degree to enter the security field and you can often gain skills through certifications.

 

 

pisces-sign.pngK Seer Ellis (aka: Casey Ellis, Bugcrowd)

In 2016, we reached a level of dystopian weirdness that will be hard to top in 2017. Toasters brought down half the Internet, a hacker accidentally bricked an entire metropolitan transit system – and then got hacked himself by a vigilante, and there was a steady stream of "biggest breach ever" events. But we know that it will be topped. Gazing into the future, I see DDOS and Ransomware evolving and becoming more pervasive in both consumer and corporate contexts, leading to the rapid formation of policy around security best practices for consumer products and increased consumer pressure on vendors to demonstrate their proxy indicators of security. Finally, as companies learn how to use the crowd, we’ll see an evolution and improvement of penetration testing and eventually the widespread adoption of vulnerability disclosure programs as a means to achieve and maintain resilience faster.

 

 

aries.pngThe Todonomicon (aka: Tod Beardsley, Rapid7)

Peering into my BlueTooth-ready crystal ball, I can see that many, many more hobby hackers publishing vulnerabilities in IoT. Cost of entry is low, there are tons of new devices with old bugs and the DMCA now exempts consumer device research, which means boatloads of public vulnerability disclosures. Which is good – and also chaotic. You could say that how IoT manufacturers respond to these disclosures will be make or break for the industry. On the one hand, you might expect more mature companies to respond quickly and positively – patching and updating devices – but it’s also plausible that smaller, younger companies will be more nimble, and therefore able to respond faster.

 

 

taurus-zodiac-symbol-of-bull-head-front.pngKatie Moussoothsayer (aka: Katie Moussouris, Luta Security)

Gazing into the future, not everything is as clear as my crystal ball. Attribution for cyber attacks will still be hard. You can bet your nation-state-sophisticated-actor that ThreatButt.com may give as credible, if not more credible, attributions for attacks then the leading expert firms or intelligence agencies. Because who wants facts and experts, when you can instead have a pew-pew map and a nifty sticker. Besides, ThreatButt has Viking-grade encryption, unlike APTFriendFinder.com, which is basically the MySpace of parody attribution - ahead of its time. Additionally, the next US administration's cybersecurity policies will likely defy most conventional wisdom in this area, known as "the Cyber" to the president-elect. Not just any Cyber, but THE Cyber. Who knows, we may see some interesting funding for new cyber offense capabilities. Just who may find the capabilities truly offensive defies prediction.

 

And while there is no such thing as a "cyber Pearl harbor," next year will likely be one where Cyber Claus rains down with lumps of coal for the masses, both naughty and nice. On DDOS, on botnet, on malware, on clicked 'em! On ransom, on car hack, on Bitcoin, on victim! We're all on the list. Sleep tight.

 

 

gemini-zodiac-symbol-of-two-twins-faces.png

Hierophant Geiger (aka: Harley Geiger, Rapid7)

The dual nature of Gemini reflects both the progress and work still to be done this year. We may see a flat warrant standard for government access to stored digital content may pass Congress, but there is increased likelihood that there will be important exceptions that undermine the standard. Additionally, it’s possible that we’ll see action on standards for government access to stored data across borders, either through legislation and/or renegotiated trade agreements. As

Saturn makes its way through the policy house, law enforcement access to encrypted data will continue to be a hot issue. If Congress attempts to require a backdoor into encryption standards, or attempts to forbid private use of end-to-end encryption, a major battle may ensue.

 

 

cancer-1.png

Herald Deiland (aka: Deral Heiland, Rapid7)

There’s no such thing as retrograde for IoT in 2017. If 2016 was the year IoT exploded, 2017 will be the year that IoT comes to life. I believe 2017 will be the first year IoT is used to inflict physical harm on a human. I also believe that audio information – voice data – gathered from home automation systems, such as the Amazon Echo will be used for the first time to solve a crime. I also expect to see MFP device security issues directly tied to a major corporate breach.

 

 

 

leo-astrological-sign.png

Madame Bell LaPadula (aka: Wendy Nather, Duo Security)

As Mars passes into the Upper Right Magic Quadrant of the heavens, we will see the influence of cyberwar grow across the world. This will take the form of pitched battles over botnet assets rather than land, but just as civilians get caught in the real-world crossfire, consumers will pay the price for DDoS attacks, ransomware and other disruptions. Beware of empty promises and standards.

 

 

 

 

virgo-zodiac-symbol.pngRobsputin (aka: Robert Graham, Errata Security)

When Jupiter aligns with Mars, regulation of software will become a thing. "Contraband" software and IoT will become a thing, as our only choices will be boring IoT devices from big corporations like GE and Apple, or innovative new devices from the Chinese grey market. Kickstarter IoT products will be dead. IoT botnets fail to become larger than they are now, not because of regulation, but because most devices around the world are behind firewalls.

 

 

libra-sign-1.pngTeleparsons (aka: Trevor Parsons, Rapid7)

Relationships will blossom this year as more organizations look for synergies between their technology departments, namely IT and security. Connections will become deeper and more meaningful when these departments start thinking more about how they’re leveraging data, what tools are giving them the best visibility, etc., rather than accepting and managing several desperate solutions that aren’t necessarily helping to increase productivity. The year will not be without complexities for IT environments however, so monitoring tools will need to become more flexible and comprehensive in terms of data collection and correlation. Consolidation will result in more meaningful insights as we expect to see more technologies combining data sources (e.g. logs, metrics, endpoint data) to give a richer view into their environments.

 

 

sagittarius-zodiac-symbol.pngCyber Oracle Squirrel 

While 2017 may be the year of the rooster, for us it is of course always the year of the squirrel. We will continue our cyberwar operation disrupting your power over 400 times in 2017. Your pundits will continue to shout Cyberwar! from the podium and yet it is doubtful that any such cyber action will impact your power in 2017.

 

 

scorpion-shape-of-zodiac-sign.png

Craigvoyant Smith (aka: Craig Smith, Rapid7)

Much as a series of eclipses block Venus from influencing travel, we will see malware used to shut down a major transportation sector. I anticipate that the malware will be intentionally targeted to halt a transportation sector either for the purpose of ransomware or political gain. There will be a large uptick in hardware related security attacks. As security research increasingly bleeds into hardware, we will see creative ways to patch vulnerabilities when no update mechanism is readily available.

 

We will see the concept of an internal trusted network deteriorate. Internal networks will be treated the same as any external non-trusted network. With the increase of IoT devices, phishing attacks, and social engineering, even the concept of a corporate trusted laptop will need to be re-evaluated.

 

 

 

capricorn-goat-animal-shape-of-zodiac-sign.pngMystic Scutt (aka: Mike Scutt, Rapid7) 

Under Saturn’s watchful eye, we can expect breaches to take an earthier standpoint. We’re

expecting a significant uptick in "living off the land" style compromises and malware, a lot more script-based malware (powershell, js, vbs, etc.), and an increase in the use of native operating system tools to execute malware, persist, and perform recon.

When cybersecurity researchers find a bug in product software, what’s the best way for the researchers to disclose the bug to the maker of that software? How should the software vendor receive and respond to researchers’ disclosure? Questions like these are becoming increasingly important as more software-enabled goods - and the cybersecurity vulnerabilities they carry - enter the marketplace. But more data is needed on how these issues are being dealt with in practice.

 

Today we helped publish a research report [PDF] that investigates attitudes and approaches to vulnerability disclosure and handling. The report is the result of two surveys – one for security researchers, and one for technology providers and operators – launched as part of a National Telecommunications and Information Administration (NTIA) “multistakeholder process” on vulnerability disclosure. The process split into three working groups: one focused on building norms/best practices for multi-party complex disclosure scenarios; one focused on building best practices and guidance for disclosure relating to “cyber safety” issues, and one focused on driving awareness and adoption of vulnerability disclosure and handling best practices. It is this last group, the “Awareness and Adoption Working Group” that devised and issued the surveys in order to understand what researchers and technology providers are doing on this topic today, and why. Rapid7 - along with several other companies, organizations, and individuals - participated in the project (in full disclosure, I am co-chair of the working group) as part of our ongoing focus on supporting security research and promoting collaboration between the security community and technology manufacturers.

 

The surveys, issued in April, investigated the reality around awareness and adoption of vulnerability disclosure best practices. I blogged at the time about why the surveys were important: in a nutshell, while the topic of vulnerability disclosure is not new, adoption of recommended practices is still seen as relatively low. The relationship between researchers and technology providers/operators is often characterized as adversarial, with friction arising from a lack of mutual understanding. The surveys were designed to uncover whether these perceptions are exaggerated, outdated, or truly indicative of what’s really happening. In the latter instance, we wanted to understand the needs or concerns driving behavior.

 

The survey questions focused on past or current behavior for reporting or responding to cybersecurity vulnerabilities, and processes that worked or could be improved. One quick note – our research efforts were somewhat imperfect because, as my data scientist friend Bob Rudis is fond of telling me, we effectively surveyed the internet (sorry Bob!). This was really the only pragmatic option open to us; however, it did result in a certain amount of selection bias in who took the surveys. We made a great deal of effort to promote the surveys as far and wide as possible, particularly through vertical sector alliances and information sharing groups, but we expect respondents have likely dealt with vulnerability disclosure in some way in the past. Nonetheless, we believe the data is valuable, and we’re pleased with the number and quality of responses.

 

There were 285 responses to the vendor survey and 414 to the researcher survey. View the infographic here [PDF].

 

Key findings

Researcher survey

ntia-aag-2016-survey-infographic-block-2.png

  • The vast majority of researchers (92%) generally engage in some form of coordinated vulnerability disclosure.
  • When they have gone a different route (e.g., public disclosure) it has generally been because of frustrated expectations, mostly around communication.
  • The threat of legal action was cited by 60% of researchers as a reason they might not work with a vendor to disclose.
  • Only 15% of researchers expected a bounty in return for disclosure, but 70% expected regular communication about the bug.

 

Vendor survey

  • Vendor responses were generally separable into “more mature” and “less mature” categories. Most of the more mature vendors (between 60 and 80%) used all the processes described in the survey.
  • Most “more mature” technology providers and operators (76%) look internally to develop vulnerability handling procedures, with smaller proportions looking at their peers or at international standards for guidance.
  • More mature vendors reported that a sense of corporate responsibility or the desires of their customers were the reasons they had a disclosure policy.
  • Only one in three surveyed companies considered and/or required suppliers to have their own vulnerability handling procedures.

 

Building on the data for a brighter future

With the rise of the Internet of Things we are seeing unprecedented levels of complexity and connectivity for technology, introducing cybersecurity risk in all sorts of new areas of our lives. Adopting robust mechanisms for identifying and reporting vulnerabilities, and building productive models for collaboration between researchers and technology providers/operators has never been so critical.

 

It is our hope that this data can help guide future efforts to increase awareness and adoption of recommended disclosure and handling practices. We have already seen some very significant evolutions in the vulnerability disclosure landscape – for example, the DMCA exemption for security research; the FDA post-market guidance; and proposed vulnerability disclosure guidance from NHTSA. Additionally, in the past year, we have seen notable names in defense, aviation, automotive, and medical device manufacturing and operating all launch high profile vulnerability disclosure and handling programs. These steps are indicative of an increased level of awareness and appreciation of the value of vulnerability disclosure, and each paves the way for yet more widespread adoption of best practices.

 

The survey data itself offers a hopeful message in this regard - many of the respondents indicated that they clearly understand and appreciate the benefits of a coordinated approach to vulnerability disclosure and handling. Importantly, both researchers and more mature technology providers indicated a willingness to invest time and resources into collaborating so they can create more positive outcomes for technology consumers.

 

Yet, there is still a way to go. The data also indicates that to some extent, there are still perception and communication challenges between researchers and technology providers/operators, the most worrying of which is that 60% of researchers indicated concern over legal threats. Responding to these challenges, the report advises that:

 

“Efforts to improve communication between researchers and vendors should encourage more coordinated, rather than straight-to-public, disclosure. Removing legal barriers, whether through changes in law or clear vulnerability handling policies that indemnify researchers, can also help. Both mature and less mature companies should be urged to look at external standards, such as ISOs, and further explanation of the cost-savings across the software development lifecycle from implementation of vulnerability handling processes may help to do so.”

 

The bottom line is that more work needs to be done to drive continued adoption of vulnerability disclosure and handling best practices. If you are an advocate of coordinated disclosure – great! – keep spreading the word. If you have not previously considered it, now is the perfect time to start investigating it. ISO 29147 is a great starting point, or take a look at some of the example policies such as the Department of Defense or Johnson and Johnson. If you have questions, feel free to post them here in the comments or contact community [at] rapid7 [dot] com.

 

As a final thought, I would like to thank everyone that provided input and feedback on the surveys and the resulting data analysis - there were a lot of you and many of you were very generous with your time. And I would also like to thank everyone that filled in the surveys - thank you for lending us a little insight into your experiences and expectations.

 

~ @infosecjen

2016 has been a big year for information security, as we’ve seen attacks by both cybercriminals and state actors increase in size and public awareness, and the Internet of Things comes into its own as a field of study. But today we’d like to talk about a very old (but no less dangerous) type of attacker tool – web shells – and new techniques Rapid7 is developing for identifying them quickly and accurately.

 

What is a Web Shell?

 

Web shells are web-based applications that provide a threat actor with the ability to interact with a system – anything from file access and upload to the ability to execute arbitrary code on the exploited server. They’re written in a variety of languages, including PHP, ASP, Java and JavaScript, although the most common is PHP (since the majority of systems support PHP). Once they’re in your system, the threat actor can use them to steal data or credentials, gain access to more important servers in the network, or as a conduit to upload more dangerous and extensive malware.

 

Why should I care?

 

Because web shells can hit pretty much anyone. They most commonly show up on small business web presences, particularly Wordpress-powered sites, as Wordpress plugins and themes are a favoured target for web shell authors (since vulnerabilities show up in them all the time).

 

Wordpress isn't, of course, alone - virtually all web applications are released with vulnerabilities from time to time. So if you have a website that accepts and stores any kind of user input, from forum posts to avatar images, now is a fine time to learn about web shells, because you could very well be vulnerable to them.

 

How Web Shells Work and How They’re Used

 

The first step with a web shell is uploading it to a server, from which the attacker can then access it. This “installation” can happen in several ways, but the most common techniques involve:

 

  • exploiting a vulnerability in the server’s software,
  • getting access to an administrator portal, or
  • taking advantage of an improperly configured host.

 

As an example, Rapid7’s Incident Response Team has dealt with several engagements where the attackers took advantage of a vulnerability in a third-party plugin used by a customer’s CMS enabling them to upload a simple PHP web shell.

 

Once a web shell is uploaded, it’s used to exploit the system. What this looks like differs from actor to actor, and from web shell to web shell, because shells can come with a variety of capabilities. Some are very simple and simply open a connection to the outside world, allowing an actor to drop in more precise or malicious code, and then execute whatever they receive. Others are more complex and come with database or file browsers, letting the attacker rifle through your code and data from thousands of miles away.

 

 

Whatever the design, web shells can be both extremely dangerous and common – US-CERT has identified them as a regularly used tool of both cyber-criminals and Advanced Persistent Threats (APTs). If they’re not detected and eliminated, they can provide an attacker with not only a solid, persistent backdoor into your environment but potentially root access, depending on what they compromise.

 

Web Shell Detection

 

Web shells aren’t new, and people have spent a lot of time working to detect and halt them. Once the breach of a system is discovered, it’s fairly straightforward (although time consuming) to just go through the server looking at the upload and modification dates of files, relative to the discovery date, and manually check suspicious-looking uploads to see if they’re the source of the problem. But what about detecting web shells before they’re used to cause harm?

 

There are a couple of ways of doing it. One approach is to have an automated system look at the contents of newly uploaded or changed files and see if they match a known web shell, just as antivirus software does with other forms of malware. This works well if an attacker is using a known web shell, but quickly falls apart when confronted with custom code.

 

Another technique is to use pattern matching to look for code fragments (down to the level of individual function calls) that are commonly malicious, such as calls out to the system to manipulate files or open connections. The problem there is that web shell authors are fully aware of this technique, and deliberately write their code in a very opaque and confusing way that makes pattern matching extraordinarily difficult to do with any real accuracy.

 

A Better Way

 

If we can detect web shells, we can stop them, and if we can stop them, we can protect our customers – but as you see, all the existing approaches have some pretty severe drawbacks. Meaning they miss a lot.

Rapid7 Labs has been working on a system that uses data science to classify web shell threats based on static and dynamic analysis of PHP files. In a static analysis context, our classifier looks for both  dangerous looking function calls and file signatures plus coding techniques that developers simply wouldn’t do if they were writing legitimate, production ready code – things that only appear when the developer is trying to hide their purpose. In a dynamic analysis context the potentially malicious file is executed on a monitored, standalone system so our classifier can see what it does.


The results from both these methods are then fed into a machine learning model, which predicts whether the file is malicious or not, and the accuracy rate has been extremely promising, with the system detecting 99% of the hundreds of web shells we’ve tested it on, including custom, single use shells, with only a 1% false-positive rate. The result is that (to broadly generalize), if our AR team is faced with 1,000 files, they only need to manually check 10. (For those ML-nerds out there, yes we’ve checked for over-fitting.)

 

In the future we hope to use the system to pre-emptively detect web shells, identifying and isolating them before they exploit the system. Until that point, it’s being used by our managed detection and response team, letting them identify the source of customer breaches far more quickly than teams relying solely on traditional, arduous and error-prone manual methods.

 

Oliver Keyes

Senior Data Scientist

 

Tim Stiller 

Senior Software Engineer

Inmates running the asylum. The fox guarding the henhouse. You've no doubt heard these terms before. They’re clever phrases that highlight how the wrong people are often in charge of things. It's convenient to think that the wrong people are running the show elsewhere but have you taken the time to reflect inward and determine how this very dilemma might be affecting your organization? I see this happening all the time in terms of security assessments. In organizations both large and small, I see improper testing - or no testing at all - of the systems that should otherwise be in scope and fair game for assessment. The very people in charge of security assessments are the ones who are determining how things are going to turn out.

I see everyone from CIOs to network admins and all of the IT/security roles in between setting parameters on what can and cannot be tested. Ditto for how it can be tested. Oftentimes, an external security assessment/penetration test is performed but not everything is being looked at. Sometimes it's something relatively benign like a marketing website but other times it's actual enterprise applications that are being overlooked (often in the name of “someone else is hosting it and therefore responsible for it"). Other times, I hear people stating that their auditors or customers aren't asking for certain systems to be tested, therefore, it doesn't really matter.

 

I think the general consensus of the stakeholders who are reading the actual assessment reports is that they are getting the current status of everything. But that's not the case in many situations. There’s no doubt that people need what they need and nothing more. In fact, legal counsel would likely advise to do the bare minimum, document the bare minimum, and share the bare minimum. That’s just how the world works. At the end of the day, people presumably know what they want/need. I'm just not convinced that the approaches that are currently being taken whereby IT and security staff are defining the systems to be tested along how they need to be tested (and, therefore, the outcomes) is the best approach.

 

Most security assessment reports have a notes/scope section that outlines what was tested and what was not. However, what's often missing is all of the details regarding other things that people don't often think about in this context such as:

  • How the systems that may not have been tested can/will impact those that were tested if they have exploitable vulnerabilities
  • Whether or not authentication was used (it’s important to distinguish the two – and do both)
  • What network security controls, i.e. firewall and IPS, were disabled or left in place (you have to look past any blocking)
  • What level of manual analysis/penetration testing was performed, including how much time was spent and whether or not social engineering/email phishing were a part of the testing (it makes a difference!)

 

There are so many caveats associated with modern-day security testing that no one really knows if everything has been looked at in the proper ways. So, what do you do? Do you question the validity of existing testing methods and reports? Do you step back and ask tougher questions of those who were doing the testing? Perhaps there needs to be an arm's-length entity involved with defining what gets tested and how it gets tested, including very specific approaches, systems in scope, and tools that are to be used.

 

This challenge is similar to certain aspects of healthcare – something we can all relate to. Take, for instance, when a patient gets a MRI or CAT scan, the results for the radiologist to analyze will be different than a more focused x-ray or ultrasound. Perhaps the prescribing doctor thinks the patient just needs an x-ray when, in fact, they actually need a PET scan. That very thing happened to my mother when she was fighting lung cancer. Her doctors focused on her lungs and hardly anything else. Her chemotherapy was working well and her lungs continued to look good over time. What was missed, however, was the cancer that had spread to other parts of her body. The prescribed diagnostics were focused on what was thought to be important but they completely missed what was going on in the rest of her body. Unfortunately, given how much time had passed for the cancer to spread elsewhere (while being overlooked), the outcome was not a positive one. Similarly, it’s important to remember that any security testing that’s assumed to paint the entire picture may not be doing that at all.

 

Are the inmates running the asylum? Is the fox guarding the henhouse? Perhaps a bit here and there. However, it’s not always people in IT and security intentionally “limiting” the scope of security testing by keeping certain systems out of the loop and looking the other way to generate false or misrepresented outcomes. I do know that is going on in many situations but this issue is a bit more complicated. I don't think there’s a good solution to this other than closer involvement on the part of management, internal auditors, and outside parties involved with scoping and performing these assessments. If anything, the main focus should be ensuring that expectations are properly set.

 

A false sense of security is the enemy of decision makers. Never, ever should someone reading a security assessment report assume that all the proper testing has been done or that the report is a complete and accurate reflection of where things truly stand. Odds are that’s it not.

Recently we all have found ourselves talking about the risk and impact of poorly secured IoT technology and who is responsible. Fact is there is enough blame to go around for everyone, but let's not go there. Let us start focusing on solutions that can help secure IoT technology.

 

Usability has been an issue that has plagued us since the beginning of time. As an example, just going back to my youth and seeing my parents VCR flashing 12:00 all the time. We laugh at that, because it showed us their lack of understanding around how technology works, and of course it was not a real risk to anything other then knowing what time it is and not being able to preset and record shows. Today, the inability to understand or configure our technology is much more of a risk than the flashing 12:00 on our parents' VCR. Such misconfigured IoT devices can lead to various compromises of our information or allow our technology to be used in attacks against others. Currently we often find IoT devices working out of the box with every feature enabled and also using default passwords, and of course this approach has come back to haunt us in a number of cases.

 

I am sure we all agree that the days of every feature being enabled and default passwords out the box needs to change. Although, don’t get me wrong, I still think IoT technology should be easy to deploy -- but with security built in. So what should that look like? Let me break it down in a few basic items that I think are paramount to getting to that point.

 

  • password.pngNo default passwords enabled. It is easy enough during deployment of a product to have the user set a strong password. In cases where each device has a unique default password, those should also be changed and if not, the user should be warned that the password has not been changed and then forced to acknowledge that with a multi-step process each time they use the products. Check out this new tool built by the Metasploit team at Rapid7 called IoTSeeker which scans your network for IoT devices and let's you know if the default password is being used.

 

  • checkbox.pngInitial installation should only enable needed services for functionality. These extra services should only be configurable after initial setup. A check box list for features during initial setup is not the way to go. It will only lead to user selecting every one of them just to make sure the installation works. I know, because in a past life I have watched coworkers do exact thing during product installations.

 

 

  • documents-icon-64257.pngGood documentation is critical for walking a user through the secure setup. This documentation, beside covering standard setups and which may include an intuitive web wizard, should include guidance on enabling expanded features or specific services that have security implications the user should be made aware of during setup. With the ever-expanding list of capabilities and features associated with IoT its imperative that the end user is given guidance "Good Documentation" to help with selecting and implementing the most secure methods during setup of their device.

 

  • patching.pngAutomated firmware patching should be the default. If not, the user should be prompted every time they use the product when firmware updates are available. Patching allows us to fix security issues within the products moving forward. We are always going to have problems and having the ability to correct them on the fly is important.

 

 

 

This simple list points out items that create a solid foundation from where we can continue building on IoT security and at the same time maintain a solid resemblance of usability; however, I expect it will still take a while before we see these items commonplace within all new IoT -- and I am looking forward to that day.

 

If you are looking for a second opinion on how you should be securing the IoT devices used within your environment, check out our IoT security services.

by Tod Beardsley and Bob Rudis

 

What's Going On?

 

Early in November, a vulnerability was disclosed affecting Zyxel DSL modems, which are rebranded and distributed to many DSL broadband customers across Europe. Approximately 19 days later, this vulnerability was leveraged in widespread attacks across the Internet, apparently connected with a new round of Mirai botnet activity.

 

If you are a DSL broadband customer, you can check to see if your external TCP port 7547 is accessible to the Internet by using popular public portscanning services provided by WhatsMyIP, SpeedGuide, or your own favorite public portscanning service. If it is, your ISP should be able to confirm if your modem is vulnerable to this attack.

 

Vulnerability Details

 

On November 7, "Kenzo" disclosed two vulnerabilities affecting the Zyxel D1000 DSL modem on the Reverse Engineering blog, here. This DSL modem is used by residential DSL subscribers in Ireland, and appears to be distributed by the Irish ISP, Eir. It's unknown if Kenzo disclosed these issues to either Zyxel or Eir prior to public disclosure.

 

Two issues were identified, both involving the TR-064 SOAP service on the DSL modem, running on TCP port 7547. The first is a command injection vulnerability in the way the device parses new NTP server configurations, where an attacker can enclose an arbitrary shell command in backticks when setting the NewNTPServer1 parameter. The second is an information leak vulnerability where an attacker can access the GetSecurityKeys command to learn the device's WiFi and administrative password.

 

Kenzo provided a proof-of-concept Metasploit module to exercise these vulnerabilities to expose the administrative web service on the Internet-facing side of the modem and to extract the administrative password to that admin web service.

 

On November 26th, the command injection issue was being actively exploited in the wild, apparently as part of another wave of Mirai-style IoT botnet activity. In particular, DSL modems provided to Deutsche Telekom customers in Germany and Austria, under the brandname "Speedport," appeared to be vulnerable. As a result of this attack, many Telekom subscribers were knocked offline on November 27th, and DT has since updated the Speedport firmware.

 

Today, on November 29th, the Metasploit open source project has started work on converting Kenzo's original, special purpose proof-of-concept exploit to a more generally useful Metasploit module that can be used to test the vulnerability in a safer and more controlled way. That work continues on Pull Request #7626.

 

Exploit Activity Details

 

timeline3.png

 

Rapid7’s Heisenberg Cloud started picking up malicious SOAP HTTP POST requests to port 7547 on November 26th. We were able to pick up these requests due to the “spray and pray” nature of the bots searching for vulnerable targets. To-date, we’ve seen over 63,000 unique source IP addresses associated with these attempts to take over the routers, peaking at over 35,000 unique attempts per day on November 27th.

 

As the map below shows, the bots attempting to take over the routers are geographically dispersed.

 

map.png

As the below temporal flow diagram shows, different regions are more prevalent as sources of this activity on different days (the source regions are on the left, the days they were active are on the right).

 

region.png

There was little change in the top 10 source countries (by unique node count) over the attack period, but some definitely stood out more than others (like Brazil and the U.K.).

 

We’ve also seen all malicious payload types, though not all of them appeared on all days as seen in the third chart.

 

cc.png

payloads.png

Not all payloads were evenly distributed across all countries:

 

heatmap.png

 

What Can You Do to Protect Yourself?

 

The vulnerabilities being exploited in these attack are present in Zyxel DSL modems, which are commonly used in European and South American consumer markets, and may be rebranded by regional ISPs, as they are for Eir and Deutsche Telekom. The vulnerabilities described do not appear to affect the cable modems commonly used in the United States.

 

If you are a DSL customer and concerned that you may be vulnerable, you can use popular portscanning services provided by WhatsMyIP, SpeedGuide, or others to assess if your external TCP port 7547 is accessible from the Internet. If the port times out or is otherwise not reachable, you're in the clear. If it is accessible, you should contact your ISP to see if a) this can be restricted, and b) if you are running a vulnerable device.

 

For ISPs, it is A Bad Idea to expose either TR-069 or TR-064 services to arbitrary sources on the Internet; while ISPs may need access to this port to perform routine configuration maintenance on customer equipment, it should be possible for local and edge firewall rules to restrict access to this port to only IP addresses that originate from the ISP's management network.

 

Meanwhile, penetration testers should follow the progress of converting Kenzo's original proof-of-concept to a fully functional Metasploit module over at Pull Request #7626 on Metasploit's GitHub repository. If you are an open source security software developer, we'd love to have your input there.

 

Update (2016-Nov-30): This blog post originally referred to the vulnerable service as TR-069, but it's since become clear this issue is in a related service, TR-064.

Last week, some important new developments in the way the US government deals with hackers were unveiled: the first ever vulnerability disclosure policy of the Department of Defense. Touted by Secretary Ash Carter as a ‘see something, say something’ policy for the digital domain, this not only provides guidelines for reporting security holes in DoD websites, it also marks the first time since hacking became a felony offense over 30 years ago, that there is a legal, safe channel for helpful hackers who want to help secure DoD websites, a way to do so without fear of legal prosecution.

 

This is historic.

 

We don’t often see this type of outreach to the hacker community from most private organizations, let alone the US government. In a survey of the Forbes Global 2000 companies last year, only 6% had a public method to report a security vulnerability. That means 94% of well-funded companies that spend millions on security today, buying all kinds of security products and conducting “industry best practice” security measures and scans, have not yet established a clear public way to tell them about the inevitable: that their networks or products still have security holes, despite their best efforts.

 

The simple fact that all networks and all software can be hacked isn’t a surprise to anyone, especially not attackers. Breach after breach is reported in the news. Yet the software and services upon which the Internet is built have never had the broad and consistent benefit of friendly hacker eyes reporting security holes to get them fixed. Holes go unplugged, breaches continue.

 

This is because instead of creating open door policies for vulnerability disclosure, too many organizations would rather postpone having to deal with it, often until it’s too late. Instead, helpful hackers who “see something,” often don’t “say something” because they were afraid that it might land them in jail.

 

I myself as a hacker have observed and not reported vulnerabilities in the past that I stumbled upon outside of a paid penetration test because of that very real fear. I’ve built vulnerability disclosure programs at two major vendors (Symantec and Microsoft), in order to shield employees of those companies who found other organizations’ vulnerabilities from the concerns that they may face an angry bug recipient alone.

 

Even then, wearing the mighty cape of a mega corporation, I and others trying to disclose security holes to other organizations encountered the same range of reactions that independent researchers face: from ignoring the report, to full on legal threats, and one voicemail that I wish I’d saved because I learned new swear words from it. For me, that is rare. But what wasn’t rare was the fear that often fuels those negative reactions from organizations that haven’t had a lot of experience dealing with vulnerability disclosure.

 

Fear, as they say in Dune, is the mind-killer. Organizations must not fear the friendly hacker, lest they let the little death bring total oblivion.

 

There is no excuse for organizations letting fear of working with hackers prevent them from doing so for defense. There is no excuse for lacking a vulnerability disclosure policy, in any organization, private or public sector.

 

The only barrier is building capabilities to handle what can be daunting in terms of facing the world of hackers. Big companies like Google, Apple, and Microsoft have had to deal with this issue for a very long time, and have worked out systems that work for them. But what about smaller organizations? What about other industries outside of the tech sector? What about IoT? And what about governments, who must walk the line between getting the help they need from the hacker community without accidentally giving free license to nation-states to hack them with an overly permissive policy?

 

There are guidelines for this process in the public domain, too many to list. 2017 will mark my ninth year attending ISO standards meetings, where I’ve dedicated myself to helping create the standards for ISO 29147 Vulnerability disclosure, and ISO 30111 Vulnerability handling processes. Until April of 2016, both of these standards were not available for free. Now the essential one to start with, ISO 29147, is available for download from ISO at no cost. Most people don’t even know it exists, let alone that it’s now free. But both standards act as a guide for best practices, not a roadmap for an organization to start building their vulnerability disclosure program, bit by bit.

 

Enter the first Vulnerability Coordination Maturity Model – a description of 5 capability areas in vulnerability disclosure, that I designed to help organizations gauge their readiness to handle incoming vulnerability reports. These capabilities go beyond engineering, and look at an organization’s overall strengths in executive support, communication, analytics, and incentives.

The VCMM provides an initial baseline, and a way forward for any organization, small or large, public or private, that wants to confront their fear of working with friendly hackers, in defense against very unfriendly attackers.

 

The model was built over my years of vulnerability disclosure experiences, on all sides of that equation. I’ve done so in open source and closed source companies, as the hacker finding the vulnerability or as the vendor responding to incoming vulnerabilities, and as the coordinator between multiple vendors in issues that affected shared libraries, many years before Heartbleed was a common term heard around the dinner table.

 

I was fortunate to be able to present this Vulnerability Coordination Maturity Model at the Rapid7 UNITED Summit a few weeks ago, and my company was honored to work directly with the Department of Defense on this latest bit of Internet history. And though I’m known for creating Microsoft’s first ever bug bounty programs, and advised the Pentagon on the first ever bug bounty program of theirs, now my work focuses more heavily on core vulnerability disclosure capability-building, and helping organizations overcome their fears in dealing with hackers.

 

The way I see it, if 94% of Forbes Global 2000 is still lagging behind the US government in its outreach to helpful hackers, my work is best done far earlier in an organization’s life than when they are ready to create cash incentives for bugs. In fact, not everyone is ready for bug bounties, not public ones anyway, unless they have the fundamentals of vulnerability disclosure ready. But that’s a topic for another day.

 

Today, as we bear witness to a significant positive shift in the US government’s public work with hackers, I’m filled with hope. Hope that the DoD’s new practice of vulnerability disclosure programs and bounties will expand as a successful model to the rest of the US government, hope that other governments will start doing this too, hope that the rest of the Forbes top 2000 will catch up, and hope for every interconnected device looming on the Internet of Things to come.

 

Today, we have no time to fear our friends, no matter where in the world or on the Internet they come from. There is no room for xenophobia when it comes to defending the Internet. Together, we must act as defenders without borders, without fear.

We live in an interesting time for research related to Internet scanning.

 

There is a wealth of data and services to aid in research. Scanning related initiatives like Rapid7's Project Sonar, Censys, Shodan, Shadowserver or any number of other public/semi-public projects have been around for years, collecting massive troves of data.  The data and services built around it has been used for all manner of research.

 

In cases where existing scanning services and data cannot answer burning security research questions, it is not unreasonable for one to slap together some minimal infrastructure to perform Internet wide scans.  Mix the appropriate amounts of zmap or masscan with some carefully selected hosting/cloud providers, a dash of automation, and a crash-course in the legal complexities related to "scanning" and questions you ponder over morning coffee can have answers by day's end.

 

So, from one perspective, there is an abundance of signal.  Data is readily available.

 

There is, unfortunately, a significant amount of noise that must be dealt with.

 

Dig even slightly deep into almost any data produced by these scanning initiatives and you'll have a variety of problems to contend with that can waylay researchers. For example, there are a variety of snags related the collection of the scan data that could influence the results of research:

 

  • Natural fluctuation of IPs and endpoint reachability due to things like DHCP, mobile devices, or misconfiguration.
  • When blacklists or opt-out lists are utilized to allow IP "owners" to opt-out from a given project's scanning efforts, how big is this blacklist?  What IPs are in it?  How has it changed since the last scan?
  • Are there design issues/bugs in the system used to collect the scan data in the first place that influenced the scan results?
  • During a given study, were there routing or other connectivity issues that affected the reachability of targets?
  • Has this data already been collected?  If so, can that data be used instead of performing an additional scan?

 

Worse, even in the absence of any problems related to the collection of the scan data, the data itself is often problematic:

 

  • Size.  Scans of even just a single port and protocol can result in a massive amount of data to be dealt with.  For example, a simple HTTP GET request to every 80/TCP IPv4 endpoint currently results in a compressed archive of over 75G.  Perform deeper HTTP 1.1 vhost scans and you'll quickly have to contend with a terabyte or more.  Data of this size requires special considerations when it comes to the storage, transfer and processing.
  • Variety.  From Internet-connected bovine milking machines, to toasters, *** toys, appliances and an increasingly large number of "smart" or "intelligent" devices are being connected to the Internet, exposing services in places you might not expect them.  For example, pick any TCP port and you can guarantee that some non-trivial number of the responses will be from HTTP services of one type or another.  These potentially unexpected responses may need to be carefully handled during data analysis.
  • Oddities.  There is not a single TCP or UDP port that wouldn't yield a few thousand responses, regardless of how seemingly random the port may be.  12345/TCP?  1337/UDP?  65535/TCP?  Sure.  You can believe that there will be something out there responding on that port in some way.  Oftentimes these responses are the result of some security device between the scan source and destination.  For example, there is a large ISP that responds to any probe on any UDP port with an HTTP 404 response over UDP.  There is a vendor with products and services used to combat DDoS that does something similar, responding to any inbound TCP connection with HTTP responses.

 

Lastly there is the issue of focus.  It is very easy for research that is based on Internet scanning data to quickly venture off course and become distracted. There is seemingly no end to the amount of strange things that will be connected in strange ways to the public IP space that will tempt the typically curious researcher.

 

Be careful out there!

In my last blog post, we reviewed the most prevalent detection strategies and how we can best implement them. This post dives into understanding how to catch what our other systems missed, using attacker behavior analytics and anomaly detection to improve detection.

 

Understand Your Adversary – Attack Methodology Detection

Contextual intelligence feeds introduce higher fidelity and the details needed to gain insight into patterns of attacker behavior. Attackers frequently rotate tools and infrastructure to avoid detection, but when it comes to tactics and techniques, they often stick with what works. The methods they use to deliver malware, perform reconnaissance, and move laterally in a network do not change significantly.

 

A thorough understanding of attacker methodology leads to the creation and refinement of methodology-based detection techniques. Knowledge of applications targeted by attackers enables more focused monitoring of those applications for suspicious behaviors, thus optimizing the effectiveness and efficiency of an organization’s detection program. An example of application anomaly-based detection is webshells on IIS systems:

 

w3wp.PNG

 

It is anomalous for IIS to spawn a command prompt, and the execution of “whoami.exe” and “net.exe” indicate likely reconnaissance activity. By understanding the methods employed by attackers we generate detections that will identify activity without relying on static indicators such as hashes or IPs. In this case we are using the low likelihood of IIS running CMD and the rare occurrence of CMD executing ‘whoami’ and ‘net [command]’ to drive our detection of potential attacker activity.

 

Additionally, attackers must reconnoiter networks both internally and externally to identify target systems and users. Reviewing logs for unusual user-to-system authentication events, suspicious processes (for example, ‘dsquery’, ‘net dom’, ‘ping –n 1’, and ‘whoami’), especially over abbreviated time periods, can provide investigative leads to identify internal reconnaissance.

 

Even without a constant stream of real-time data from endpoints, we can model behavior and identify anomalies based upon the frequency of an item across a group of endpoints. By gathering data on persistent executables across a network, for example, we can perform frequency analysis and identify rare or unique entries for further analysis. Simple techniques like frequency analysis will often reveal investigative leads from prior (or even current) compromises, and can be applied to all manner of network and endpoint data.

 

Expanding beyond a reliance primarily on traditional static indicator-based detection and adding a focus on attacker behavior increases the likelihood of detecting previously unknown malware and skilled attackers. A culmination of multiple detection strategies is necessary for full coverage and visibility: proactive detection technology successfully blocks known-bad, contextual intelligence assists in identifying less common malware and attackers, and methodology-based evidence gathered from thorough observation provides insight into potential indicators of compromise.

 

 

Use the Knowledge You Have

IT and security staff know their organization’s users and systems better than anyone else. They work diligently on their networks every day ensuring uptime of critical components, enablement of user actions, and expedient resolution of problems. Their inherent knowledge of the environment provides incredible depth of detection capabilities. In fact, IT support staff are frequently the first to know something is amiss, regardless if the problem is caused by attacker activity.  Compromised systems may often exhibit unusual symptoms and become problematic for users, who report the problems to their IT support staff.

 

Environment-specific threat detection is Rapid7’s specialty. Our InsightIDR platform continuously monitors user activity, authentication patterns, and process activity to spot suspicious behavior. By tracking user authentication history, we can identify when a user authenticates to a new system, over a new protocol, and from a new IP. By tracking the processes executed on each system we can identify if a user is running applications that deviate from their normal patterns or if they are running suspicious commands (based on our knowledge of attacker methodology). Combining user authentication with process execution history ensures that even if an attacker accesses a legitimate account, his tools and reconnaissance techniques will give him away. Lastly, by combining this data with threat intelligence from previous findings, industry feeds, and attacker profiles we ensure that we prioritize high-fidelity investigative leads and reduce overall detection time, enabling faster and more effective response.

 

Let’s walk through an example: Bob’s account is compromised internally:

 

After compromising the system, an attacker would execute reconnaissance commands that are historically dissimilar to Bob’s normal activity. Bob does not typically run ‘whoami’ on the command line or execute psexec, nor has Bob ever executed a powershell command – those behaviors are investigative elements that individually are not significant enough to alert on, but in aggregate present a trail of suspicious behavior that warrants an investigation.

 

Knowledge of your environment and what is statistically ‘normal’ per user and per system enables a ‘signature-less’ addition to your detection strategy. Regardless of the noisy and easily bypassed malware hashes, domains, IPs, IDS alerts, firewall blocks, and proxy activity your traditional detection technology provides, you can identify attacker activity and ensure that you are not missing events due to stale or inaccurate intel.

 

Once you have identified an attack based on user and system anomaly detection, extract useful indicator data from your investigation and build your own ‘known-bad’ threat feed. By adding internal indicators to your traditional detection systems, you have greater intel context and you can simplify the detection of attacker activity throughout your environment. Properly combining detection strategies dramatically increases the likelihood of attack detection and provides you with the context you need to differentiate between ‘weird’, ‘bad’, and ‘there goes my weekend’.

Filter Blog

By date: By tag: