Skip navigation
All Places > InsightOps > Blog

InsightOps

11 posts

The Need for Behavior Driven Development (BDD) Testing Tools

It should come as no surprise to learn that testing is at the

practical-examples-of-behavior-driven-dev-testing

heart of our engineers' daily activities. Testing is intrinsic to our development process, both in practical terms and in our thinking. Our engineers work with complex systems that are made up of complex components. Individual components may have many external dependencies.

 

When testing, the scope of what is to be tested is important – it can be system wide, focused on a particular feature or down deep into the methods and classes of the code. To be able to focus our testing, we want to be able to mimic or ‘mock’ the behavior of external dependencies using a BDD testing tool.

The purpose of this post is to walk through a couple of simple code examples and provide an overview of and explain the need for Behavior Driven Development (BDD) testing.

 

BDD, Acceptance Tests, and Automation

At Rapid7 we apply the BDD methodology which is an extension of Test Driven Development (TDD). BDD and TDD both advocate that tests should be written first, which for BDD this means acceptance tests (ATs), followed by unit tests driven by the ATs. For now, let's say that at the outset of any task, BDD focus is on capturing the required behavior in User Stories, and from these acceptance tests (ATs) are written.

 

Over time a large number of ATs are generated. Therefore not only is the methodology important but also the supporting tools to automate and manage our work. For some background on this, another colleague, Vincent Riou has described theautomated testing, continuous integration and code-quality control toolsthat we use.

 

BDD Testing Example: Ubiquitous Language and AT Scenarios

To borrow from Vincent’s post, “The idea with acceptance testing is to write tests (or behavioral specifications) that describe the behavior of your software in a language which is not code but is more precise than standard English."

Doing this allows people who are not software engineers, but have knowledge of the requirements, such as Product Management or Marketing, to write the scenarios that make up our ATs. This is a powerful thing when it comes to capturing the required behavior. People in the BDD community sometimes refer to this as a ‘Ubiquitous Language’.

Again borrowing from what Vincent states “Additionally, those tests can be run using a parser which will allow you to easily match your language to functions in the programming language of your choice."

 

Here is an example AT scenario – in this case following a template and language constructs used by the Cucumber / Gherkin parser.

Given the customer has logged into their current account 
And the balance is shown to be 100 euros
When the customer transfers 75 euros to their savings account
Then the new current account balance should be 25 euros

 

Example step definition in Python

The following is an example of mapping a step definition to a Python function. In this case, the final step Then is shown. This is where an ‘assert’ is used to verify if the AT will pass or fail, depending on the final account balance.

 

@Then('^the new current account balance should be "([^"]*)" euros$')
def the_new_current_account_balance_should_be(self, expected_bal):
  expected_bal = int(expected_bal)
  assert expected_bal >= 0, "Balance cannot be negative"
  new_bal = get_balance(account_id)
  assert int(new_bal) == expected_bal, "Expected to get %d euros. Instead got %d euros" % (new_bal, expected_bal)

 

view rawEgBehaviorDrivenDevelopmentTesting-stepDef.py hosted with ❤ by GitHub

 

Since we are writing our tests before the actual implementation of the behavior, the AT will fail – so it's important that the error message thrown by the ‘assert’ is meaningful. Remember also that an AT may fail at a future date if some behavior of the ‘system under test’ (SUT) is modified, intentionally or not – this is part of the value of having a body of automated ATs.

 

Mocking Behavior of External Dependencies

The components and sub-systems that we work with have many external dependencies that can be complex. When running an AT against a particular component, it may be necessary to mock the external dependencies of that component. This is different from using a framework as described below in unit testing. Instead this is about trying to mimic the behavior of a second black-box so we can test the behavior of the first black-box.

 

In our work we encounter this all the time, especially where a SUT has a dependency on the behavior of an external server. One approach for example is to build a simple mock server in Python using the Bottle module, that gives us a basic server to build on. We mock the behavior that is required to meet the needs of the SUT. Note that this is not building a duplicate of an existing component – we are trying to mimic the behavior as seen by the SUT to complete our testing.

 

BDD Testing Example: Unit Testing

After completing the acceptance tests come the unit tests. These are more closely coupled with the code of the final implementation, although at this stage we still do not start our implementation until the required unit tests are in place. This approach of acceptance tests and unit tests are applicable to GUIs.

 

Unit Testing Example: Mocking with some JSON

The following example is a combination of using the Junit framework with the Mockito library to create mock objects. In this example we want to show in a simple way a technique to mock a response that contains data in JSON format from a GET request on some external server. The test data, in JSON format, can be an actual sample captured in a live production scenario. We are also going to use a Google library to help with handling the JSON file.

 

In this simple example we are testing a method ‘getCountOfStudents’, found in a data access class, that is used by our imaginary application to get the number of students on a course using that course ID. The actual details for that course is held on some database externally – for the purposes of testing we don’t care about this database.

 

What we are interested in, however, is that the method ‘getCountOfStudents’ will have a dependency on another piece of code – it will call ‘jsonGetCourseDetails’ which is found in an object called ‘HttpClient’ – as the name implies this object is responsible for handling HTTP traffic to some external server – and it is from this server our application gets course data. For our test to work we therefore need to mimic the response from the server – which returns the data in JSON format – which means we want to mock the response of the ‘jsonGetCourseDetails’.

 

The following code snippets come from a Junit Test Class, that is testing the various methods found in the class that defines our data access object. Note the required imports for the Mockito and Google libraries are added.

 

import static org.mockito.Matchers.any;
import static org.mockito.Matchers.eq;
import static org.mockito.Mockito.*;
import com.google.common.io.Resources;
import com.google.common.base.Charsets;

 

view rawEgBehaviorDrivenDevelopmentTesting-JunitImport.java hosted with ❤ by GitHub

 

Prior to running the test a mock object of the HttpClient is created using the test class ‘setup()’ method, and tidied up afterwards with ‘teardown()’.

 

private HttpClient httpClient;
@Before
public void setup() {
  apiClient = mock(HttpClient.class);
  ...
  ...
}
@After
public void teardown() {
  reset(httpClient);
  ...
  ...
}

 

view rawEgBehaviorDrivenDevelopmentTesting-JunitHttpClient.java hosted with ❤ by GitHub

 

For the test method itself, we use the Mockito when, so when the ‘jsonGetCourseDetails’ on the mock ‘httpClient’ object is called with the ‘course_id’, it then returns a mock response. We create the mock response using some test data, in JSON, we have in a file ‘course_details.json’.

 

@Test
public void testGetCountOfStudentsWithCourseID() throws IOException {
  private String course_id = "CS101";
  when(httpClient.jsonGetCourseDetails(eq(course_id))
  .thenReturn(getMockResponse("./tests/course/course_details.json"));
  Integer count = dao.getCountOfStudents(course_id);
  Assert.assertEquals(10, count);
}

 

view rawEgBehaviorDrivenDevelopmentTesting-JunitJSONrep.java hosted with ❤ by GitHub

 

To create the mock response there is a utility method we have written that uses the Google library ‘Resources’ class. For this example the method simply returns a mock response as the String from the ‘Resources.toString’.

 

private String getMockResponse(String jsonResource){
  String MockResponse = null;
  try {
  MockResource = Resources.toString(Resources.getResource(jsonResource), Charsets.UTF_8);
  }catch(IOException ex){
  ex.printStackTrace();
  }
  return MockResponse;
}

 

view rawEgBehaviorDrivenDevelopmentTesting-JunitRepString.java hosted with ❤ by GitHub

 

At this stage we have a unit test with a mock object and we can use data in JSON format. Of course, also at this stage the test method will fail. Why? Because the implementation of the ‘dao.getCountOfStudents(course_id)’ has not yet been done! We are writing our tests first, mocking the external dependencies (behavior) our code is reliant on.

 

When writing the code for the implementation, we will know we are finished when all the tests are passing.

 

Ready to Implement BDD Testing?

Try our free log management tool today.

Analyzing System Performance Using Log Data

Recently we examined some of the most common behaviors that our community of 25,000 users looked for in their logs, with a particular focus on web server logs. In fact, our research identified the top 15 web server tags and alerts created by our customers—you can read more about these in our https://logentries.com/doc/community-insights/ section—and you can also easily create tags or alerts based on the patterns to identify these behaviors in your systems log data.

 

This week we are focusing on system performance analysis using log data. Again we looked across our community of over 25,000 users and identified five ways in which people use log data to analyze system performance. As always, customer data was anonymized to protect privacy. Over the course of the next week, we will be diving into each of these areas in more detail and will feature customers' first-hand accounts of how they are using log data to help identify and resolve issues in their systems and analyze overall system performance.

 

Our research looked at more than 200,000 log analysis patterns

from across our community to identify important

Top Uses of Log Data for System Performanceevents in log data. With a particular focus on system performance related issues, we identified the following five areas as trending and common across our user base.

 

5 Key Log Analysis Insights

 

1. Slow Response Times

Response times are one of the most common and useful system performance measures available from your log data. They give you an immediate understanding of how long a request is taking to be returned. For example, web server log data can give you insight into how long a request takes to return a response to a client device. This can include time taken for the different components behind your web server (application servers, DBs) to process the request, so it can offer an immediate view of how well your application is performing. Recording response times from the client device/browser can give you an even more complete picture since this also captures page load time in the app/browser as well as network latency.

 

A good rule of thumb when measuring response times is to follow the 3 response time limits as outlined by Jakob Nielsen in his publication on ‘Usability Engineering’ back in 1993 (still relevant today). 0.1 second is about the limit for having the user feel that the system is reacting instantaneously, 1.0 second is about the limit for the user’s flow of thought to stay uninterrupted, and 10 seconds is about the limit for keeping the user’s attention focused on the dialogue.

 

Slow response time patterns almost always follow the pattern below:

  • response_time>X

In this context, response_time is the field value representing the server or client’s response, and ‘X’ is a threshold. If this threshold is exceeded, you want the event to be highlighted or to receive a notification so that you and your team are aware that somebody is having a poor user experience.

 

2. Memory Issues and Garbage Collection

Outofmemory errors can be pretty catastrophic, as they often result in the application's crashing due to lack of resources. Thus, you want to know about these when they occur; creating tags and generating notifications via alerts when these events occur is always recommended.

 

Your garbage collection behavior can be a leading indicator of outofmemory issues. Tracking this and getting notified if heap used vs free heap space is over a particular threshold, or if garbage collection is taking a long time, can be particularly useful and may point you toward memory leaks. Identifying a memory leak before an outofmemory exception can be the difference between a major system outage and a simple server restart until the issue is patched.

 

Furthermore, slow or long garbage collection can also be a reason for a user’s experiencing slow application behavior. During garbage collection, your system can slow down; in some situations it blocks until garbage collection is complete (e.g. with ‘stop the world’ garbage collection).

 

Below are some examples of common patterns used to identify the memory related issues outlined above:

  • Out of memory
  • exceeds memory limit
  • memory leak detected
  • java.lang.OutOfMemoryError
  • System.OutOfMemoryException
  • memwatch:leak: Ended heapDiff
  • GC AND stats

 

3. Deadlocks and Threading Issues

Deadlocks can occur in many shapes and sizes and can have a range of negative effects—from simply slowing your system down to bringing it to a complete halt. In short, a deadlock is a situation in which two or more competing actions are each waiting for the other to finish, and thus neither ever does. For example, we say that a set of processes or threads is deadlocked when each thread is waiting for an event that only another process in the set can cause.

 

Not surprisingly, deadlocks feature as one of our top 5 system performance related issues that our users write patterns to detect in their systems.

 

Most deadlock patterns simply contain the keyword ‘deadlock’, but some of the common patterns follow the following structure:

  • ‘deadlock’
  • ‘Deadlock found when trying to get lock’
  • ‘Unexpected error while processing request: deadlock;’

 

4. High Resource Usage (CPU/Disk/ Network)

In many cases, a slow down in system performance may not be as a result of any major software flaw, but instead is a simple case of the load on your system increasing without increased resources available to deal with this. Tracking resource usage can allow you to see when you require additional capacity such that you can kick off more server instances (for example).

 

Example patterns used when analyzing resource usage:

  • metric=/CPUUtilization/ AND minimum>X
  • cpu>X
  • disk>X
  • disk is at or near capacity
  • not enough space on the disk
  • java.io.IOException: No space left on device
  • insufficient bandwidth

 

5. Database Issues and Slow Queries

Knowing when a query failed can be useful, since it allows you to identify situations when a request may have returned without the relevant data and thus helps you identify when users are not getting the data they need. There can be more subtle issues, however, such as when a user is getting the correct results but the results are taking a long time to return. While technically the system may be fine (and bug-free), a slow user experience hurts your top line.

 

Tracking slow queries allows you to track how your DB queries are performing. Setting acceptable thresholds for query time and reporting on anything that exceeds these thresholds can help you quickly identify when your user's experience is being affected.

 

Example patterns:

 

  • SqlException
  • SQL Timeout
  • Long query
  • Slow query
  • WARNING: Query took longer than X
  • Query_time > X

 

Continued Log Data Analysis

As always, let us know if you think we have left out any important issues that you like to track in your log data. To start tracking your own system performance, sign up for our free log management tool and include these patterns listed above to automatically create tags and alerts relevant for your system.

 

Ready to start getting insights from your applications? Try InsightOps today.

Today we are announcing the general availability of a brand new solution: Rapid7 InsightOps. This latest addition to the Insight platform continues our mission to transform data into answers, giving you the confidence and control to act quickly. InsightOps is Rapid7’s first IT-specific solution, enabling users to centralize data from infrastructure, assets and applications, so they can monitor and troubleshoot operational issues.

 

Getting in with the IT crowd

Every day, IT and security teams work hand-in-hand towards keeping their organizations secure, optimized and operational. Yet today’s IT environment is more complex than ever. Infrastructure is hosted across physical servers, virtual machines, Docker containers and cloud services. The corporate network is accessed by internal and remote employees, from a mix of known and unknown devices that are all using applications, both internally hosted and cloud-based.

 

This complexity creates enormous amounts of data, dispersed across the modern IT environment. Managing this data is critical, but for most resource constrained IT and security teams, it’s simply too complex or too expensive to monitor it all. And unmonitored IT data creates risk.

 

That’s where Rapid7 comes in. Today, our customers leverage the Rapid7 Insight platform to collect data from across their entire IT environment for identifying security vulnerabilities with Rapid7 InsightVM and catching attackers in the act with Rapid7 InsightIDR. InsightOps builds on this, enabling them to manage and optimize IT operations across their technology landscape.

 

Introducing Rapid7 InsightOps

We built InsightOps to be easy to set up and scale. It requires no infrastructure to run, no configuration of indexers to search, and you can collect data in any format from anywhere in your environment. With your data centralized in one place, it’s easier to then monitor for known issues or anomalous trends. Monitoring with InsightOps helps you proactively address issues before they become widespread.


Ultimately, InsightOps was built for turning IT data into answers. With features like Visual Search and Endpoint Interrogator, it’s easier to get answers from your data without ever even having to type a search query.

 

Visual Search.png

 

And log data is just the beginning. Sometimes you need answers directly from your IT assets, like what software is running on an employee workstation or which servers are over 75% disk utilization. InsightOps combines log management with IT asset visibility and interrogation, enabling you to trace issues all the way from discovery to resolution.

 

Endpoint Interrogrator - Questions.png

 

Ready to transform your unmonitored IT data into answers? Start your free 30-day trial of InsightOps today.

The Benefits of Vagrant Plugins

 

Image result for aws logo high res

Following on from my recent DevOps blog posts, The DevOps Tools We Use & How We Use Them and Vagrant with Chef-Server, we will take another step forward and look into provisioning our servers in the cloud. There are many cloud providers out there, most who provide some sort of APIs. Dealing with the different APIs and scripts can become cumbersome and confusing when your main focus is a fault tolerant, scalable system. This is where Vagrant and many of its plugins shine.

 

 

Image result for digitalocean logo high res

Vagrant has a wide range of plugins, from handling Chef and Puppet to provisioning servers on many different cloud providers. We’re going to focus on 3 plugins in specific: vagrant-aws, vagrant-digitalocean and vagrant-omnibus. The AWS and Digital Ocean plugins allow us to utilize both our Chef-server and the public infrastructure provided by both Amazon and Digital Ocean. The omnibus is used for installing a specified version of Chef on your servers.

 

Installation of Vagrant Plugins

To install the Vagrant plugins, run the following commands:

vagrant plugin install vagrant-aws 
vagrant plugin install vagrant
-digitalocean
vagrant plugin install vagrant
-omnibus

Running Vagrant plugin list should give you the following output:

vagrant-aws (0.4.1) 
vagrant
-digitalocean (0.5.3)
vagrant
-omnibus (1.3.1)

More information on the plugins can be found here:
https://github.com/mitchellh/vagrant-aws
https://github.com/smdahlen/vagrant-digitalocean
https://github.com/schisamo/vagrant-omnibus

Amazon AWS

The first cloud provider we will look at is Amazon AWS. If you don’t already have an account I suggest you sign up for one here, and have a quick read of their EC2_GetStarted docs. Once signed up you will need to generate an account access and secret key, to do this follow the instruction below:

  1. Go to the IAM console.
  2. From the navigation menu, click Users.
  3. Select your IAM user name.
  4. Click User Actions, and then click Manage Access Keys.
  5. Click Create Access Key.
  6. Your keys will look something like this:
    1.     Access key ID example: ABCDEF0123456789ABCD
    2.     Secret access key example: abcdef0123456/789ABCD/EF0123456789abcdef
  7. Click Download Credentials, and store the keys in a secure location.
Vagrant AWS with Chef-Server

Now we have everything to get started. Again we will create a Vagrant file similar to what was done in my last blog post DevOps: Vagrant with Chef-Server, however, this time we will omit a few things like config.vm.box and config.vm.boxurl. The reason for this is that we are going to point our Vagrant file to use an Amazon AMI instead.

# -*- mode: ruby -*- 
# vi: set ft=ruby :
Vagrant.require_plugin 'vagrant-aws'
Vagrant.require_plugin 'vagrant-omnibus'
Vagrant.configure("2") do |config|
config
.omnibus.chef_version = :latest
config
.vm.synced_folder '.', '/vagrant', :disabled => true
config
.vm.box = "dummy"
config
.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/
dummy.box"

# Provider
config
.vm.provider :aws do |aws, override|
aws
.access_key_id = "ABCDEF0123456789ABCD"
aws
.secret_access_key = "abcdef0123456/789ABCD/EF0123456789abcdef"
aws
.keypair_name = "awskey"
aws
.ami = "ami-9de416ea" #Ubuntu 12.04 LTS
aws
.region = "eu-west-1"
aws
.instance_type = "t1.micro"
aws
.security_groups = ["default"]
override.ssh.username = "ubuntu"
override.ssh.private_key_path = "path/to/your/awskey.pem"
aws
.tags = {
'Name' => 'Java'
}
end
# Provisioning
config
.vm.provision :chef_client do |chef|
chef
.chef_server_url = 'https://api.opscode.com/organizations/logentries'
chef
.validation_key_path = '../../.chef/logentries-validator.pem'
chef
.validation_client_name = 'logentries-validator'
chef
.log_level = 'info'
chef
.add_recipe 'java_wrapper'
end
end

You should replace the above bold/italics options with your own settings.
The only difference this time in running Vagrant is that you need to pass a provider argument, so your command should look like this:

vagrant up --provider=aws

Once your Chef run has completed you will have a new instance in Amazon running your specified version of Java. Your output should look similar to this:

root@ip-172-31-28-193:~# java -version 
java version
"1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
Digital Ocean

Another cloud provider which has been gaining a lot of attention lately is Digital Ocean. If you don’t already have a Digital Ocean account you can sign up for here. Also have a look at their getting getting started guide if you’re new to Digital Ocean. The main difference between this Vagrant file and the AWS Vagrant file is that the provider block is different.

# -*- mode: ruby -*- 
# vi: set ft=ruby :
Vagrant.require_plugin 'vagrant-digitalocean'
Vagrant.require_plugin 'vagrant-omnibus'
Vagrant.configure('2') do |config|
config
.vm.provider :digital_ocean do |provider, override|
provider
.client_id = 'abcdef0123456789ABCDEF'
provider
.api_key = 'abcdef0123456789abcdef0123456789'
provider
.image = 'Ubuntu 12.04.3 x64'
provider
.region = 'Amsterdam 1'
provider
.size = '512MB'
provider
.ssh_key_name = 'KeyName'
override.ssh.private_key_path = '/path/to/your/key'
override.vm.box = 'digital_ocean'
override.vm.box_url = "https://github.com/smdahlen/vagrant-digitalocean/raw/
master/box/digital_ocean.box"

provider
.ca_path = "/usr/local/opt/curl-ca-bundle/share/ca-bundle.crt"
end
config
.vm.provision :chef_client do |chef|
chef
.chef_server_url = 'https://api.opscode.com/organizations/logentries'
chef
.validation_key_path = '../../.chef/logentries-validator.pem'
chef
.validation_client_name = 'logentries-validator'
chef
.log_level = 'info'
chef
.node_name = 'do-java'
chef
.add_recipe 'java_wrapper'
end
end

You should replace the above bold/italics options with your own settings. This time when we run Vagrantup we will pass a provider parameter of digital_ocean

vagrant up --provider=digital_ocean

Once the Vagrant run has complete, SSHing into your server and running java -version should give you the following output.

root@do-java:~# java -version 
java version
"1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

If you do run into issues it might be that your cookbooks are out of date, try running the following

librarian-chef update 
knife cookbook upload
-a

Now with the above scripts at your disposal you can use Chef-server with Vagrant and as many different cloud providers as you wish to keep your systems up to date, in sync and running to keep your customers happy!

Ready to bring Vagrant to your production and development?

Check out our free log management tool and get started today.

The Benefits and Challenges of D3 Angular Combination

Today we'll be focusing on how to combine D3 with the AngularJS framework. As we all know, Angular and

Combining d3 and angularjs

 

D3 frameworks are very popular, and once they work together they can be very powerful and helpful when creating dashboards. But, they can also be challenging and confusing especially when new to these frameworks. The right way to incorporate D3 with Angular is to use custom directives. Directives in Angular are essentially functions that are executed on a DOM element. Let’s go through a simple example together.

 

 

D3 Bar Chart Example

  • Our first step is to create a factory which will enable us to inject the D3 module into our custom directive.

 

blog-post-one.PNG

 

  • After creating the factory, we can now inject the module into our main app.

 

blog-post-two.PNG

 

  • For the purpose of this example, our directive will display a simple bar chart. We can call the directive from a HTML tag.

 

blog-post-three.PNG

 

  • We inject our d3 module into our custom directive.

 

blog-post-four.PNG

 

  • In order to use the D3 library, we need to include our service. You can either copy and paste the entire D3 library or just reference a CDN.

 

blog-post-five.PNG

 

  • In the service, we will use the notion of a promise. We need to wait for the promise to resolve by using the “then” method of our d3Service. For simplicity, we will use a link function which registers listeners and sets up binding.

 

blog-post-six.PNG

 

  • We are now ready to write actual D3 code inside our AngularJS directive. The bar chart can also be made dynamic for the developer. We can pass attributes such as width of the bar chart to the directive and then use them with the D3 code. We will in our example set up an event listener to track changes in the size of the window.

 

blog-post-seven.PNG

 

  • After doing that we can focus on the D3 code for the bar chart.

 

blog-post-eight.PNG

 

  • One important thing to remember before the previous statement is that if SVG already existed, it would need to be emptied. Here is how you would remove all the preexisting SVG elements.

 

blog-post-nine.PNG

 

D3 and Angular: Easy Implementation

This example only shows a very small subset of the D3 code that could go into our directive. As well, we only worked with hardcoded data whereas in a real world scenario we would use data from a back-end service. Thanks to AngularJS, we are able to easily integrate that data into our custom D3 directive by using bi-directional binding.

 

All in all, combining D3 with AngularJS now looks straight forward and relatively easy.  Let us know how it goes for you!

Combine D3 and Angular Today

Ready to see what combining D3 and Angular can do for you? Check out our free log management tool.

Rapid Seven

What is Syslog?

Posted by Rapid Seven Employee May 24, 2017

This post has been written by Dr. Miao Wang, a Post-Doctoral Researcher at the Performance Engineering Lab at University College Dublin.

This post is the first in a multi-part series of posts on the many options for collecting and forwarding log data from different platforms and the pros and cons of each. In this first post we will focus on Syslog, and will provide background on the Syslog protocol.

What is Syslog?

Syslog has been around for a number of decades and provides a protocol used for transporting event messages between computer systems and software applications. The Syslog protocol utilizes a layered architecture, which allows the use of any number of transport protocols for transmission of Syslog messages. It also provides a message format that allows vendor-specific extensions to be provided in a structured way. Syslog is now standardized by the IETF in RFC 5424 (since 2009), but has been around since the 80’s and for many years served as the de facto standard for logging without any authoritative published specification.

Syslog has gained significant popularity and wide support on major operating system platforms and software frameworks and is officially supported on almost all versions of Linux, Unix, and MacOS platforms. On Microsoft Windows, Syslog can also be supported through a variety of open source and commercial third-party libraries.

Syslog best practices often promote storing log messages on a centralized server that can provide a correlated view on all the log data generated by different system components. Otherwise, analyzing each log file separately and then manually linking each related log message is extremely time-consuming. As a result, forwarding local log messages to a remote log analytics server/service via Syslog has been commonly adopted as a standard industrial logging solution.

How does Syslog work?

The Syslog standard defines three different layers, namely the Syslog content, the Syslog application and the Syslog transport. The content refers to the information contained in a Syslog event message. The application layer is essentially what generates, interprets, routes, and stores the message while the Syslog transport layer transmits the message via the network.

Screen Shot 2014-08-29 at 8.55.34 AM

Diagram 1 from the RFC 5424 Syslog Spec

According to the Syslog specification, there is no acknowledgement for message delivery and although some transports may provide status information, the Syslog protocol is described as a pure simplex protocol. Sample deployment scenarios in the spec show arrangements where messages are said to be created by an ‘originator’ and forwarded on to a ‘collector’ (generally a logging server or service used for centralized storage of log data). Note ‘relays ’ can also be used between the originator and the collector and can do some processing on the data before it is sent on (e.g. filtering out events, combining sources of event data).

Applications can be configured to send messages to multiple destinations, and individual Syslog components may be running in the same host machine.

The Syslog Format

Sharing log data between different applications requires a standard definition and format on the log message, such that both parties can interpret and understand each other’s information. To provide this, RFC 5424 defines the Syslog message format and rules for each data element within each message.

A Syslog message has the following format: A header, followed by structured-data (SD), followed by a message.

The header of the Syslog message contains “priority”, “version”, “timestamp”, “hostname”, “application”, “process id”, and “message id”. It is followed by structured-data, which contains data blocks in the “key=value” format enclosed in square brackets “[]”, e.g. [SDID@0 utilization=“high” os=”linux”] [SDPriority@0 class=”medium”]. In the example image below, the SD is simply represented as “-“, which is a null value (nilvalue as specified by RFC 5424). After the SD value, BOM represents the UTF-8 and “su root failed on /dev/pts/7” shows the detailed log message, which should be encoded UTF-8. (For more details of the data elements of SLP, please refer to: http://tools.ietf.org/html/rfc5424)

Untitled

A Sample Syslog Message with Format Broken Out

Why Syslog?

The complexity of modern application and systems is ever increasing and to understand the behavior of complex systems, administrators/developers/Ops etc. often need to collect and monitor all relevant information produced by their applications. Such information often needs to be analyzed and correlated to determine how their systems are behaving. Consequently, administrators can apply data analytic techniques to either diagnose root causes once problems occur or gain an insight into current system behavior based on statistical analysis.

Frequently, logs have been applied as a primary and reliable data source to fulfill such a mission for lots of reasons, some of which I’ve listed here:

  • Logs can provide transient information for administrators to roll back the system to a proper status after a failure accident. E.g. when a banking system fails, all transactions lost from the main memory can be recorded in the logs.
  • Logs can contain a rich diversity of substantial information produced by individual applications to allow administrators/developers/ops teams to understand system behavior from many aspects such as current system statistics, trend predictions, and troubleshooting.
  • Logs are written externally by the underlying application to hard disks and external services such that by reading these log files, there will not be any direct performance impact on the monitored system. Therefore, in a production environment administrators can safely monitor running applications via their logs without worrying about impacting performance.

 

However, a key aspect of log analysis is to understand the format of the arriving log data, especially in a heterogeneous environment where different applications may be developed using different log formats and network protocols to send these log data. Unless this is well defined, it is quite difficult to interpret log messages sent by an unknown application. To solve this issue Syslog defines a logging standard for different systems and applications to follow in order to easily exchange log information. Based on the logging protocol, Syslog helps applications effectively interpret each log attribute to understand the meaning of the log message.

Ready to put Syslog into action?

Try our free log management tool today.

It’s generally a good practice to minify and combine your assets (Javascript & CSS) when deploying to production. This process reduces the size of your assets and dramatically improves your website’s load time.

 

what are javascript source maps

Source maps create a map from these compressed asset files back to the source files.

This source map allows you to debug and view the source code of your compressed assets, as if you were actually working with the original CSS and Javascript source code.

Take a look at jQuery minified & combined code that was generated from the original source code. The code is practically unreadable and would be difficult to debug.

 

But, as we all know, no matter how thoroughly you test, sometimes bugs will fall through the cracks. This is why it’s useful to debug Javascript code in production, and that’s when source maps come in handy.

How do you use Javascript source maps?

With InsightOps we use UglifyJS for minification and Javascript source map generation. UglifyJS is a NodeJS library written in Javascript.

To install Uglify JS with NPM:

npm install uglify-js -g

Minify the files and generate source maps:

uglify-js file1.js file2.js -o output.js --source-map output.map.js

The code above tells UglifyJS to:

  • Take file1.js and file2.js as input
  • Compress input files and output them to output.js
  • Generate the source map for the compressed file and output it to output.map.js

 

Marrying source maps and Django Compressor

 

Django Compressor is a great Django plugin to mark assets for minification right inside your templates:

{% load compress %} {% compress js %} <script src="/static/js/one.js" 
type
="text/javascript" charset="utf-8"></script> <script type="text/ja
vascript"
charset="utf-8">obj.value = "value";</script> {% endcompress
%}

Behind the scenes you can develop logic to combine and minify the files with any algorithm or third party tools of your choosing.

A great blog post by Chris Roby goes into great detail about how to extend compressor to work with UglifyJS and produce source maps. It is definitely worth the read if you’re running Django.

Browser support

Source maps are a new addition to the developer toolbox. Although the source maps spec lives in Google docs (no kidding), they’re supported by all major browsers: Chrome, Safari, Firefox, and IE11. By default, source maps are disabled so your users will not incur any unnecessary bandwidth overheads.

To enable source maps in Google Chrome, go to Developer Tools, click the little cog icon, and then make sure that “Enable Javascript source maps” is checked.

Enable source maps

That’s it.

Now each compressed asset file contains a link pointing to its source map, and we’ve just told Chrome not to ignore them.

See Javascript source maps in action

If you’d like to see Javascript source maps in action, check out our free log management tool and take a look at our source code.

 

Logentries source code

The files highlighted in green are compressed Javascript files; the folders highlighted in blue are generated from source maps and contain the original source code that’s mapped onto the compressed files. We can set breakpoints on mapped code, inspect variables, step through, and do pretty much anything that we can with original code.

Pretty cool, huh?

Rapid Seven

Heroku Dynos Explained

Posted by Rapid Seven Employee May 24, 2017

What are Heroku Dynos?

 

If you’ve ever hosted an application on Heroku, the popular platform as a service, you’re likely at least aware of the existence of “Dynos”. But what exactly are Heroku Dynos and why are they important?

 

heroku-dynos-explained

As explained in Heroku’s docs, Dynos are simply lightweight Linux containers dedicated to running your application processes. At the most basic level, a newly deployed app to Heroku will be supported by one Dyno for running web processes. You then have the option of adding additional Dynos and specifying Dyno processes in your procfile. Dynos actually come in three different flavors:

 

  • Web Dynos: for handling web processes
  • Worker Dynos: for handling any type of process you declare (like background jobs)
  • One-off Dynos: for handling one-off tasks, such as database migrations

 

One of the great things about Heroku Dynos is how easy they are to scale up and out. Through Heroku’s admin portal or via the command line, you can easily add more Dynos or larger Dynos. Adding additional Dynos can help speed up your application’s response time by handling more concurrent requests, whereas adding larger Dynos can provide additional RAM for your application.

Using Heroku Dynos to get the insights you need

Great. I get it. Dynos make it easy to run my application with less hassle. For this reason, I should have to think very little about Heroku Dynos, right? Wrong!

As Dynos are individual containers and identified uniquely in your Heroku logs, they can provide some great insight into where issues may be stemming from when things go wrong.

When sending Heroku logs to InsightOps, a Dyno’s unique ID will automatically be logged in key-value pair format along with information about the process it handled. In the example below, we see web Dynos being identified along with the HTTP requests being handled:

heroku_log_view

An easier way to view this data in InsightOps is to use the Table View:

heroku_tableview

As you can see, the Heroku Dyno is easily identified along other pertinent data. Since I’ve enabled Heroku’s beta log-runtime-metrics from Heroku Labs, I can also see data related to CPU and memory per Dyno, which is particularly useful for identifying issues like too much swap memory being used (perhaps indicating a need to scale up my Dynos).

 

Since Dynos are uniquely identified in key value pairs, you can also use them to visualize data. In the example below, I’m visualizing how much swap memory each Dyno is using over a given period of time:

heroku_swap_memory

You can also create visualizations to help you monitor how much each Dyno is being used compared to others:

Screen Shot 2016-04-27 at 1.03.45 AM

Finally, checking the Heroku Dyno related to errors in your logs could hint at Dyno-related issues. In the example below, we see that Dyno web.2 is related to both errors, which happen to be backend connection timeouts. While Heroku Dynos are allowed to fail and automatically restart on a different server, this finding could warrant you manually restarting your Dynos to alleviate the issue.

Screen Shot 2016-04-27 at 1.08.23 AM

Start logging with Heroku Dynos today

Ready to start logging from your Heroku app today? Check out our free log management tool.

Server monitoring is a requirement, not a choice. It is used for your entire software stack, web-based enterprise suites, custom applications, e-commerce sites, local area networks, etc. Unmonitored servers are lost opportunities for optimization, difficult to maintain, more unpredictable, and more prone to failure.

 

While it is very likely that your team has a log management and analysis initiative, it’s important to determine if you are only responding to what the logs tell you about the past, or are you planning ahead based on the valuable log data you are monitoring and analyzing?

 

There are two basic approaches to server monitoring: passive monitoring and active monitoring. They are as much a state of mind as a process. And there are significant differences in the kinds of value each type of server monitoring provides; each type has its own advantages and disadvantages.

 

What is Passive Server Monitoring?

 

Passive server monitoring looks at real-world historical performance by monitoring actual log-ins, site hits, clicks, requests for data, and other server transactions. When it comes to addressing issues in the system, the team will review historical log data, and from there they analyze the logs to troubleshoot and pinpoint issues. This was previously done with a manual pull of logs. While this helps developers identify where issues are, using a powerful modern log analysis service to simply automate an existing process is a waste.

 

Passive monitoring only shows how your server handles existing conditions, but it may not give you much insight into how your server will deal with future ones. For example, if one of the components of the system, a database server, is likely to be overloaded when the load rate of change is reached. This is not going to be clear when server log data has already been recorded, unless your team is willing to stare at a graph in real-time, 24/7…which has been nearly the case in some NOC operations I have witnessed.


What is Active Server Monitoring?

 

The most effective way to get past these limits is by using active server monitoring. Active monitoring is the approach that leverages smart recognition algorithms to take current log data and use it to predict future states. This is done by some complex statistics (way over my head) that compare real-time to previous conditions, or past issues. For example it leverages anomaly detection, steady state analysis, and trending capabilities to predict that a workload is about to hit its max capacity. Or there is a sudden decrease in external network-received packets, a sign of public web degradation.

 

Besides finding out what is possibly going to happen, active server monitoring also helps to avoid the time spent on log deep dives. Issues will sometimes still pass you by, and you will still need to take a deeper look, but because information is pushed to you, some of the work is already done, and you can avoid the log hunt.

 

Oh and active monitoring can help the product and dev team from an architectural standpoint. If, for example, a key page is being accessed infrequently, or if a specific link to that page is rarely used, it may indicate a problem with the design of the referring page, or with one of the links leading to that page. A close look at the log can also tell you whether certain pages are being accessed more often than expected  —  which can be a sign that the information on those pages should be displayed or linked more prominently.

 

Any Form of Server Monitoring is Better Than None

 

Log analysis tools are the heart of both server monitoring approaches. Log analysis can indicate unusual activity which might slip past and already overloaded team. Another serious case is security. A series of attempted page hits that produce “page not found” or access denied” errors, for example, could just be coming from a bad external link  —  or they could be signs of an attacker probing your site. HTTP request that are pegging a server process could be a sign that a denial of service attack has begun.

 

It is hard to make the shift from passive to active monitoring. Why? Not because you and your team are not interested in thinking ahead. But more so because many operations are entrenched in existing processes that are also reactive. And sometimes teams are just unaware that their tool can provide this type of functionality. Until one day it does it automatically for you, and you have a pleasant surprise.

 

Active server monitoring can mean the difference between preventing problems before they get a chance to happen, or rushing to catch up with trouble after it happens. And they are the difference between a modern version of an old process, and moving forward to a modern software delivery pipeline.

 

Ready to Make the Shift from Passive to Active Monitoring?

 

Sign up for our free log management tool today.

Merry HaXmas to you! Each year we mark the 12 Days of HaXmas with 12 blog posts on hacking-related topics and roundups from the year. This year, we’re highlighting some of the “gifts” we want to give back to the community. And while these gifts may not come wrapped with a bow, we hope you enjoy them.

 

Machine generated log data is probably the simplest and one of the most used data source for everyday use cases such as troubleshooting, monitoring, security investigations … the list goes on. Since log data records exactly what happens in your software over time it is extremely useful for understanding what had caused an outage or security vulnerability. With technologies like InsightOps, it can also be used to monitor systems in real time by looking at live log data which can contain anything from resource usage information, to error rates, to user activity etc.

 

So in short when used for the right job, log data is extremely powerful…. until it’s NOT!

 

When is it not useful to look at logs? When your logs don’t contain the data you need. How many times during an investigation have your logs contained enough information to point you in the right direction, but then fell short of giving you the complete picture. Unfortunately, it is quite common to run out of road when looking at log data; if only you had recorded 'user logins’, or some other piece of data that was important with hindsight, you could figure out what user installed some malware and your investigation would be complete. 

 

Log data, by its very nature, provides an incomplete view of your system, and while log and machine data is invaluable for troubleshooting, investigations and monitoring, it is generally at its most powerful when used in conjunction with other data sources. If you think about it, knowing exactly what to log up front to give you 100% code or system coverage is like trying to predict the future. Thus when problems arise or investigations are underway, you may not have the complete picture you need to identify the true root cause.

 

So our gift to you this HaXmas is the ability to generate log data on the fly through our new endpoint technology, InsightOPs, which enables you to  fill in any missing information during troubleshooting or investigations. InsightOps is pioneering the ability to generate log data on the fly by allowing end users to ask questions of their environment, InsightOps is pioneering the ability to generate log data on the fly by returning answers in the form of logs. Essentially, it will allow you to create synthetic logs which can be combined with your traditional log data - giving you the complete picture! It also gives you all this information in one place (so no need to combine a bunch of different IT monitoring tools to get all the information you need).

 

Live+Questions_InsightOps.png

You will be able to ask anything from 'what processes are running on every endpoint in my environment’ to ‘what is the memory consumption' of a given process or machine. In fact, our vision is to allow users to ask any question that might be relevant for their environment such that you will never be left in the dark and never again have to say ‘if only I had logged that.’

 

Memory+Utilization_InsightOps+Question.png

Interested in trying InsightOps for yourself? Don’t forget to sign up to our beta here: https://www.rapid7.com/products/insightops/beta-request  

 

Happy HaXmas!

Our mission at Rapid7 is to solve complex security and IT challenges with simple, innovative solutions. Late last year Logentries joined the Rapid7 family to help to drive this mission. The Logentries technology itself had been designed to reveal the power of log data to the world and had built a community of 50,000 users on the foundations of our real time, easy to use yet powerful log management and analytics engine.

 

Today we are excited to announce InsightOps, the next generation of Logentries. InsightOps builds on the fundamental premise that in a world where systems are increasingly distributed, cloud-based and made up of connected/smart devices, log and machine data is inherently valuable to understand what is going on, be that from a performance perspective, troubleshooting customer issues or when investigating security threats.

 

However, InsightOps also builds on a second fundamental premise, which is that log data is very often an incomplete view of your system, and while log and machine data is invaluable for troubleshooting, investigations and monitoring, it is generally at its most powerful when used in conjunction with other data sources.

 

If you think about it, knowing exactly what to log up front to give you 100% code or system coverage is like trying to predict the future. Thus when problems arise or investigations are underway, you may not have the complete picture you need to identify the true root cause.

 

To solve this problem InsightOps allows users to ask questions of specific endpoints in your environment. The endpoints return answers to these questions, in seconds, in the form of log events such that they can be correlated with your existing log data. I think of it as being able to generate 'synthetic logs' on the fly - logs designed to answer your questions as you investigate or need vital missing information. How often have you said during troubleshooting or an investigation "I wish I had logged that…”? Now you can ask questions in real time to fill in the missing details e.g. “who was the last person to have logged into this machine?”

 

Live Questions_InsightOps.png

Fig1. InsightOPs Endpoint Question Examples

 

InsightOps combines both log data and endpoint information such that users can get a more complete understanding of their infrastructure and applications through a single solution. InsightOps will now deliver this IT data in one place and thus avoids the need for IT professionals to jump between several, disparate tools in order to get a more complete picture of their systems. By the way - this is the top pain point IT professionals have reported across lots and lots of conversations that we have had, and that we continue to have, with our large community of users.

 

Memory Utilization_InsightOps Question.pngFig2. InsightOPs Endpoint Data Example

 

To say I am excited about this is an understatement - I’ve been building and researching log analytics solutions for more than 10 years and I truly believe the power provided by combining logs and endpoints will be a serious game changer for anybody who utilizes log data as part of their day to day responsibilities -- be that for asset management, infrastructure monitoring, maintaining compliance or simply achieving greater visibility, awareness and control over your IT environment.

 

InsightOps will also be providing some awesome new capabilities beyond our new endpoint technology, including:

 

Visual Search: Visual search is an exciting new way of searching and analyzing trends in your log data by interacting with auto-generated graphs. InsightOps will automatically identify key trends in your logs and will visualize these when in visual search mode. You can interact with these to filter your logs allowing you to search and look for trends in your log data without having to write a single search query.

 

New Dashboards and Reporting: We have enhanced our dashboard technology making it easier to configure dashboards as well as providing a new, slicker look and feel. Dashboards can also be exported to our report manager where you can store and schedule reports, which can be used to provide a view of important trends e.g. reporting to management or for compliance reporting purposes.

 

Data Enrichment: Providing additional context and structuring log data can be invaluable for easier analysis and ultimately to drive more value from your log and machine data. InsightOps enhances your logs by enriching them in 2 ways, (1) by combining endpoint data with your traditional logs to provide additional context and (2) by normalization your logs into a common JSON structure such that it is easier for users to work with, run queries against, build dashboards etc.

 

 

Visual Search_InsightOps.pngFig3. InsightOPs Visual Search

 

As always check it out and let us know what you think - we are super excited to lead the way into the next generation of log analytics technologies. You can apply for access to the InsightOps beta program here: https://www.rapid7.com/products/insightops/beta-request