Getting More from Skylight

Learn how Skylight works. Enable instrumentation for additional libraries and add custom instrumentation to your own code.

Available Instrumentation Options

Rails

See the Rails setup instructions.

In Rails applications, we use ActiveSupport::Notifications to track the following.

  • Controller Actions
  • Controller File Sending
  • View Collection Rendering
  • View Partial Rendering
  • View Template Rendering
  • View Layout Rendering
  • ActiveRecord SQL Queries
  • ActiveRecord Instantiation
  • ActiveSupport Cache

Grape

See the Grape setup instructions.

  • Endpoint Execution
  • Endpoint Filters
  • Endpoint Rendering

Sinatra

See the Sinatra setup instructions..

  • Endpoint Execution

ActiveModel::Serializers

Add active_model_serializers to probes list.

  • Serialization

Excon

Add excon to probes list.

  • HTTP Requests

Faraday

Add faraday to probes list.

  • HTTP Requests

Middleware

Enabled by default

  • Tracks Rack Middleware in Rails applications.

Mongo (Official Driver)

Add mongo to probes list.

  • Database Queries

Moped

Add moped to probes list.

  • Database Queries

Mongoid

Add mongoid to probes list.

Depending on the version, Mongoid either uses Moped or the official Mongo driver under the hood. This probe will just enable the correct probe for one of these two libraries.

Net::HTTP

Enabled by default

  • HTTP Requests

Redis

Add redis to probes list.

  • All Commands
Note: We do not instrument AUTH as it would include sensitive data.

Sequel

Enabled automatically with Sinatra or Add sequel to probes list.

  • SQL Queries

Tilt

Enabled automatically with Sinatra or Add tilt to probes list.

  • Template Rendering

Custom App Instrumentation

The easiest way to add custom instrumentation to your application is by specifying methods to instrument, but it is also possible to instrument specific blocks of code. This way, you can see where particular methods or blocks of code are called in the Event Sequence. Check out an example of custom instrumentation in action here. Be sure to follow our custom instrumentation best practices in order to avoid causing errors in your Skylight trace.

Method instrumentation

Instrumenting a specific method will cause an event to be created every time that method is called. The event will be inserted at the appropriate place in the Event Sequence. You can see an example of how this looks below.

To instrument a method, the first thing to do is include Skylight::Helpers into the class that you will be instrumenting. Then, annotate each method that you wish to instrument with instrument_method.

class MyClass
  include Skylight::Helpers

  instrument_method
  def my_method
    do_expensive_stuff
  end
end

You may also declare the methods to instrument at any time by passing the name of the method as the first argument to instrument_method.

class MyClass
  include Skylight::Helpers

  def my_method
    do_expensive_stuff
  end

  instrument_method :my_method
end

By default, the event will be titled using the name of the class and the method. The event name in our previous example would be: MyClass#my_method. Alternatively, you can customize the title by using the :title option.

class MyClass
  include Skylight::Helpers

  instrument_method title: 'Expensive work'
  def my_method
    do_expensive_stuff
  end
end

Block instrumentation

Method instrumentation is preferred, but if more fine-grained instrumentation is required, you may use the block instrumenter.

class MyClass
  def my_method
    Skylight.instrument title: "Doin' stuff" do
      step_one
      step_two
    end
    step_three
  end
end

Just like above, the title of the event can be configured with the :title option. If you don’t add a title, the default for block instrumentation is app.block. We recommend creating your own titles to differentiate the various instrumented code blocks from one another in the Skylight UI. If you don’t, they will be aggregated together, which won’t be helpful for you.

Custom Instrumentation Best Practices

Use Static String Literals for Custom Titles

IMPORTANT: The title of an event must be the same for all requests that hit the code path. Skylight aggregates Event Sequences using the title. You should pass a string literal and avoid any interpolation. Otherwise, there will be an explosion of nodes that show up in your aggregate Event Sequence.

Use Method Instrumentation Wherever Possible

While block instrumentation can be very useful for zeroing in on problematic minutiae, it can lead to overly complex request traces. Prefer method instrumentation where possible.

Avoid Over-Instrumenting Your Action

Add custom instrumentation only where necessary to understand an issue in order to avoid overwhelming yourself with too much information. Additionally, avoid instrumenting a method or block that happens a ton of times in repetition (e.g. a method called within a loop). Doing so may cause your event to exceed the size limit for an event trace in the Skylight agent.

Custom instrumentation in action

When we first implemented GitHub sign-in and sign-up, we noticed the act of signing in or signing up with GitHub was pretty slow. We wanted to see which methods in our GitHub integration were taking the longest, so we added a bit of method instrumentation. The Event Sequence for the endpoint that included those methods then looked more like this:

Custom instrumentation

Clearly the OctokitService#repo_summaries method was the culprit of the slowdown, so we knew where more refactoring was needed and that we might even consider creating a worker to take over much of this process (just look at how long those GitHub API requests are!). Custom instrumentation to the rescue!

How Skylight Works

If you’re curious about how our instrumentation works, you’re in the right place.

Featherweight Agent

The Skylight agent is written primarily in Rust, a systems programming language on par with C. This means we have a very low overhead so it’s safe to run Skylight in production, even in memory limited environments like Heroku. You can read more about this aspect of our agent here and more about the agent’s 1.0 release here.

Normalizers

Our preferred method of instrumentation is ActiveSupport::Notifications events. When a library has added instrumentation, all we need to do is subscribe to the event and do a little bit of normalization of the data.

To standardize this process, we introduced Normalizers. Each type of instrumentation has its own normalizer which handles the, well, normalization. You can take a look at some of them in the source.

Probes

While we think most libraries should include ActiveSupport::Notifications (anyone can subscribe to these notifications, not just Skylight), unfortunately, many still don’t. In these circumstances, we have to carefully monkey-patch the libraries at their key points.

To make sure we do this in a sane fashion, we developed Probes. Probes are small classes that keep an eye out for specific modules and then hook into them when they’re loaded. All probes can be disabled in the event of any conflicts and we only autoload probes that we have a high degree of confidence in. You can take a look at some probes in the source.

And, since we don’t really like having to monkey-patch things either, when at all possible, we submit pull requests to relevant projects to add in ActiveSupport::Notifications.

For more information about how to add an existing probe (such as Mongo, Moped, Excon, etc.) to your Skylight setup, go here.