Learn how Skylight works. Enable instrumentation for additional libraries and add custom instrumentation to your own code.
See the Rails setup instructions.
In Rails applications, we use ActiveSupport::Notifications to track the following.
See the Grape setup instructions.
See the Sinatra setup instructions..
active_model_serializers to probes list.
excon to probes list.
mongo to probes list.
moped to probes list.
mongoid to probes list.
Depending on the version, Mongoid either uses Moped or the official Mongo driver under the hood. This probe will just enable the correct probe for one of these two libraries.
Enabled by default
redis to probes list.
Enabled automatically with Sinatra or Add
sequel to probes list.
Enabled automatically with Sinatra or Add
tilt to probes list.
The easiest way to add custom instrumentation to your application is by specifying methods to instrument, but it is also possible to instrument specific blocks of code. This way, you can see where particular methods or blocks of code are called in the Event Sequence. Check out an example of custom instrumentation in action here. Be sure to follow our custom instrumentation best practices in order to avoid causing errors in your Skylight trace.
Instrumenting a specific method will cause an event to be created every time that method is called. The event will be inserted at the appropriate place in the Event Sequence. You can see an example of how this looks below.
To instrument a method, the first thing to do is include
Skylight::Helpers into the class that you will be instrumenting. Then, annotate each method that you wish to instrument with
1class MyClass 2 include Skylight::Helpers 3 4 instrument_method 5 def my_method 6 do_expensive_stuff 7 end 8end
You may also declare the methods to instrument at any time by passing the name of the method as the first argument to
1class MyClass 2 include Skylight::Helpers 3 4 def my_method 5 do_expensive_stuff 6 end 7 8 instrument_method :my_method 9end
By default, the event will be titled using the name of the class and the method. The event name in our previous example would be:
MyClass#my_method. Alternatively, you can customize the title by using the :title option.
1class MyClass 2 include Skylight::Helpers 3 4 instrument_method title: 'Expensive work' 5 def my_method 6 do_expensive_stuff 7 end 8end
Method instrumentation is preferred, but if more fine-grained instrumentation is required, you may use the block instrumenter.
1class MyClass 2 def my_method 3 Skylight.instrument title: "Doin' stuff" do 4 step_one 5 step_two 6 end 7 step_three 8 end 9end
Just like above, the title of the event can be configured with the :title option. If you don’t add a title, the default for block instrumentation is
app.block. We recommend creating your own titles to differentiate the various instrumented code blocks from one another in the Skylight UI. If you don’t, they will be aggregated together, which won’t be helpful for you.
Add custom instrumentation only where necessary to understand an issue in order to avoid overwhelming yourself with too much information. Additionally, avoid instrumenting a method or block that happens a ton of times in repetition (e.g. a method called within a loop). Doing so may cause your event to exceed the size limit for an event trace in the Skylight agent.
When we first implemented GitHub sign-in and sign-up, we noticed the act of signing in or signing up with GitHub was pretty slow. We wanted to see which methods in our GitHub integration were taking the longest, so we added a bit of method instrumentation. The Event Sequence for the endpoint that included those methods then looked more like this:
OctokitService#repo_summaries method was the culprit of the slowdown, so we knew where more refactoring was needed and that we might even consider creating a worker to take over much of this process (just look at how long those GitHub API requests are!). Custom instrumentation to the rescue!
If you’re curious about how our instrumentation works, you’re in the right place.
The Skylight agent is written primarily in Rust, a systems programming language on par with C. This means we have a very low overhead so it’s safe to run Skylight in production, even in memory limited environments like Heroku. You can read more about this aspect of our agent here and more about the agent’s 1.0 release here.
Our preferred method of instrumentation is
ActiveSupport::Notifications events. When a library has added instrumentation, all we need to do is subscribe to the event and do a little bit of normalization of the data.
To standardize this process, we introduced
Normalizers. Each type of instrumentation has its own normalizer which handles the, well, normalization. You can take a look at some of them in the source.
While we think most libraries should include
ActiveSupport::Notifications (anyone can subscribe to these notifications, not just Skylight), unfortunately, many still don’t. In these circumstances, we have to carefully monkey-patch the libraries at their key points.
To make sure we do this in a sane fashion, we developed
Probes. Probes are small classes that keep an eye out for specific modules and then hook into them when they’re loaded. All probes can be disabled in the event of any conflicts and we only autoload probes that we have a high degree of confidence in. You can take a look at some probes in the source.
And, since we don’t really like having to monkey-patch things either, when at all possible, we submit pull requests to relevant projects to add in
For more information about how to add an existing probe (such as Mongo, Moped, Excon, etc.) to your Skylight setup, go here.