Background Jobs

Setting up and managing instrumentation for your workers.

Skylight for Background Jobs

While Skylight was originally designed to profile web requests, we understand that the web interface is only a part of your server-side application. We developed Skylight for Background Jobs in order to help you discover and correct hidden performance issues in your Sidekiq, DelayedJob, and ActiveJob queues.

By default, the Skylight agent only enables itself for web requests. In many cases, your applications might run processes in the background, via Sidekiq, Delayed::Job, or some other framework. In fact, we recommend moving certain slow actions into background jobs in our Performance Tips. This page shows you how to configure Skylight to profile your background jobs so that you can view them separately in the Skylight UI:

Currently Supported Libraries and Frameworks

We currently have instrumentation for:

• Sidekiq
• Delayed::Job
• ActiveJob (supporting Sidekiq, Backburner, Que, Delayed::Job, Sneakers, Shoryuken, and others)

Resque is officially not supported for now, due to incompatibilities between Resque and Skylight’s process model (it doesn’t work via ActiveJob either).

Limitations

• Currently, jobs running over four hours will not be tracked (we have incrementally raised this limit and are still investigating the potential effects of raising it further).
• Jobs and requests with more than 2,048 items in the event sequence will result in “pruned” traces. In the job view in the UI, you may see nodes with the description “exceeded the maximum number of events allowed by the Skylight agent,” which means that some nodes were dropped from the trace before it was submitted. You can read more about this limitation in our Troubleshooting docs. Typically, you would see this error for individual jobs that do a lot of repetitive work— we generally recommend splitting these up so one job instance performs one unit of work.

Default Jobs Component Naming

By default, we will detect that you are running jobs and report them as a worker component. If you would like to see another name in the Skylight user interface, the component name can be customized, according to the instructions for your specific configuration setup.

Migrating Your Existing Background Jobs Setup

Enabling Background Job Instrumentation

The process for enabling background jobs instrumentation is straightforward but varies depending on how you configure Skylight and whether or not you use Rails. Be sure you are following the correct set of instructions below.

Using skylight.yml

1. Upgrade to the latest Skylight agent.

Background jobs instrumentation is available with version 4.0.0 (the current version is 5.0.1) and up of the Skylight agent:

# Gemfile
gem 'skylight', '~> 5.0.1'

2. Enable Skylight for your job processing framework.

If you use Sidekiq:

# config/skylight.yml
authentication: <app auth token>
enable_sidekiq: true

If you use Sidekiq with ActiveJob, you can enable either the Sidekiq config or the ActiveJob probe (read on to the next section), or both (it’s not necessary to do both, but it won’t hurt).

If you use ActiveJob:

# config/application.rb
config.skylight.probes << 'active_job'

If you use Delayed::Job:

# config/application.rb
config.skylight.probes << 'delayed_job'

3. [OPTIONAL] Specify a component name.

If you would like to use a custom component name:

Use environment-specific configuration to override the environment name.

# config/skylight.yml
authentication: <app auth token>
enable_sidekiq: true # if you are using sidekiq
worker_component: 'sidekiq' # or <%= worker_component_name %>

4. Deploy and watch your data trickle in!

Deploy the above changes following your normal deploy process. Your new worker component should show up in the components selector in the Skylight UI nav bar.

Using Environment Variables

Alternatively, if you use environment variables to configure Skylight, follow these instructions:

1. Deploy your app with the latest Skylight agent.

Background jobs instrumentation is available with version 4.0.0 (the current version is 5.0.1) and up of the Skylight agent:

# Gemfile
gem 'skylight', '~> 5.0.1'

2. Enable Skylight for your jobs.

Set the following environment variables wherever you start your jobs (e.g. in the Procfile on Heroku):

SKYLIGHT_AUTHENTICATION=<app auth token>
SKYLIGHT_ENABLE_SIDEKIQ=true # for sidekiq instrumentation

For ActiveJob and Delayed::Job, you’ll need to enable the appropriate probes in config/application.rb and deploy that change.

3. [OPTIONAL] Specify a component name.

If you would like to use a custom component name:

Set the environment name explicitly wherever you start your jobs. For example:

SKYLIGHT_COMPONENT=sidekiq

Background Jobs for Open Source Apps

By default, the Skylight agent only enables itself for web requests, but enabling Skylight for background jobs is super easy! We welcome our Skylight for Open Source customers and their contributors to profile their background jobs so that you can view them separately in the Skylight UI:

If you would like to help an open source app enable Skylight background job instrumentation, read the above documentation to learn about Background Jobs, then open a PR to the open source project.

Learn more about the Skylight for Open Source program at the OSS page.

Background Job Terminology

Although most of the Skylight UI remains unchanged when viewing your jobs, there are a few subtle differences in terminology that you’ll see in your Skylight dashboard. Below are a few helpful definitions to help you understand your jobs performance.

Problem duration

The 95th percentile job duration. One out of every 20 jobs processed is slower than the 95th percentile (equivalent to problem response time for web requests). {#problem-response}

Typical duration

The 50th percentile job duration. This number indicates the median of your processed jobs duration, which is to say that half of the jobs processed will be faster than this, while the other half will be slower (equivalent to typical response time for web requests).

Jobs processed

The total number of jobs that were processed in a given time period.

Queues

The queue in which a background job was run. This may be the default queue or a specific, named queue.

Frequency

The number of times this job was processed per minute. A higher frequency of a job means more jobs processed.

Agony

A weighted measure of how slow a job was and how frequently it was processed, ranging from 0 to 3; equivalent to our web request version of endpoint agony.

Additional Information

Pricing

Billing is based on the total number of monthly traces across all of your apps. Web requests and background jobs are both considered traces, and are treated equally in terms of billing. Every account gets 100,000 free traces per month, with tiers starting at $20 per month for more. Learn more on our pricing page.

Submitting Feedback

We welcome feedback and bug reports via the in-app messenger or by email at support@skylight.io.