Troubleshooting

Having trouble with the agent? Take a look here.

Below are some troubleshooting steps you can try. If you’re still having trouble, please contact us via email at support@skylight.io.

Skylight Doctor

If you’re having issues setting up Skylight, run skylight doctor in your production terminal. You should see something like this as the output:

Checking for Rails
  Rails application detected

Checking for native agent
  Native agent installed

Checking for valid configuration
  Configuration is invalid
    authentication token required
Pro Tip: On Heroku you can access your production terminal with heroku run bash.

If you don’t see any helpful information after running skylight doctor, be sure to check out the rest of our troubleshooting guide below.

Data Not Reporting

I deployed my application, but no data is appearing in Skylight.

Is your environment correct?

Verify that the application is running with the correct Rails environment. By default, the agent only starts in the production environment, but this can be configured.

To learn how to change what environments Skylight starts in, see Railtie Environments.

Is the skylight gem in the right group?

Verify that, in your app’s Gemfile, you’ve added the skylight gem to a group that will be installed in production. For example, if you add skylight to the development group, it will not run when you deploy to production.

If you’re using Sinatra or Grape without Rails, did you follow the installation instructions?

Is the sockfile path writable?

By default, Skylight uses your Rails tmp path as the sockfile directory. In the event that this path isn’t writable you should set daemon.sockdir_path in your config.

Are you using NFS, possibly with Vagrant?

Skylight’s socket file can’t be located on an NFS mount. Set daemon.sockdir_path in your config to a non-NFS path.

If you’re running Unicorn, did you restart your master?

To make sure Skylight is activated, you may need to restart your Unicorn masters.

Skylight was working before, but now I’m only seeing “No requests in this time range” when viewing an app.

You might see this bug if the agent running in your Rails app stops reporting performance data to Skylight. If you were able to see data before but it has stopped working recently, restarting your server will usually fix the issue.

If the agent encounters multiple errors in a short span of time, it will shut itself down. This is done out of an abundance of caution to ensure that a potential bug in the agent doesn’t bring down your app in production.

We are working to add more logging to the agent so we can better diagnose what causes the agent to shutdown and recover gracefully in the event of an error. If you find this is happening regularly, please let us know!

I’m using Heroku but no data is showing up in Skylight.

First, make sure that there is traffic to your application. If your Rails app is handling requests, you should start to see data in Skylight in just a few minutes.

We have received reports of problems running on the cedar-10 stack. You should upgrade to the cedar-14 stack.

If your app is running and has traffic, but you’re still not seeing anything in Skylight, verify that the skylight gem is installed and running properly.

Make sure you are using the latest version of the Skylight gem. In the directory for your Rails app, run this command:

bundle list | grep skylight

You should see something like the following output:

* skylight (version number)

If the gem is installed and up-to-date, the next step is to verify that it is running correctly:

  1. In one terminal window, run heroku logs -t to show the application log.
  2. In another terminal window, run heroku restart to restart the app.

Keep an eye on the log terminal. If you see an error message stating that the Skylight agent is missing an authentication token, verify that you have configured Skylight in Heroku properly.

Platform Issues

The Skylight native extension wasn’t found, but my platform claims to be supported.

To avoid taking your production application down due to an installation failure, Skylight does not raise an exception when it can’t install the native agent. Currently, we support Linux 2.6.18+ and Mac OS X 10.8+. If you’re running a compatible OS and still see errors, try running your application with SKYLIGHT_REQUIRED=true. This will cause Skylight to raise an exception when the native agent is missing. This exception may be useful in troubleshooting the problem. If you need help, send this to us at support@skylight.io.

GitHub Integration Issues

I logged in with GitHub but still don’t see the GitHub-connected app(s) I’m expecting to see!

First, try syncing your GitHub account with Skylight. If that doesn’t work, it’s possible the GitHub organization that owns the repo needs to whitelist Skylight in order for us to see that you are a member of the repo. Have an admin for that GitHub org add Skylight to its whitelist, then sync your GitHub account again. You should then have access to the app(s) you were expecting to see!

I followed an email invitation and signed up with GitHub. How come I don’t see any apps when I log in?

This is actually a known bug, but it’s quite rare these days and we’ve had trouble reproducing it, so have had trouble fixing it. Please do email support@skylight.io and let us know exactly what steps you took when signing up, especially if there were any errors or issues along the way, or if you refreshed the invitation signup page at all before clicking “sign up with GitHub.” Any information about anything out of the ordinary helps!

In the meantime, you can ask the person who initially invited you to send another invitation to the email address associated with your account. This will add you to the app automatically and you should have no further issues.

I’m trying to add a repo to my Skylight app but I don’t see the field to do so.

There are a few reasons why this might happen.

  1. You are not the owner of the app on Skylight. You may contact the owner to have them add a repo to the app, or have them transfer ownership to you (this would entail transferring billing to you as well, so you may not want to do that).
  2. You are the owner, but are not logged in with GitHub. We need to verify your credentials, so be sure you are logged in via GitHub.

I’m having problems connecting my Skylight app to a personal GitHub repo.

Sorry, right now we only support repos that are connected to an organization, though we may allow use of personal repos in the future. Shoot us an email at support@skylight.io to let us know if this feature is important to you!

Skylight Errors

The Skylight agent will log errors to log/skylight.log or on Heroku in STDOUT (Look for lines starting with [SKYLIGHT]).

Event Errors

failed to instrument span; msg=native Trace#start_span failed

This error occurs when an event is too complex for Skylight’s agent to track. Here are a few potential causes for this issue:

  1. The request itself is very complex. For example, requests that generate a significant number of SQL queries have been known to trigger this error. Be sure to check out our Performance Tips to reduce the complexity of your request.
  2. Your custom instrumentation is overly complex. Be sure to follow our custom instrumentation best practices.

SQL Query Errors

internal error: failed to lex SQL query

These errors indicate that the agent is unable to parse a SQL query in your application. There errors won’t prevent your application from operating, though they will reduce the information that we can display in the UI for these queries. They will be displayed in the Event Sequence as simply “SQL”, not the full sanitized query.

Generally, the reason you will see this error is because you’re using a syntax we can’t recognize, often a more complex syntax or a non-standard syntax. For example, the agent doesn’t currently support square brackets allowed by MS SQL and Postgres. We’ve optimized for the most common syntax constructions and hope to support more in the future. When running across this error, please feel free to report it so we can learn what queries we’re failing to parse.

Other Errors

ERROR:skylight::cli: skylightd exiting abnormally; err=DaemonLockFailed

As per this issue in the agent repo, the fix for this is relatively quick. Just explicitly set the sockfile_path in your config/skylight.yml file like so:

daemon:
  sockdir_path: /tmp/my-app-name

If that doesn’t work and you are using an earlier version of Skylight, you can also try:

agent:
  sockfile_path: /tmp/my-app-name

If that still doesn’t work, drop us a line and we’d be happy to help debug!

Other Issues

I ran a memory profiler. Why are so many object allocations attributed to Skylight?

Memory profiler gems like memory_profiler and derailed generally use Ruby’s ObjectSpace—a really nifty way to get information about all the objects allocated in your Ruby application that is great for troubleshooting memory issues.

Digging into the source of both derailed and memory_profiler (which is used by derailed), we discovered the following line (source):

1file = ObjectSpace.allocation_sourcefile(obj)

This line looks at an object and asks ObjectSpace who caused it to be allocated. As it turns out, objects allocated at require time are attributed to the file that calls the original Kernel#require method.

Now, this is all well and good except for one small problem: Skylight overwrites Kernel#require in order to properly install probes. This means that it is Skylight that calls the original Kernel#require, not whatever other gem is really doing the require. So, as soon as Skylight is loaded, every future require call is attributed to Skylight.

Skylight isn’t the only library to do this. ActiveSupport has a similar hook in it’s Loadable module that is included into Object. Without Skylight, it would be ActiveSupport taking the blame. However, because this hook uses super, it calls to the superclass Kernel that is now the version that Skylight created.

Even RubyGems itself gets in on the overwriting behavior. Though, if you use Bundler (as everyone is), this is actually reversed.

So there you have it, run derailed without Bundler and you’ll see RubyGems blamed for your requires. Add in Bundler and the correct files will get blamed. Bring in ActiveSupport and then ActiveSupport will be blamed. Include Skylight and blame will shift to us. Good times!

So what’s the moral here? Know your tools. ObjectSpace and the libraries that use it do some really useful things. They aren’t perfect, but they’re open source. Read the code and find out a bit about how they work and you’ll be able to put them to better use!

What if I need to load the Skylight gem before Rails?

Skylight performs setup when the gem is required, at which time Rails will be detected and tapped into. However, you may find you have to manually require skylight/railtie if you need to load the Skylight gem before Rails.

I’m suddenly seeing many more requests than usual and my bill has gone way up, but my app has very few page views. What’s going on?

Unfortunately, it’s possible that your app has been the target of some malicious bots or abusive requests. Even if you see very few page views, that is not an accurate indicator of how many requests your app is receiving behind the scenes. An overzealous health checker is another possible source, if you are using such a service.

We strongly recommend that you look into the source of the abusive requests so they don’t continue or return. Many Skylight users have reported success using something like Rack Attack to deal with these requests. If you block any requests using Rack Attack and are using the 1.4.0beta agent to instrument middleware, you will begin to see Rack::Attack listed as an endpoint in Skylight, which you can then ignore if you like. If you are not instrumenting your middleware, Rack::Attack will not be separated out into its own endpoint. Instead, you will only see the Rack endpoint (which you would then need to ignore).

You may gain additional insight by reordering or instrumenting your middleware. If you find that a particular middleware is causing the issue and none of the above suggestions have helped, you can configure Skylight to run after the problematic middleware.