Having trouble with the agent? Take a look here.
Below are some troubleshooting steps you can try. If you’re still having trouble, please contact us via email at firstname.lastname@example.org.
If you’re having issues setting up Skylight, run
skylight doctor in your production terminal. You should see something like this as the output:
Checking for Rails Rails application detected Checking for native agent Native agent installed Checking for valid configuration Configuration is invalid authentication token required
heroku run bash.
If you don’t see any helpful information after running
skylight doctor, be sure to check out the rest of our troubleshooting guide below.
Verify that the application is running with the correct Rails environment. By default, the agent only starts in the
production environment, but this can be configured.
To learn how to change what environments Skylight starts in, see Environments.
skylightgem in the right group?
Verify that, in your app’s
Gemfile, you’ve added the
skylight gem to a group that will be installed in production. For example, if you add
skylight to the
development group, it will not run when you deploy to production.
By default, Skylight uses your Rails tmp path as the sockfile directory. In the event that this path isn’t writable you should set
daemon.sockdir_path in your config.
Skylight’s socket file can’t be located on an NFS mount. Set
daemon.sockdir_path in your config to a non-NFS path.
To make sure Skylight is activated, you may need to restart your Unicorn masters.
You might see this bug if the agent running in your Rails app stops reporting performance data to Skylight. If you were able to see data before but it has stopped working recently, restarting your server will usually fix the issue.
If the agent encounters multiple errors in a short span of time, it will shut itself down. This is done out of an abundance of caution to ensure that a potential bug in the agent doesn’t bring down your app in production.
We are working to add more logging to the agent so we can better diagnose what causes the agent to shutdown and recover gracefully in the event of an error. If you find this is happening regularly, please let us know!
First, make sure that there is traffic to your application. If your Rails app is handling requests, you should start to see data in Skylight in just a few minutes.
We have received reports of problems running on the
cedar-10 stack. You should upgrade to the
If your app is running and has traffic, but you’re still not seeing anything in Skylight, verify that the
skylight gem is installed and running properly.
Make sure you are using the latest version of the Skylight gem. In the directory for your Rails app, run this command:
bundle list | grep skylight
You should see something like the following output:
* skylight (version number)
If the gem is installed and up-to-date, the next step is to verify that it is running correctly:
heroku logs -tto show the application log.
heroku restartto restart the app.
Keep an eye on the log terminal. If you see an error message stating that the Skylight agent is missing an authentication token, verify that you have configured Skylight in Heroku properly.
This is due to our use of two different compression algorithms for the data. In some rare cases, especially when there were only one or two requests in a range, there’s a slight mismatch between the algorithms. We hope to resolve this in future iterations.
To avoid taking your production application down due to an installation failure, Skylight does not raise an exception when it can’t install the native agent. Currently, we support all *nix servers with glibc 2.15+ and Mac OS X 10.8+. We also have experimental support for FreeBSD and musl. If you’re running a compatible OS and still see errors, try running your application with
SKYLIGHT_REQUIRED=true. This will cause Skylight to raise an exception when the native agent is missing. This exception may be useful in troubleshooting the problem. If you need help, send this to us at email@example.com.
First, try syncing your GitHub account with Skylight. If that doesn’t work, it’s possible the GitHub organization that owns the repo needs to whitelist Skylight in order for us to see that you are a member of the repo. Have an admin for that GitHub org add Skylight to its whitelist, then sync your GitHub account again. You should then have access to the app(s) you were expecting to see!
If you aren’t seeing your organization in the dropdown when trying to connect an app or your users are having issues accessing your GitHub-connected apps, you may have your GitHub organization’s “Third-party application access policy” set to “Access restricted.”
You can check our current permissions by visiting the “Third-party access” settings for your organization. If the permissions are correct, Skylight will be marked as “Approved”:
Your organization must allow third-party access to Skylight so that we can use the GitHub API to access any information about your org. Click the link below the “Connect Your GitHub Repository” dropdown on the app settings page to update your GitHub permissions, which will whitelist Skylight. This link takes you to a page in GitHub, where you can click the “Grant” button to give Skylight access to the necessary information:
There are a few reasons why this might happen.
Sorry, right now we only support repos that are connected to an organization, though we may allow use of personal repos in the future. Shoot us an email at firstname.lastname@example.org to let us know if this feature is important to you!
This is actually a known bug, but it’s quite rare these days and we’ve had trouble reproducing it, so have had trouble fixing it. Please do email email@example.com and let us know exactly what steps you took when signing up, especially if there were any errors or issues along the way, or if you refreshed the invitation signup page at all before clicking “sign up with GitHub.” Any information about anything out of the ordinary helps!
In the meantime, you can ask the person who initially invited you to send another invitation to the email address associated with your account. This will add you to the app automatically and you should have no further issues.
The Skylight agent will log errors to
log/skylight.log or on Heroku in STDOUT (Look for lines starting with
[E0001] Spans were closed out of orderand
invalid span nesting
This error indicates that a parent span (event sequence item) was closed before all of its children were.
One common cause of this issue is a Middleware that doesn’t conform to the Rack SPEC. Specifically, “If the body is replaced by a middleware after action, the original body must be closed first, if it responds to
close.” If you are unable to fix the Middleware, you can remove the Middleware probe with
config.skylight.probes -= ['middleware'] in your Rails config (note that you will no longer see individual Middleware in your endpoints list or endpoint event sequences; all Middleware will show up as “Rack” instead).
[E0002] You've exceeded the number of unique span descriptions per-request.and
A payload description produced <too many uniques>
Skylight limits the number of unique span (event sequence item) descriptions in a request to 100 to prevent misbehaving apps from causing trouble. In most cases, this will be fine as even items such as similar repeated queries will be normalized into a single description (e.g.
SELECT * FROM users WHERE id = 1 and
SELECT * FROM users WHERE id = 2 both become
SELECT * FROM users WHERE id = ?). However, if your request is extremely complex it is possible to exceed this limit. To resolve this issue, reduce the number of uniquely named items that you instrument.
[E0003]Exceeded maximum number of spans
Skylight limits the maximum number of spans (event sequent items) in a request to 2,048 to prevent a data overload. If your request is very complex, you may hit this limit and data will no longer be tracked for that request. A couple common causes include:
[E0004] Failed to extract binds from SQL query.and
Failed to lex SQL query
These errors indicate that the agent is unable to parse a SQL query in your application. There errors won’t prevent your application from operating, though they will reduce the information that we can display in the UI for these queries. They will be displayed in the Event Sequence as simply “SQL”, not the full sanitized query.
Generally, the reason you will see this error is because you’re using a syntax we do not recognize (often a more complex or non-standard syntax). We’ve optimized for the most common syntax constructions and plan to support more in the future. When running across this error, please feel free to report it so we can learn what queries are important to our customers.
ERROR:skylight::cli: skylightd exiting abnormally; err=DaemonLockFailed
As per this issue in the agent repo, the fix for this is relatively quick. Just explicitly set the
sockfile_path in your
config/skylight.yml file like so:
daemon: sockdir_path: /tmp/my-app-name
If that doesn’t work and you are using an earlier version of Skylight, you can also try:
agent: sockfile_path: /tmp/my-app-name
If that still doesn’t work, drop us a line and we’d be happy to help debug!
Memory profiler gems like
derailed generally use Ruby’s
ObjectSpace—a really nifty way to get information about all the objects allocated in your Ruby application that is great for troubleshooting memory issues.
Digging into the source of both
memory_profiler (which is used by derailed), we discovered the following line (source):
file = ObjectSpace.allocation_sourcefile(obj)
This line looks at an object and asks
ObjectSpace who caused it to be allocated. As it turns out, objects allocated at require time are attributed to the file that calls the original
Now, this is all well and good except for one small problem: Skylight overwrites
Kernel#require in order to properly install probes. This means that it is Skylight that calls the original
Kernel#require, not whatever other gem is really doing the require. So, as soon as Skylight is loaded, every future
require call is attributed to Skylight.
Skylight isn’t the only library to do this. ActiveSupport has a similar hook in it’s
Loadable module that is included into
Object. Without Skylight, it would be ActiveSupport taking the blame. However, because this hook uses
super, it calls to the superclass
Kernel that is now the version that Skylight created.
So there you have it, run
derailed without Bundler and you’ll see RubyGems blamed for your requires. Add in Bundler and the correct files will get blamed. Bring in ActiveSupport and then ActiveSupport will be blamed. Include Skylight and blame will shift to us. Good times!
So what’s the moral here? Know your tools.
ObjectSpace and the libraries that use it do some really useful things. They aren’t perfect, but they’re open source. Read the code and find out a bit about how they work and you’ll be able to put them to better use!
Skylight performs setup when the gem is required, at which time Rails will be detected and tapped into. However, you may find you have to manually require
skylight/railtie if you need to load the Skylight gem before Rails.
Unfortunately, it’s possible that your app has been the target of some malicious bots or abusive requests. Even if you see very few page views, that is not an accurate indicator of how many requests your app is receiving behind the scenes. An overzealous health checker is another possible source, if you are using such a service.
We strongly recommend that you look into the source of the abusive requests so they don’t continue or return. Many Skylight users have reported success using something like Rack Attack to deal with these requests. You can also use a library like IPCat with Rack Attack to access a list of known IPs for “datacenters, co-location centers, shared and virtual webhosting providers. In other words, ip addresses that end web consumers should not be using.”
If you want to avoid seeing these requests in Skylight, we recommend placing Rack::Attack at the top of your middleware stack (good practice even without Skylight) and then running Skylight immediately afterward.
If you find yourself in this position, here are some tips (from an actual Skylight customer!) on how you might deal with finding and blocking the abusive requests: