Troubleshooting VividCortex

If you have any trouble with VividCortex and the answers below don’t resolve it, please contact us with the chat icon in the bottom left navigation bar.

I see data in the Profiler, but I don’t see charts.

You can see the queries because there are two database-specific plugins. One plugin decodes the database network protocol, recording queries directly, so it doesn’t need credentials. On the other hand, the metrics plugin needs to connect to the database to get the server’s status and other data, so it needs credentials. If you don’t see any charts with database status metrics, it’s likely that the agent is having trouble logging into the database to get the status metrics. Check the Events dashboard for any information the agent may have sent about its error, or check /var/log/vividcortex/ for any error messages.

I provided credentials, but the agent cannot connect.

First, please double-check that you have created a user correctly in your database and can log in as that user with the specified password. To verify this in MySQL, which has “anonymous” users that can cause unexpected results, issue the SHOW GRANTS command after logging in. The column name should be something like Grants for user@host. Verify that this is as expected.

A common problem with this process in MySQL is specifying which host the agent is allowed to connect from. The VividCortex agent will only connect over TCP/IP, not a Unix socket. If you’ve specified the allowed host in the GRANT statement as user@locahost, then you’ve created a MySQL user account that only permits Unix socket logins. You will need to specify a hostname or IP address, such as user@127.0.0.1, to permit a TCP/IP login.

You can check to mimic how the agent will log into the database server, substituting appropriate values below:

$ mysql --protocol=tcp -uUSER -pPASSWORD -hHOST --port=PORT
ERROR 1045 (28000): Access denied for user 'USER'@'127.0.0.1' (using password: YES)

Please note the user from which access was denied. If this doesn’t match how you’ve created the user, that could be the problem.

You may also have specified a wrong host, such as specifying 127.0.0.1 for the host to which the agent should connect, when MySQL is not listening on that IP address. If this is the case, you’ll get an error message similar to this:

ERROR 2003 (HY000): Can't connect to MySQL server on 'HOST' (61)

After you get everything working, please be sure to remove any unwanted users, and be sure to grant all necessary privileges to the new user you create to replace it.

I do not see queries that I know are executing.

Our agent only captures queries sent to the server over unencrypted TCP/IP connections, not Unix sockets. We won’t see the following types of queries:

  • Queries sent to localhost over a Unix socket
  • Queries sent over an encrypted connection
  • Queries executed from stored routines (stored procedures, events, or triggers)
  • Queries that execute via replication

We don’t analyze server log files to capture queries; read more about why if you are curious.

If none of the above seems to be the culprit, something else may be happening. Contact us and we will troubleshoot right away.

I was invited to join VividCortex, but my invitation expired.

When someone invites you to join a VividCortex account, the invitation sent to your email is valid for 10 days. We can help you renew it if necessary.

I do not see agents installed on a new host.

If you’re using a free trial or you’re on a capped contract, you may have exceeded your quota. Check the /var/log/vividcortex/vc-agent-007.log log file for messages to that effect. Contact us if you need to increase your quota or if you’re interested in evaluating VividCortex on more hosts during your trial.

Also see Missing Hosts on the Managing Hosts page.

How do I remove hosts?

If you’d like to deactivate a host so it doesn’t contribute towards your usage quota, you can do so with the following steps:

  • Select the host in the Hosts Dashboard, and click on the gear that appears to the right of its name. You will see a popup like the one below:

Delete Host Popup

  • Check the box labeled “I know what I’m doing”, and click on the Delete button to confirm deletion. Once deleted a host will automatically be removed from your license count.

You may also uninstall the agents and wait a bit; because usage is metered by the hour, the number of active hosts in use will decrease after two hours.

Why don’t I see any database or table size metrics in the Profiler?

Database and table size metrics, such as data_length and index_length, are disabled by default as collecting those metrics can be costly. To enable them, you can either contact VividCortex Customer Support (recommended) by chatting with us using the ? button in the lower right corner, or you can enable table sizes using a VividCortex config file with the option enable-table-sizes. This option applies to the vc-mysql-metrics and vc-pgsql-metrics plugins.

Why don’t I see any InnoDB mutex data?

If you do not see any data reported in the Profiler or Metrics pages for InnoDB mutexes, check that the wait/synch/mutex/innodb/% instruments are enabled in performance_schema. To check if these are enabled, execute:

SELECT * FROM performance_schema.setup_instruments where NAME like 'wait/synch/mutex/innodb/%'

To enable these instruments, you can either add

performance-schema-instrument='wait/synch/mutex/innodb/%=ON'

to a my.cnf or execute:

UPDATE performance_schema.setup_instruments SET ENABLED='YES' WHERE NAME LIKE 'wait/synch/mutex/innodb/%';

The instruments can be enabled without restarting the server if you use the UPDATE statement, however not all of the instruments will be active until a restart. See the MySQL documentation for more information about these instruments.

How can I change the name of an environment?

At the moment, environment names can only be changed by VividCortex Customer Support. Please contact us using the ? button in the lower right corner of the application.

Why are my query samples truncated?

By default, we limit the length of captured query samples to 4096 bytes. If your queries are longer than this length, you can contact VividCortex Customer Support to have this lengthened, or you can use a VividCortex config file and use the option max-sample-length.

If your queries are being truncated and they are smaller than the max-sample-length and you are using the Off-Host configuration, it is likely that MySQL or PostgreSQL is shortening the queries in PERFORMANCE_SCHEMA or pg_stat_statements. Both MySQL and PostgreSQL truncate queries at 1024 characters by default. For MySQL, you can change this value using the performance_schema_max_sql_text_length variable in version 5.7.6 (and later). For PostgreSQL, use the track_activity_query_size variable.

Why am I not seeing query samples?

In on-host installations, the most common reason for not seeing query samples is either that they are either disabled for some or all of your hosts by your administrator, or you are logged in as a user without proper permissions to view query samples; Read-Write permissions are required for customers using RBAC. We will also be unable to capture prepared statements which are executed as part of a long-running connection.

In off-host configurations, the most common reasons are not having events_statements_history_long (MySQL) or pg_stat_activity (PostgreSQL) enabled and collecting query data, or that the VividCortex user does not have sufficient permissions to access that data. We are also unable to collect query samples for prepared statements for MySQL (as query samples are not made available by the database). PostgreSQL does not provide query samples with the literals included, so we do not show this text as it is identical to the query digest.

Lastly, query samples may be disabled for some or all of your hosts by your account administrator.

Why are some queries not displaying EXPLAIN plans?

We need query samples in order to see EXPLAIN plans, so first ensure you are collecting query samples (above) for the queries you want to have EXPLAINed.

If you have query samples, we will not collect EXPLAIN plans for queries which could cause unpredictable load to the server during an EXPLAIN, such queries containing a sub-query. The VividCortex user connecting to the database also needs the proper permissions to execute EXPLAIN.

Lastly, EXPLAIN plans can be disabled by your account administrator.

Why are some of my PostgreSQL queries displaying <insufficient privilege>?

Typically this occurs when the user VividCortex is connecting to the database with does not have the SUPERUSER privilege or the rds_superuser role. In this case, <insufficient privilege> will typically appear as much or most of the query volume. You can read more about the required database privileges here.

However, some query volume for hosted PostgreSQL instances on Amazon, such as RDS and Heroku, will always display as <insufficient privilege> due to Amazon using a custom user (such as rdsadmin) to perform monitoring and maintenance functions, including fetching CloudWatch metrics. The queries being run are not viewable by any user, including users who have been assigned the rds_superuser role. The same applies to Heroku, where an application (“Dyno”) gets a dedicated user and database assigned to it, but one that shares the host server with other applications’ databases.