Managing Hosts

VividCortex is a dynamic system that adapts to the presence or absence of hosts in your application, with no need to explicitly add or delete hosts. VividCortex does this by tracking the existence of hosts as a time-series fact. When agents are sending data to VividCortex, the host exists. When they stop, the host no longer exist. It’s that simple.

One of the benefits of this approach is that host/metric continuity is built in by design. If you decommission a server and replace it with a new server that has the same hostname, the metrics from the old and new host will be visible together in the VividCortex application, so the history of, say, the “master database host” can be viewed on a single chart even if it is spread across multiple physical servers over time.

Adding a Host

To add a host, just install and start the agents on the host. See the installation instructions for details.

Naming and Moving Hosts

When the agent is installed, it registers itself with the API. As part of this registration, it communicates the hostname and operating system of the host on which it is installed. The agent then uses this information when reporting collected metrics in order to identify the service being monitored.

By default, the agent combines the hostname and operating system to form the full name of the host and the identifier the agent will use. This is fine for most cases where the host operating system where the agents run will not change frequently. However, if the physical host where the agent runs is decomissioned and the agent is installed on a new host, the new data and old data will not be ‘tied together’ because it is seen as a new host.

To avoid this, you can override the default, application-given name of a host with your own name. This will allow you to start a new server later with the same name, and the application will see both servers as being identical. This allows the data to be continuous after the move. This is useful when the agent is running on servers that need to be reprovisioned and continue monitoring an off-host database instance without interruptions.

To enable this configuration, create a /etc/vividcortex/vc-agent-007.conf JSON file with the following format when first setting up the host that may later be reprovisioned:

  "hostname": "{{ unique_os_hostname }}"

This will override the hostname to unique_os_hostname. When the agent combines hostname and operating system to identify a host, it will use that hostname. It must be a unique identifier, and does not have to be the actual hostname of the server. For example, you could use the name rds-monitor-1 for a host which monitors an RDS instance.

Next, install VividCortex on the server as usual, and set up monitoring through the UI.

Later, if you wish to move the agent from one host to another but continue to monitor the same remote database with no discontinuity in data, first shutdown the agent on the original host. On the new host, ensure that the vc-agent-007.conf file has the same hostname on the new server (rds-monitor-1 in the example above), and install VividCortex on the new server. The off-host agent and credential configuration will follow to the new server. There should be no visible changes in the UI. The migration would look like the agent simply restarted on the same host.

Deleting a Host

To remove a host, simply shut down the agent on the host. After a timeout of two hours, the host will no longer be shown, but it remains in the historical views if you wish to look at it in the past. If you’d like to remove a host immediately, you can do so with the following steps:

  • Select the host in the Hosts Dashboard, and click on the gear that appears to the right of its name. You will see a popup like the one below:

    Delete Host Popup

  • Check the box labeled “I know what I’m doing”, and click on the Delete button to confirm deletion. Once deleted a host will automatically be removed from your license count. The host’s historical data will remain, but restarting the agent will create a new host rather than reuse the existing one.

Once a host is deleted it’s also a good idea to disable the service (for example, chkconfig --del vividcortex) so it doesn’t restart. You can also uninstall the agent permanently.

Moving a Host to a Different Environment

You can move a host to a new environment by stopping the agent, changing the /etc/vividcortex/global.conf file to contain the new environment’s API token, and restarting the agent. The API token can be found in the “Add Host” Wizard while in the destination environment. Note that it is not possible to move a host’s data to a new environment at this time. Environments are isolated from each other by design, and data is not portable across environments.

Host Quotas

Your environment may have a configured limit on the number of hosts that can be active at any one time. This is represented by the number of licenses available in your account. To view the number of used and total licenses, go to the Settings page, and under “Organization Overview”, check Licenses.

When the agent starts up, if it tries to register a new host with our API and there are too many active hosts already, it will not succeed and won’t start fully. You’ll see a message in the vc-agent-007 log, similar to the following:

unable to register OS (retrying in 1m0s): maximum number of hosts exceeded;
terminate some and wait 2h, or contact VividCortex

The agent will continue to retry until the number of active registered hosts is smaller than your quota. If you’ve deactivated an old host you no longer want, it may continue to be counted as active for up to two hours, hence the instruction to wait for 2 hours.

If you need a larger quota, please contact us with the support chat icon.

Missing Hosts

If you’ve installed the agent on a host but you don’t see the host in your dashboard, please check the following:

  • Is the agent running and sending data to our APIs? If not, after a timeout the host will be considered to have gone away. Check the agent logs in /var/log/vividcortex/.
  • Does the time selector in the top-right corner of the app include a time range where the host was active?
  • Is the host within your quota? See the previous section.