Skip to content
Snippets Groups Projects
metrics.rst 8.47 KiB

Metrics

Satella's metrics

Metrics and instruments are a system to output real-time statistics.

Metrics are defined and meant to be used in a similar way to Python logging. It has a name (hierarchical, dot-separated), that does not have to correspond to particular modules or classes. It can be in one of 4 states:

  • DISABLED
  • RUNTIME
  • DEBUG
  • INHERIT

These are contained in a enum:

By default, it runs in RUNTIME mode. This means that statistics are collected only from metrics of this instrument that are set to at least RUNTIME. If a user wants to dig deeper, it can switch the instrument to DEBUG, which will cause more data to be registered. If a metric is in state INHERIT, it will inherit the metric level from it's parent, traversing the tree if required. The tree node separator is a dot in the metric's name, eg. satella.metrics.my_metric.

INHERIT is the default state for all other metrics than root, for root the default is RUNTIME. Root metric cannot be set to INHERIT, as it would not make sense.

Also, if parent is RUNTIME and child is DEBUG, the metrics reported by the child won't be included in parent metric data.

You can switch the metric anytime by assigning a correct value to it's level property, or by specifying it's metric level during a call to getMetric().

Note that a decision to accept/reject a handle()-provided value happens when handle() is called, based on current level. If you change the level, it may take some time for the metric to return correct values.

The call to getMetric() is specified as follows

You obtain metrics using getMetric() as follows:

metric = getMetric(__name__+'.StringMetric', 'string', MetricLevel.RUNTIME, **kwargs)

Please note that metric name must match the following regex:

[a-zA-Z_:][a-zA-Z0-9_:]*

internal is for those cases where the application is the consumer of the metrics, and you don't want them exposed to outside. Take care to examine this field of MetricData if you write custom exporters!

Where the second argument is a metric type. Following metric types are available:

  • base - for just a container metric

  • int - for int values

  • float - for float values

  • empty - disregard all provided values, outputs nothing

  • counter - starts from zero, increments or decrements the counter value. Also optionally can register the amount of calls

  • cps - will count given amount of calls to handle() during last time period, as specified by user

Note

Normally you should use a counter and calculate a rate() from it, but since some platforms suck at rate a decision was made to keep this.

  • linkfail - for tracking whether given link is online or offline

  • summary - a metric that counts a rolling window of values, and provides for a way to calculate percentiles. Corresponds to Prometheus' summary metrics.

  • histogram - a metric that puts given values into predefined buckets. Corresponds to Prometheus' histogram metric

  • callable - a metric whose value is a result of a given callable

  • uptime - a metric to report uptime

Note that metric.measure() will include time spent processing the generator's content by the client, so you might want to avoid measuring generators. However, if this is the behaviour that you want, you get it.

Note that if you request a different type of existing metric via getMetric, an MetricAlreadyExists exception will be raised:

Third parameter is optional. If set, all child metrics created during this metric's instantiation will receive such metric level. If the metric already exists, it's level will be set to provided metric level, if passed.

All child metrics (going from the root metric to 0) will be initialized with the value that you just passed. In order to keep them in order, an additional parameter passed to getMetric(), metric_level, if specified, will set given level upon returning the even existing metric.

This will be set on all children created by this call. If you have any children from previous calls, they will remain unaffected.

If you specify any kwargs, they will be delivered to the last metric's in chain constructor.

Since metrics in Satella are primarily though out to end up on a Prometheus, it is very important to understand Prometheus' data model.

Root metric's to_metric_data will output a flat set, called MetricDataCollection:

which consists of MetricData:

On most metrics you can specify additional labels. They will serve to create an independent "sub-metric" of sorts, eg.

metric = getMetric('root', 'int')
metric.runtime(2, label='value')
metric.runtime(3, label='key')
assert metric.to_metric_data() == MetricDataCollection(MetricData('root', 2, {'label': value}),
                                                       MetricData('root', 3, {'label': 'key}))

This functionality is provided by the below class:

Rolling your own metrics

In order to roll your own metrics, you must first subclass Metric. You can subclass one of the following classes, to the best of your liking. Please also refer to existing metric implementations on how to best subclass them.

Remember to define a class attribute of CLASS_NAME, which is a string defining how to call your metric. After everything is done, register it by using the following decorator on your metric class

To zip together two or more metrics, you can use the following class:

To automatically apply labels you can use this class:

Exporting data

In order to export data to Prometheus, you can use the following function:

For example in such a way:

def export_to_prometheus():
    metric = getMetric()
    return metric_data_collection_to_prometheus(metric.to_metric_data())

Dots in metric names will be replaced with underscores.

Or, if you need a HTTP server that will export metrics for Prometheus, use this class that is a daemonic thread you can use to easily expose metrics to Prometheus:

Useful data structures

Sometimes you want to have some data structures with metrics about themselves. Here go they: