satella.instrumentation.memory package

Submodules

satella.instrumentation.memory.conditions module

class satella.instrumentation.memory.conditions.All(*conditions)

Bases: OperationJoin

This is true if all arguments are True

Parameters:

conditions (BaseCondition) –

class satella.instrumentation.memory.conditions.Any(*conditions)

Bases: OperationJoin

This is true if one of the arguments is True

Parameters:

conditions (BaseCondition) –

class satella.instrumentation.memory.conditions.BaseCondition

Bases: object

abstract can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

class satella.instrumentation.memory.conditions.CustomCondition(callable_)

Bases: BaseCondition

A custom condition. Condition that is true if attached callable/0 returns True.

Parameters:
  • callable – callable to call upon asking whether this condition is valid. This should be relatively cheap to compute.

  • callable_ (Callable[[], bool]) –

callable
can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

class satella.instrumentation.memory.conditions.GlobalAbsoluteValue(value)

Bases: MemoryCondition

If free memory globally falls below this many bytes, given severity level starts

Parameters:

value (int) –

can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

class satella.instrumentation.memory.conditions.GlobalRelativeValue(value)

Bases: MemoryCondition

If percentage of global free memory falls below this much percents, given severity level starts

Parameters:

value (int) –

can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

class satella.instrumentation.memory.conditions.LocalAbsoluteValue(value)

Bases: MemoryCondition

If free memory falls below this many bytes from what the program can maximally consume this severity level starts

Parameters:

value (int) –

can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

class satella.instrumentation.memory.conditions.LocalRelativeValue(value)

Bases: MemoryCondition

If percentage of memory available to this process in regards to what the program can maximally consume falls below this level, given severity level starts

Parameters:

value (int) –

can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

class satella.instrumentation.memory.conditions.Not(condition)

Bases: BaseCondition

True only if provided condition is false

Parameters:

condition (BaseCondition) –

can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

condition
class satella.instrumentation.memory.conditions.ZerothSeverity

Bases: BaseCondition

can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

satella.instrumentation.memory.default module

satella.instrumentation.memory.default.install_force_gc_collect(severity_level=1)

Install a default first severity level handler that forces a GC collection

Parameters:

severity_level (int) – severity level on which to call

Return type:

None

satella.instrumentation.memory.dump_frames_on module

satella.instrumentation.memory.dump_frames_on.dump_memory_on(output=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>)

Dump statistics about current Python memory usage to target stream.

Each Python object will be printed, along with a breakdown of most types and their total usage.

Make sure you have enough memory to generate a breakdown. You can preallocate something at the start for example.

Warning

This will return size of 0 on PyPy

Parameters:

output (TextIO) – output, default is stderr

satella.instrumentation.memory.dump_frames_on.install_dump_memory_on(signal_number, output=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>)

Instruct Python to dump all frames onto output, along with their local variables upon receiving given signal

Parameters:
  • signal_number – number of the signal

  • output (TextIO) – output

satella.instrumentation.memory.get_object_size module

satella.instrumentation.memory.get_object_size.get_size(obj, seen=None)

Recursively finds the total size of an object (object + it’s components).

Parameters:

obj – object to measure

Returns:

size in bytes of the object and all of it’s subcomponents

Raises:

RuntimeError – when ran on PyPy

Return type:

int

satella.instrumentation.memory.memthread module

class satella.instrumentation.memory.memthread.MemoryPressureManager(maximum_available=None, severity_levels=None, check_interval=10, log_transitions=True)

Bases: IntervalTerminableThread

Manager of the memory pressure.

The program is in some severity state. The baseline state is 0, meaning everything’s OK.

Please note that it is sufficient to instantiate this class for the thread to run.

Eg.

>>> mt = MemoryPressureManager(maximum_available=4*GB, severity_levels=[GlobalRelativeValue(20),
>>>                            GlobalRelativeValue(10)])
>>> @mt.register_on_severity(1)
>>> def trigger_a():
>>>     print('80% consumption of memory exceeded')
>>> @mt.register_on_severity(2)
>>> def trigger_b():
>>>     print('90% consumption of memory exceeded')

As well, this object is a singleton.

Parameters:
  • maximum_available – maximum amount of memory that this program can use

  • severity_levels – this defines the levels of severity. A level is reached when program’s consumption is other this many percent of it’s maximum_available amount of memory. Note that you need to specify only the abnormal memory levels, the default level of 0 will be added automatically.

  • check_interval – amount of seconds of pause between consecutive checks, or a time string

  • log_transitions – whether to log to logger when a transition takes place

Variables:

severity_level – current severity level (int) 0 means memory is OK, 1 and more means memory is progressively more limited

Note that this is called in the constructor’s thread. Use .prepare() to run statements that should be ran in new thread.

Parameters:

terminate_on – if provided, and loop() throws one of it, swallow it and terminate the thread by calling terminate(). Note that the subclass check will be done via isinstance so you can use the metaclass magic :) Note that SystemExit will be automatically added to list of terminable exceptions.

advance_to_severity_level(target_level)
Parameters:

target_level (int) –

calculate_severity_level()

This returns a severity level. 0 is the baseline severity level.

Return type:

int

static cleanup_on_entered(target_level, obj, collector=<function MemoryPressureManager.<lambda>>)

Attempt to recover memory by calling a particular method on an object.

A weak reference will be stored to this object

Parameters:
  • target_level (int) – cleanup will be attempted on entering this severity level

  • obj (Any) – object to call this on

  • collector (Callable[[Any], None]) – a lambda to call a routine on this object

loop()

Override me!

Return type:

None

static register_on_entered_severity(severity)

Register this handler to fire on entered a particular severity level.

This means that situation has gotten worse.

Use like this:

>>> MemoryPressureManager.register_on_entered_severity(1)
>>> def entered_severity_one():
>>>     print('Entered memory severity level 1')
Parameters:

severity (int) – severity level to react to

static register_on_left_severity(severity)

Register a handler to be called when given severity level is left. This means that we have advanced to a lower severity level.

>>> MemoryPressureManager.register_on_left_severity(1)
>>> def entered_severity_one():
>>>     print('Memory comsumption no longer 1')
Parameters:

severity (int) – severity level to leave

static register_on_memory_normal(fun)

Register this handler to fire when memory state falls back to 0.

This will be fired once, once memory state falls back to normal.

Parameters:

fun (Callable) – callable to register

Returns:

a CancellableCallback under this callback is registered

Return type:

CancellableCallback

static register_on_remaining_in_severity(severity, call_no_more_often_than=0)

Register this handler to fire on remaining in a particular severity level. Use like this:

>>> MemoryPressureManager.register_on_remaining_in_severity(0, 30)
>>> def entered_severity_one():
>>>     print('Memory comsumption OK. I am called no more often than each 30 seconds')
Parameters:
  • severity (int) – severity level

  • call_no_more_often_than (int) – call no more often than this amount of seconds

resume()

Resume the operation of this thread

stop()

Stop this thread from operating

Module contents

class satella.instrumentation.memory.All(*conditions)

Bases: OperationJoin

This is true if all arguments are True

Parameters:

conditions (BaseCondition) –

class satella.instrumentation.memory.Any(*conditions)

Bases: OperationJoin

This is true if one of the arguments is True

Parameters:

conditions (BaseCondition) –

class satella.instrumentation.memory.CustomCondition(callable_)

Bases: BaseCondition

A custom condition. Condition that is true if attached callable/0 returns True.

Parameters:
  • callable – callable to call upon asking whether this condition is valid. This should be relatively cheap to compute.

  • callable_ (Callable[[], bool]) –

callable
can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

class satella.instrumentation.memory.GlobalAbsoluteValue(value)

Bases: MemoryCondition

If free memory globally falls below this many bytes, given severity level starts

Parameters:

value (int) –

can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

class satella.instrumentation.memory.GlobalRelativeValue(value)

Bases: MemoryCondition

If percentage of global free memory falls below this much percents, given severity level starts

Parameters:

value (int) –

can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

class satella.instrumentation.memory.LocalAbsoluteValue(value)

Bases: MemoryCondition

If free memory falls below this many bytes from what the program can maximally consume this severity level starts

Parameters:

value (int) –

can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

class satella.instrumentation.memory.LocalRelativeValue(value)

Bases: MemoryCondition

If percentage of memory available to this process in regards to what the program can maximally consume falls below this level, given severity level starts

Parameters:

value (int) –

can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

class satella.instrumentation.memory.MemoryPressureManager(maximum_available=None, severity_levels=None, check_interval=10, log_transitions=True)

Bases: IntervalTerminableThread

Manager of the memory pressure.

The program is in some severity state. The baseline state is 0, meaning everything’s OK.

Please note that it is sufficient to instantiate this class for the thread to run.

Eg.

>>> mt = MemoryPressureManager(maximum_available=4*GB, severity_levels=[GlobalRelativeValue(20),
>>>                            GlobalRelativeValue(10)])
>>> @mt.register_on_severity(1)
>>> def trigger_a():
>>>     print('80% consumption of memory exceeded')
>>> @mt.register_on_severity(2)
>>> def trigger_b():
>>>     print('90% consumption of memory exceeded')

As well, this object is a singleton.

Parameters:
  • maximum_available – maximum amount of memory that this program can use

  • severity_levels – this defines the levels of severity. A level is reached when program’s consumption is other this many percent of it’s maximum_available amount of memory. Note that you need to specify only the abnormal memory levels, the default level of 0 will be added automatically.

  • check_interval – amount of seconds of pause between consecutive checks, or a time string

  • log_transitions – whether to log to logger when a transition takes place

Variables:

severity_level – current severity level (int) 0 means memory is OK, 1 and more means memory is progressively more limited

Note that this is called in the constructor’s thread. Use .prepare() to run statements that should be ran in new thread.

Parameters:

terminate_on – if provided, and loop() throws one of it, swallow it and terminate the thread by calling terminate(). Note that the subclass check will be done via isinstance so you can use the metaclass magic :) Note that SystemExit will be automatically added to list of terminable exceptions.

advance_to_severity_level(target_level)
Parameters:

target_level (int) –

calculate_severity_level()

This returns a severity level. 0 is the baseline severity level.

Return type:

int

static cleanup_on_entered(target_level, obj, collector=<function MemoryPressureManager.<lambda>>)

Attempt to recover memory by calling a particular method on an object.

A weak reference will be stored to this object

Parameters:
  • target_level (int) – cleanup will be attempted on entering this severity level

  • obj (Any) – object to call this on

  • collector (Callable[[Any], None]) – a lambda to call a routine on this object

loop()

Override me!

Return type:

None

static register_on_entered_severity(severity)

Register this handler to fire on entered a particular severity level.

This means that situation has gotten worse.

Use like this:

>>> MemoryPressureManager.register_on_entered_severity(1)
>>> def entered_severity_one():
>>>     print('Entered memory severity level 1')
Parameters:

severity (int) – severity level to react to

static register_on_left_severity(severity)

Register a handler to be called when given severity level is left. This means that we have advanced to a lower severity level.

>>> MemoryPressureManager.register_on_left_severity(1)
>>> def entered_severity_one():
>>>     print('Memory comsumption no longer 1')
Parameters:

severity (int) – severity level to leave

static register_on_memory_normal(fun)

Register this handler to fire when memory state falls back to 0.

This will be fired once, once memory state falls back to normal.

Parameters:

fun (Callable) – callable to register

Returns:

a CancellableCallback under this callback is registered

Return type:

CancellableCallback

static register_on_remaining_in_severity(severity, call_no_more_often_than=0)

Register this handler to fire on remaining in a particular severity level. Use like this:

>>> MemoryPressureManager.register_on_remaining_in_severity(0, 30)
>>> def entered_severity_one():
>>>     print('Memory comsumption OK. I am called no more often than each 30 seconds')
Parameters:
  • severity (int) – severity level

  • call_no_more_often_than (int) – call no more often than this amount of seconds

resume()

Resume the operation of this thread

stop()

Stop this thread from operating

class satella.instrumentation.memory.Not(condition)

Bases: BaseCondition

True only if provided condition is false

Parameters:

condition (BaseCondition) –

can_fire(local_memory_data, local_maximum_consume)

Has this severity level been reached?

Parameters:

local_maximum_consume (Optional[int]) –

Return type:

bool

condition
satella.instrumentation.memory.dump_memory_on(output=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>)

Dump statistics about current Python memory usage to target stream.

Each Python object will be printed, along with a breakdown of most types and their total usage.

Make sure you have enough memory to generate a breakdown. You can preallocate something at the start for example.

Warning

This will return size of 0 on PyPy

Parameters:

output (TextIO) – output, default is stderr

satella.instrumentation.memory.get_size(obj, seen=None)

Recursively finds the total size of an object (object + it’s components).

Parameters:

obj – object to measure

Returns:

size in bytes of the object and all of it’s subcomponents

Raises:

RuntimeError – when ran on PyPy

Return type:

int

satella.instrumentation.memory.install_dump_memory_on(signal_number, output=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>)

Instruct Python to dump all frames onto output, along with their local variables upon receiving given signal

Parameters:
  • signal_number – number of the signal

  • output (TextIO) – output

satella.instrumentation.memory.install_force_gc_collect(severity_level=1)

Install a default first severity level handler that forces a GC collection

Parameters:

severity_level (int) – severity level on which to call

Return type:

None