satella.instrumentation.memory package¶
Submodules¶
satella.instrumentation.memory.conditions module¶
- class satella.instrumentation.memory.conditions.All(*conditions)¶
Bases:
OperationJoin
This is true if all arguments are True
- Parameters:
conditions (BaseCondition) –
- class satella.instrumentation.memory.conditions.Any(*conditions)¶
Bases:
OperationJoin
This is true if one of the arguments is True
- Parameters:
conditions (BaseCondition) –
- class satella.instrumentation.memory.conditions.BaseCondition¶
Bases:
object
- abstract can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- class satella.instrumentation.memory.conditions.CustomCondition(callable_)¶
Bases:
BaseCondition
A custom condition. Condition that is true if attached callable/0 returns True.
- Parameters:
callable – callable to call upon asking whether this condition is valid. This should be relatively cheap to compute.
callable_ (Callable[[], bool]) –
- callable¶
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- class satella.instrumentation.memory.conditions.GlobalAbsoluteValue(value)¶
Bases:
MemoryCondition
If free memory globally falls below this many bytes, given severity level starts
- Parameters:
value (int) –
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- class satella.instrumentation.memory.conditions.GlobalRelativeValue(value)¶
Bases:
MemoryCondition
If percentage of global free memory falls below this much percents, given severity level starts
- Parameters:
value (int) –
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- class satella.instrumentation.memory.conditions.LocalAbsoluteValue(value)¶
Bases:
MemoryCondition
If free memory falls below this many bytes from what the program can maximally consume this severity level starts
- Parameters:
value (int) –
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- class satella.instrumentation.memory.conditions.LocalRelativeValue(value)¶
Bases:
MemoryCondition
If percentage of memory available to this process in regards to what the program can maximally consume falls below this level, given severity level starts
- Parameters:
value (int) –
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- class satella.instrumentation.memory.conditions.Not(condition)¶
Bases:
BaseCondition
True only if provided condition is false
- Parameters:
condition (BaseCondition) –
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- condition¶
- class satella.instrumentation.memory.conditions.ZerothSeverity¶
Bases:
BaseCondition
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
satella.instrumentation.memory.default module¶
- satella.instrumentation.memory.default.install_force_gc_collect(severity_level=1)¶
Install a default first severity level handler that forces a GC collection
- Parameters:
severity_level (int) – severity level on which to call
- Return type:
None
satella.instrumentation.memory.dump_frames_on module¶
- satella.instrumentation.memory.dump_frames_on.dump_memory_on(output=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>)¶
Dump statistics about current Python memory usage to target stream.
Each Python object will be printed, along with a breakdown of most types and their total usage.
Make sure you have enough memory to generate a breakdown. You can preallocate something at the start for example.
Warning
This will return size of 0 on PyPy
- Parameters:
output (TextIO) – output, default is stderr
- satella.instrumentation.memory.dump_frames_on.install_dump_memory_on(signal_number, output=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>)¶
Instruct Python to dump all frames onto output, along with their local variables upon receiving given signal
- Parameters:
signal_number – number of the signal
output (TextIO) – output
satella.instrumentation.memory.get_object_size module¶
- satella.instrumentation.memory.get_object_size.get_size(obj, seen=None)¶
Recursively finds the total size of an object (object + it’s components).
- Parameters:
obj – object to measure
- Returns:
size in bytes of the object and all of it’s subcomponents
- Raises:
RuntimeError – when ran on PyPy
- Return type:
int
satella.instrumentation.memory.memthread module¶
- class satella.instrumentation.memory.memthread.MemoryPressureManager(maximum_available=None, severity_levels=None, check_interval=10, log_transitions=True)¶
Bases:
IntervalTerminableThread
Manager of the memory pressure.
The program is in some severity state. The baseline state is 0, meaning everything’s OK.
Please note that it is sufficient to instantiate this class for the thread to run.
Eg.
>>> mt = MemoryPressureManager(maximum_available=4*GB, severity_levels=[GlobalRelativeValue(20), >>> GlobalRelativeValue(10)]) >>> @mt.register_on_severity(1) >>> def trigger_a(): >>> print('80% consumption of memory exceeded') >>> @mt.register_on_severity(2) >>> def trigger_b(): >>> print('90% consumption of memory exceeded')
As well, this object is a singleton.
- Parameters:
maximum_available – maximum amount of memory that this program can use
severity_levels – this defines the levels of severity. A level is reached when program’s consumption is other this many percent of it’s maximum_available amount of memory. Note that you need to specify only the abnormal memory levels, the default level of 0 will be added automatically.
check_interval – amount of seconds of pause between consecutive checks, or a time string
log_transitions – whether to log to logger when a transition takes place
- Variables:
severity_level – current severity level (int) 0 means memory is OK, 1 and more means memory is progressively more limited
Note that this is called in the constructor’s thread. Use .prepare() to run statements that should be ran in new thread.
- Parameters:
terminate_on – if provided, and
loop()
throws one of it, swallow it and terminate the thread by callingterminate()
. Note that the subclass check will be done via isinstance so you can use the metaclass magic :) Note that SystemExit will be automatically added to list of terminable exceptions.
- advance_to_severity_level(target_level)¶
- Parameters:
target_level (int) –
- calculate_severity_level()¶
This returns a severity level. 0 is the baseline severity level.
- Return type:
int
- static cleanup_on_entered(target_level, obj, collector=<function MemoryPressureManager.<lambda>>)¶
Attempt to recover memory by calling a particular method on an object.
A weak reference will be stored to this object
- Parameters:
target_level (int) – cleanup will be attempted on entering this severity level
obj (Any) – object to call this on
collector (Callable[[Any], None]) – a lambda to call a routine on this object
- loop()¶
Override me!
- Return type:
None
- static register_on_entered_severity(severity)¶
Register this handler to fire on entered a particular severity level.
This means that situation has gotten worse.
Use like this:
>>> MemoryPressureManager.register_on_entered_severity(1) >>> def entered_severity_one(): >>> print('Entered memory severity level 1')
- Parameters:
severity (int) – severity level to react to
- static register_on_left_severity(severity)¶
Register a handler to be called when given severity level is left. This means that we have advanced to a lower severity level.
>>> MemoryPressureManager.register_on_left_severity(1) >>> def entered_severity_one(): >>> print('Memory comsumption no longer 1')
- Parameters:
severity (int) – severity level to leave
- static register_on_memory_normal(fun)¶
Register this handler to fire when memory state falls back to 0.
This will be fired once, once memory state falls back to normal.
- Parameters:
fun (Callable) – callable to register
- Returns:
a CancellableCallback under this callback is registered
- Return type:
- static register_on_remaining_in_severity(severity, call_no_more_often_than=0)¶
Register this handler to fire on remaining in a particular severity level. Use like this:
>>> MemoryPressureManager.register_on_remaining_in_severity(0, 30) >>> def entered_severity_one(): >>> print('Memory comsumption OK. I am called no more often than each 30 seconds')
- Parameters:
severity (int) – severity level
call_no_more_often_than (int) – call no more often than this amount of seconds
- resume()¶
Resume the operation of this thread
- stop()¶
Stop this thread from operating
Module contents¶
- class satella.instrumentation.memory.All(*conditions)¶
Bases:
OperationJoin
This is true if all arguments are True
- Parameters:
conditions (BaseCondition) –
- class satella.instrumentation.memory.Any(*conditions)¶
Bases:
OperationJoin
This is true if one of the arguments is True
- Parameters:
conditions (BaseCondition) –
- class satella.instrumentation.memory.CustomCondition(callable_)¶
Bases:
BaseCondition
A custom condition. Condition that is true if attached callable/0 returns True.
- Parameters:
callable – callable to call upon asking whether this condition is valid. This should be relatively cheap to compute.
callable_ (Callable[[], bool]) –
- callable¶
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- class satella.instrumentation.memory.GlobalAbsoluteValue(value)¶
Bases:
MemoryCondition
If free memory globally falls below this many bytes, given severity level starts
- Parameters:
value (int) –
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- class satella.instrumentation.memory.GlobalRelativeValue(value)¶
Bases:
MemoryCondition
If percentage of global free memory falls below this much percents, given severity level starts
- Parameters:
value (int) –
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- class satella.instrumentation.memory.LocalAbsoluteValue(value)¶
Bases:
MemoryCondition
If free memory falls below this many bytes from what the program can maximally consume this severity level starts
- Parameters:
value (int) –
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- class satella.instrumentation.memory.LocalRelativeValue(value)¶
Bases:
MemoryCondition
If percentage of memory available to this process in regards to what the program can maximally consume falls below this level, given severity level starts
- Parameters:
value (int) –
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- class satella.instrumentation.memory.MemoryPressureManager(maximum_available=None, severity_levels=None, check_interval=10, log_transitions=True)¶
Bases:
IntervalTerminableThread
Manager of the memory pressure.
The program is in some severity state. The baseline state is 0, meaning everything’s OK.
Please note that it is sufficient to instantiate this class for the thread to run.
Eg.
>>> mt = MemoryPressureManager(maximum_available=4*GB, severity_levels=[GlobalRelativeValue(20), >>> GlobalRelativeValue(10)]) >>> @mt.register_on_severity(1) >>> def trigger_a(): >>> print('80% consumption of memory exceeded') >>> @mt.register_on_severity(2) >>> def trigger_b(): >>> print('90% consumption of memory exceeded')
As well, this object is a singleton.
- Parameters:
maximum_available – maximum amount of memory that this program can use
severity_levels – this defines the levels of severity. A level is reached when program’s consumption is other this many percent of it’s maximum_available amount of memory. Note that you need to specify only the abnormal memory levels, the default level of 0 will be added automatically.
check_interval – amount of seconds of pause between consecutive checks, or a time string
log_transitions – whether to log to logger when a transition takes place
- Variables:
severity_level – current severity level (int) 0 means memory is OK, 1 and more means memory is progressively more limited
Note that this is called in the constructor’s thread. Use .prepare() to run statements that should be ran in new thread.
- Parameters:
terminate_on – if provided, and
loop()
throws one of it, swallow it and terminate the thread by callingterminate()
. Note that the subclass check will be done via isinstance so you can use the metaclass magic :) Note that SystemExit will be automatically added to list of terminable exceptions.
- advance_to_severity_level(target_level)¶
- Parameters:
target_level (int) –
- calculate_severity_level()¶
This returns a severity level. 0 is the baseline severity level.
- Return type:
int
- static cleanup_on_entered(target_level, obj, collector=<function MemoryPressureManager.<lambda>>)¶
Attempt to recover memory by calling a particular method on an object.
A weak reference will be stored to this object
- Parameters:
target_level (int) – cleanup will be attempted on entering this severity level
obj (Any) – object to call this on
collector (Callable[[Any], None]) – a lambda to call a routine on this object
- loop()¶
Override me!
- Return type:
None
- static register_on_entered_severity(severity)¶
Register this handler to fire on entered a particular severity level.
This means that situation has gotten worse.
Use like this:
>>> MemoryPressureManager.register_on_entered_severity(1) >>> def entered_severity_one(): >>> print('Entered memory severity level 1')
- Parameters:
severity (int) – severity level to react to
- static register_on_left_severity(severity)¶
Register a handler to be called when given severity level is left. This means that we have advanced to a lower severity level.
>>> MemoryPressureManager.register_on_left_severity(1) >>> def entered_severity_one(): >>> print('Memory comsumption no longer 1')
- Parameters:
severity (int) – severity level to leave
- static register_on_memory_normal(fun)¶
Register this handler to fire when memory state falls back to 0.
This will be fired once, once memory state falls back to normal.
- Parameters:
fun (Callable) – callable to register
- Returns:
a CancellableCallback under this callback is registered
- Return type:
- static register_on_remaining_in_severity(severity, call_no_more_often_than=0)¶
Register this handler to fire on remaining in a particular severity level. Use like this:
>>> MemoryPressureManager.register_on_remaining_in_severity(0, 30) >>> def entered_severity_one(): >>> print('Memory comsumption OK. I am called no more often than each 30 seconds')
- Parameters:
severity (int) – severity level
call_no_more_often_than (int) – call no more often than this amount of seconds
- resume()¶
Resume the operation of this thread
- stop()¶
Stop this thread from operating
- class satella.instrumentation.memory.Not(condition)¶
Bases:
BaseCondition
True only if provided condition is false
- Parameters:
condition (BaseCondition) –
- can_fire(local_memory_data, local_maximum_consume)¶
Has this severity level been reached?
- Parameters:
local_maximum_consume (Optional[int]) –
- Return type:
bool
- condition¶
- satella.instrumentation.memory.dump_memory_on(output=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>)¶
Dump statistics about current Python memory usage to target stream.
Each Python object will be printed, along with a breakdown of most types and their total usage.
Make sure you have enough memory to generate a breakdown. You can preallocate something at the start for example.
Warning
This will return size of 0 on PyPy
- Parameters:
output (TextIO) – output, default is stderr
- satella.instrumentation.memory.get_size(obj, seen=None)¶
Recursively finds the total size of an object (object + it’s components).
- Parameters:
obj – object to measure
- Returns:
size in bytes of the object and all of it’s subcomponents
- Raises:
RuntimeError – when ran on PyPy
- Return type:
int
- satella.instrumentation.memory.install_dump_memory_on(signal_number, output=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>)¶
Instruct Python to dump all frames onto output, along with their local variables upon receiving given signal
- Parameters:
signal_number – number of the signal
output (TextIO) – output
- satella.instrumentation.memory.install_force_gc_collect(severity_level=1)¶
Install a default first severity level handler that forces a GC collection
- Parameters:
severity_level (int) – severity level on which to call
- Return type:
None