Geneos
Geneos
Recommended Reads

RMC Plug-In - Technical Reference

Note: This is an old version of the RMC Interface plug-in documentation. For the current version, see RMC Interface Plug-in Technical Reference.

Introduction

The Geneos RMC Plug-in brings RMC parameters of any Refinitiv Market Data System (TREP) component seamlessly and in real-time into the GENEOS 'world'. These parameters become fully GENEOS manageable.

In addition to translating RMC parameters into native variables, GENEOS RMC Knowledge Module enables effective monitoring by providing a 'value added' layer on top of RMC. This layer is based on defining a number of user configurable templates that define mapping of the RMC parameters.

This additional layer provides the following benefits:

Hide RMC variables organisational complexity - ability to concentrate on the key parameters to manage. Each RMC compatible component, such as the ADH, for example, will have thousands of RMC parameters available, some duplicated. These parameters are organised in a highly hierarchical tree structure, making it difficult, confusing and time consuming to find and manage the key parameters.

Remove SSL version dependency. The names and hierarchical positions of the RMC parameters vary between different versions of SSL for the same component.

Provide rate of change calculation. Actual values of some of the parameters, such as the total number of updates, for example, are irrelevant by themselves. The rate of change of those parameters is much more useful.

Views

View

RMC Interface Plug-in allows great flexibility in the way views can be defined. Each instance of the RMC Plug-in will create a view with a title as per the name provided in the SamplerList. Mulitple RMC samplers can be processed by the same netprobe in order to obtain a full range of parameters.

Example Views:

rmc2

rmc3

rmc4

Plug-in Configuration

template

An RMC based component is typically capable of providing thousands of managed variables. A template is set up to focus the plug-in on a number of managed variables that are of significant importance. Each NetProbe can support up to 20 RMC Plug-ins with different templates.

There are three possible sections in the template: headlines, table and records. Each section defines a mapping between the GENEOS variables and the RMC variables for a specific RMDS component.

A template may either contain a table section with or without headlines or a records section. No other combinations are allowed.

In order to construct a template to monitor a RMDS component, the full list of the RMC variables supplied by the component should be available. This can be achieved by running the rmcRecorder program provided (see later section).

If you do not specify the template in the setup file, you must use the interfaceFile setting.

Mandatory: No

rmcTemplate > headlines

This section defines up to six headline-type variables that are shown at the top of the view.

E.g.:

headlines
        headline
                name: connAvailable
                rmcName:host.$instance.sink_dist.sink.server.ipc.trnBus/Connections Available
headlines
        headline
                name: connUsed
                rmcName: $host.$instance.sink_dist.sink.server.ipc.trnBus /Connections Used"
headlines
        headline
                name: connEnabled
                rmcName: $host.$instance.sink_dist.sink.server.ipc.trnBus /Connections Enabled

Mandatory: No

rmcTemplate > headlines > headline > name

The title to display against the headline on the view.

Mandatory: Yes

rmcTemplate > headlines > headline > rmcName

The RMC name identifying the source managed variable.

E.g.:

host.$instance.sink_dist.sink.server.ipc.transmissionBus/Connections Available

See also $HOST and $INSTANCE Substitutions.

Mandatory: Yes

rmcTemplate > table

This section constructs a table in the view. This should be used when the full variable names are not known at the time the template is created.

For example, when monitoring Sink Distributor User Mounts it is not possible to fully specify the variable names, as new mounts may come and go. Instead, a pattern that the variable names fall into is specified.

E.g.:

tables
        table
                recordSeed: $host.$instance.sink_dist.User Database.mount<key>.<field>                keyLike
                        matchElement: TEXT
                        matchElement: NUMBER
                fields
                        field
                                name: name
                                value: Attributes/Consumer Name
                        field
                                name: position
                                value: Attributes/Position
                        field
                                name: appID
                                value: Attributes/Application Key
                        field
                                name: open
                                value: Data Streams/Total
                showKey: true
                unifiedStrings
                        unifiedString: srv01.de.db.com
                        unifiedString: srv02.de.db.com
                        unifiedString: *.uk.db.com

Mandatory: No

rmcTemplate > table > recordSeed

Only managed variables that match the record seed will be shown in the table.

E.g.:

$host.$instance.sink_dist.User Database.mount<key>.<field>

See also $HOST and $INSTANCE Substitutions.

Mandatory: Yes

rmcTemplate > table > keyLike

The <key> in the variable name must map to a text string that is unique across all the records in the RMC data stream.

Sometimes it is necessary to only pick the records that have the <key> that match a specific pattern. For example, when monitoring Service instances the key must have a text followed by a number. If this pattern restriction is not specified, the Service summary records (text without number) will also be displayed.

<Key> pattern can be specified using combinations of matchElements in any order.

Match Element Description
NUMBER Matches any combination of 0-9 up to a '.'
TEXT Matches any combination of characters up to a '.' or a number.
ANY Matches any combination of characters up to a '.'

Mandatory: Yes

rmcTemplate > table > fields

Define the fields that will be shown in the table.

Mandatory: Yes

rmcTemplate > table > fields > field > name

The name of the column that shows this field.

Mandatory: Yes

rmcTemplate > table > fields > field > value

The RMC name identifying the source managed variable.

E.g. Attributes/Consumer Name

rmcTemplate > table > fields > field > keyElement

If the <key> pattern is composed of multiple elements, this setting adds a column to display a corresponding element, denoted by its position in the <key> pattern.

For a <key> pattern matching "route.2209.dataStreams", key element 1 would show "route", element 2 "2209", and element 3 "dataStreams".

rmcTemplate > table > fields > field > keyTemplate

This setting displays the matched <key> pattern in the format specified by the template. The key template can be any combination of the key elements and string literals.

For a <key> pattern matching "route.2209.dataStreams", the following key template will display the key as "route-2209-dataStreams":

keyTemplate:
        keyElement: 1
        literal: -
        keyElement: 2
        literal: -
        keyElement: 3
        literal: -

rmcTemplate > table > fields > field > strip

Some RMC variables contain additional information that may not be desired, this command can be used to truncate the contents of a variable at a predefined position.

For example, if a username is root/192.168.10.23 then setting below will make it root.

field
        name: username
        value: ".dataStreams/login"
        strip
                after: /
field
        name: username
        value: ".dataStreams/login"
        strip
                before: /

Mandatory: No

rmcTemplate > table > fields > field > showAsRateNew

Some RMC variables contain counters that are incremented since the RMDS component is started e.g. Total Messages Received. In itself this number is not useful, however the rate of change of this number (Message Receive Rate) is a useful parameter to monitor.

The contents of this node allow the rate of change of a variable to be shown.

Mandatory: No

rmcTemplate > table > fields > field > showAsRateNew > showAsRate

Setting this parameter to true will cause the plug-in to show rate of change of the value being monitored.

Mandatory: No
Default: False

rmcTemplate > table > fields > field > showAsRateNew > unsignedWrap

Setting this parameter will cause the plug-in to monitor for unsigned wrap-around in the value of the variable being monitored.

This avoids incorrect rate of change being reported.

Mandatory: No

rmcTemplate > table > fields > field > showAsRateNew > signedWrap

Setting this parameter will cause the plug-in to monitor for signed wrap-around in the value of the variable being monitored.

This avoids incorrect rate of change being reported.

Mandatory: No

rmcTemplate > table > fields > field > showAsRateNew > calculationMethod

This parameter changes how the rate is calculated.

Changed between syncs is the default method used when no method is explicitly set in the RMC configuration. This method exhibits the gradually declining rate, as it calculates the rate based on the difference between the current value and the last value received into the datastore from a sync. This means that the rate is calculated based on the current value and the last value read and stored by the plugin not equal to the current value. The gradually declining rate is expected.

Example:
<update> 2012/05/08 14:33:30 CentOS5.6x86-64.VM.Base.MNL.1.src_dist.netRecDB.8211.incomingMessages/state = value: 184509
<update> 2012/05/08 14:33:31 CentOS5.6x86-64.VM.Base.MNL.1.src_dist.netRecDB.8211.incomingMessages/state = value: 184510
<update> 2012/05/08 14:33:35 CentOS5.6x86-64.VM.Base.MNL.1.src_dist.netRecDB.8211.incomingMessages/state = value: 184511
<update> 2012/05/08 14:33:36 CentOS5.6x86-64.VM.Base.MNL.1.src_dist.netRecDB.8211.incomingMessages/state = value: 184512

For a 1-second sampling interval, the first sample would show a state field rate as follows:

sample 1: rate = (184510 - 184509) / (14:33:31 - 14:33:30) = 1.00 per second

The succeeding samples would show a declining rate until the state field is updated:

sample 2: rate = (184510 - 184509) / (14:33:32 - 14:33:30) = 0.50 per second
sample 3: rate = (184510 - 184509) / (14:33:33 - 14:33:30) = 0.33 per second
sample 4: rate = (184510 - 184509) / (14:33:34 - 14:33:30) = 0.25 per second
sample 5: rate = (184511 - 184510) / (14:33:35 - 14:33:31) = 0.25 per second
sample 6: rate = (184512 - 184511) / (14:33:36 - 14:33:35) = 1.00 per second

Changed between samples calculation method calculates the rate based on the difference between the current value and the value when the previous sample occurred. This means that the rate is calculated based on the current and the last sampled value, which could be the same value.

For the example feed above, the rate would be shown using this calculation method as follows:

sample 1: rate = (184510 - 184509) / (14:33:31 - 14:33:30) = 1.00 per second

Since the state field is not updated, the current value and the last sampled value in next sample would be the same. The rate should remain at 0 until the sampled value is changed:

sample 2: rate = (184510 - 184510) / (14:33:32 - 14:33:30) = 0.00 per second
sample 3: rate = (184510 - 184510) / (14:33:33 - 14:33:30) = 0.00 per second
sample 4: rate = (184510 - 184510) / (14:33:34 - 14:33:30) = 0.00 per second
sample 5: rate = (184511 - 184510) / (14:33:35 - 14:33:31) = 0.25 per second
sample 6: rate = (184512 - 184511) / (14:33:36 - 14:33:35) = 1.00 per second

Mandatory: No

rmcTemplate > table > fields > field > showAsPercentage

This feature allows for the creation of new column(s) which are based on existing field data. These are a percentage representation of two data items.

For example, the column 'percentUsage' is a percent calculation of actualusers as a percentage of maxUsers:

field
        name: actualUsers
        value: ".dataStreams/actuaUsers"
field
        name: maxUsers
        value: ".dataStreams/login"
field
        name: percentUsage
        numerator: actualUsers
        denominator:  maxUsers

Mandatory: No

rmcTemplate > table > showKey

This parameter may be specified to produce a field in the display which will indicate how the key has been constructed. This can be useful for initial setting up of the template.

Mandatory: No
Default: False

rmcTemplate > table > unifiedStrings

This is an optional parameter that indicates the specified "unified-strings" are to be considered as a single element that matches a single element in the pattern specified with keyLike.

E.g. If a.b.c is specified as a unified string, the record "a.b.c.1.d" will match the pattern TEXT+NUMBER+ TEXT.

Wildcards can be used in unified strings. Each '*' will match a single dot-separated element in the key. E.g. *.a.b will match xxx.a.b but not xxx.yyy.a.b.

Mandatory: No

rmcTemplate > records

This section constructs a list of variables in the view. This should be used when the variable names are known at the time the template is created. This section is similar to the headlines section, except the variables are displayed in a list and the number of the variables is not limited to six - currently limited to 1000. The other difference from the headlines section is that records section must be in the template on its own - a template cannot have both a LIST section and a TABLE section.

E.g.:

records
        records
                name: bytesReceived
                value: $host.$instance.sink_dist.sink.server.ipc.transmissionBus/Bytes Received
        records
                name: requestsOpen
                value: $host.$instance.sink_dist.User Database.User Requests/Open
        records
                name: requestsClosed
                value: $host.$instance.sink_dist.User Database.User Requests/Close

Mandatory: No

rmcTemplate records > record > name

The title to display against the records on the view.

Mandatory: Yes

rmcTemplate > records > record > value

The RMC name identifying the source managed variable.

E.g.:

$host.$instance.sink_dist.sink.server.ipc.transmissionBus/Bytes Received

See also $HOST and $INSTANCE Substitutions.

Mandatory: Yes

applications

Configure a predefined look up table which is used to perform a translation between the contents of the collected variable to a matching name.

E.g.:

applications
        application
                name: PDD
                key: 38
        application
                name: Kobra
                key: 56
        application
                name: DDS
                key: 27
        application
                name: Openlink
                key: 70

Mandatory: No

instance

Instance number of the RMC component being monitored.

Mandatory: Yes

sharedMemoryKey

Shared memory used by the RMC component being monitored.

Mandatory: No
Default: 81

interfaceFile

Name of the file that holds the downloaded Template.

Mandatory: No
Default: RMC_<sampler name>.rif.download

mtTime

Amount of time to spend processing incoming data in main thread in milliseconds

Mandatory: No

process

Settings relating to the publishing process.

Mandatory: No

process > name

Name of the actual publishing process that writes to the shared memory (e.g ./p2ps). If this is set then the plug-in will verify that the process is up to validate the shared memory state.

See Shared Memory Status Detection for more detail.

Mandatory: No

process > args

Arguments passed to the actual publishing process. This helps to identity the process - e.g -install .gw

See Shared Memory Status Detection for more detail.

sharedMemoryTest

Enables the shmget() shared memory tests within the RMC plugin. This is enabled by default. When disabled, the plugin proceeds with monitoring without checking the shared memory status. Disable only when diagnosing syncing issues between the plugin and the RMC component/s being monitored.

Default: Enabled

Timing Record > Use timing record

Enables shared memory status detection method using time statistics read from the shared memory. This can be used together with the default shared memory status test.

See Shared Memory Status Detection for more detail.

Mandatory: No

timingRecord > timeOut

Time out value in seconds.

See Shared Memory Status Detection for more detail.

Default: 15

timingRecord > variableName

Name of the variable that contains the timing record from the shared memory statistics.

See Shared Memory Status Detection for more detail.

Mandatory: NormcTemplate > headlines

This section defines up to six headline-type variables that are shown at the top of the view.

E.g.:

headlines
        headline
                name: connAvailable
                rmcName:host.$instance.sink_dist.sink.server.ipc.trnBus/Connections Available
headlines
        headline
                name: connUsed
                rmcName: $host.$instance.sink_dist.sink.server.ipc.trnBus /Connections Used"
headlines
        headline
                name: connEnabled
                rmcName: $host.$instance.sink_dist.sink.server.ipc.trnBus /Connections Enabled

Mandatory: No

rmcTemplate > headlines > headline > name

The title to display against the headline on the view.

Mandatory: Yes

rmcTemplate > headlines > headline > rmcName

The RMC name identifying the source managed variable.

E.g.:

host.$instance.sink_dist.sink.server.ipc.transmissionBus/Connections Available

See also $HOST and $INSTANCE Substitutions.

Mandatory: Yes

rmcTemplate > headlines > headline > showAsRate

Some RMC variables contain counters that are incremented since the RMDS component is started e.g. Total Messages Received. In itself this number is not useful, however the rate of change of this number (Message Receive Rate) is a useful parameter to monitor.

Setting this parameter to true will cause the plug-in to show rate of change of the value being monitored.

Mandatory: No
Default: False

rmcTemplate > table

This section constructs a table in the view. This should be used when the full variable names are not known at the time the template is created.

For example, when monitoring Sink Distributor User Mounts it is not possible to fully specify the variable names, as new mounts may come and go. Instead, a pattern that the variable names fall into is specified.

E.g.:

tables
        table
                recordSeed: $host.$instance.sink_dist.User Database.mount<key>.<field>                keyLike
                        matchElement: TEXT
                        matchElement: NUMBER
                fields
                        field
                                name: name
                                value: Attributes/Consumer Name
                        field
                                name: position
                                value: Attributes/Position
                        field
                                name: appID
                                value: Attributes/Application Key
                        field
                                name: open
                                value: Data Streams/Total
                showKey: true
                unifiedStrings
                        unifiedString: srv01.de.db.com
                        unifiedString: srv02.de.db.com
                        unifiedString: *.uk.db.com

Mandatory: No

rmcTemplate > table > recordSeed

Only managed variables that match the record seed will be shown in the table.

E.g.:

$host.$instance.sink_dist.User Database.mount<key>.<field>

See also $HOST and $INSTANCE Substitutions.

Mandatory: Yes

rmcTemplate > table > keyLike

The <key> in the variable name must map to a text string that is unique across all the records in the RMC data stream.

Sometimes it is necessary to only pick the records that have the <key> that match a specific pattern. For example, when monitoring Service instances the key must have a text followed by a number. If this pattern restriction is not specified, the Service summary records (text without number) will also be displayed.

<Key> pattern can be specified using combinations of matchElements in any order.

Match Element Description
NUMBER Matches any combination of 0-9 up to a '.'
TEXT Matches any combination of characters up to a '.' or a number.
ANY Matches any combination of characters up to a '.'

Mandatory: Yes

rmcTemplate > table > fields

Define the fields that will be shown in the table.

Mandatory: Yes

rmcTemplate > table > fields > field > name

The name of the column that shows this field.

Mandatory: Yes

rmcTemplate > table > fields > field > value

The RMC name identifying the source managed variable.

E.g. Attributes/Consumer Name

rmcTemplate > table > fields > field > keyElement

If the <key> pattern is composed of multiple elements, this setting adds a column to display a corresponding element, denoted by its position in the <key> pattern.

For a <key> pattern matching "route.2209.dataStreams", key element 1 would show "route", element 2 "2209", and element 3 "dataStreams".

rmcTemplate > table > fields > field > keyTemplate

This setting displays the matched <key> pattern in the format specified by the template. The key template can be any combination of the key elements and string literals.

For a <key> pattern matching "route.2209.dataStreams", the following key template will display the key as "route-2209-dataStreams":

keyTemplate:
        keyElement: 1
        literal: -
        keyElement: 2
        literal: -
        keyElement: 3
        literal: -

rmcTemplate > table > fields > field > strip

Some RMC variables contain additional information that may not be desired, this command can be used to truncate the contents of a variable at a predefined position.

For example, if a username is root/192.168.10.23 then setting below will make it root.

field
        name: username
        value: ".dataStreams/login"
        strip
                after: /
field
        name: username
        value: ".dataStreams/login"
        strip
                before: /

Mandatory: No

rmcTemplate > table > fields > field > showAsRate

Some RMC variables contain counters that are incremented since the RMDS component is started e.g. Total Messages Received. In itself this number is not useful, however the rate of change of this number (Message Receive Rate) is a useful parameter to monitor.

Setting this parameter to true will cause the plug-in to show rate of change of the value being monitored.

Mandatory: No
Default: False

rmcTemplate > table > fields > field > showAsPercentage

This feature allows for the creation of new column(s) which are based on existing field data. These are a percentage representation of two data items.

For example, the column 'percentUsage' is a percent calculation of actualusers as a percentage of maxUsers:

field
        name: actualUsers
        value: ".dataStreams/actuaUsers"
field
        name: maxUsers
        value: ".dataStreams/login"
field
        name: percentUsage
        numerator: actualUsers
        denominator:  maxUsers

Mandatory: No

rmcTemplate > table > showKey

This parameter may be specified to produce a field in the display which will indicate how the key has been constructed. This can be useful for initial setting up of the template.

Mandatory: No
Default: False

rmcTemplate > table > unifiedStrings

This is an optional parameter that indicates the specified "unified-strings" are to be considered as a single element that matches a single element in the pattern specified with keyLike.

E.g. If a.b.c is specified as a unified string, the record "a.b.c.1.d" will match the pattern TEXT+NUMBER+ TEXT.

Wildcards can be used in unified strings. Each '*' will match a single dot-separated element in the key. E.g. *.a.b will match xxx.a.b but not xxx.yyy.a.b.

Mandatory: No

rmcTemplate > records

This section constructs a list of variables in the view. This should be used when the variable names are known at the time the template is created. This section is similar to the headlines section, except the variables are displayed in a list and the number of the variables is not limited to six - currently limited to 1000. The other difference from the headlines section is that records section must be in the template on its own - a template cannot have both a LIST section and a TABLE section.

E.g.:

records
        records
                name: bytesReceived
                value: $host.$instance.sink_dist.sink.server.ipc.transmissionBus/Bytes Received
        records
                name: requestsOpen
                value: $host.$instance.sink_dist.User Database.User Requests/Open
        records
                name: requestsClosed
                value: $host.$instance.sink_dist.User Database.User Requests/Close

Mandatory: No

rmcTemplate > records > record > name

The title to display against the records on the view.

Mandatory: Yes

rmcTemplate > records > record > value

The RMC name identifying the source managed variable.

E.g.:

$host.$instance.sink_dist.sink.server.ipc.transmissionBus/Bytes Received

See also $HOST and $INSTANCE Substitutions.

Mandatory: No

rmcTemplate > records > record > showAsPercentage

This feature allows for the creation of new column(s) which are based on existing field data. These are a percentage representation of two data items.

For example, the field 'percentOpen' is a percent calculation of requestOpen as a percentage of maxRequests:

records
                name: requestsOpen
                value: $host.$instance.sink_dist.User Database.User Requests/Open
        records
                name: maxRequests
                value: $host.$instance.sink_dist.User Database.User Requests/Max
record
        name: percentOpen
        numerator: requestsOpen
        denominator:  maxRequests

Mandatory: No

private

Private testing and debugging settings.

Mandatory: No

$HOST and $INSTANCE Substitutions

RMC variable names can include $host and $instance. In this case the $host is replaced with the hostname of the machine the plug-in is running on and $instance is replaced with the INSTANCE variable of the SAMPLER_DESCRIPTOR.

Recording / Playback of the RMC Datastream

A facility to record an RMC data stream for a specific RMC component on site and then "play it back" off-site is a powerful way of creating/debugging templates.

A recorder program called rmcRecorder is provided. Run it with -v and -k command line options. The -k option must be followed by the shared memory number that the RMC component is using (usually 81).

The output from the program should be written into a file. Typically the file name should reflect the name and the version number of the RMC component.

For example:

rmcRecorder -v -k 81 | tee Sink_ssl4.1.06L18.dat

To "play back" the file, set the GENERAL_DEBUG environment variable in start_netprobe to "canned". Copy the .dat file into the NetProbe directory and rename it to source.dat.

The variable host (where XXX is the hostname of the Managed Entity that the rmcrecorder was taken from) must also be added to the RMC Sampler Descriptor.

Any number of RMC Plug-ins can be defined for "play-back". They will all share the data stream.

Running RMC Replayer

The RMC Replayer is a test utility for replaying RMC Recorder files (*.dat) as live RMC data.

To run the program, you would need to modify the hostname written inside the Recorder file with the hostname that would be running it. Also, update RMC Interface->Debug->Host field using the same hostname.

rmc5

The content of a Recorder file usually follows the format:

<hostname>.<instance>.<application_name>.<path_names> = <value>

Rename the RMC Replayer binary with the application name defined inside the Recorder file. Afterwards you can now run the executable with -f, -k, and -s command line options. The -f option is followed by the path to the RMC Recorder file. The -k option must be followed by the shared memory number that the RMC component is using. And the -s option contains the memory size allocated to run the RMC Replayer.

Sample syntax:

./<application_name> -f <rmcRecorder_file> -k <shared_memory_key> -s <memory_allocation>

Shared Memory Status Detection

There are three modes that this plug-in uses to detect the state of a publishing service (e.g. p2ps, src_dist, sink_dist).

  • Default mode

    • The default mode of operation verifies that the shared memory segment identified by the key is available, as well as if the creator process of the shared memory segment is up. This mode can also detect if a shared memory segment is reused by a restarted RMC component, and allows the plugin to resync with the new process.

  • Process Check Mode

    • Alternatively another mode of operation is to check if the actual publishing process itself is up. This can is done by setting the process parameters in the Sampler Descriptor.
    • Parameter process > name specifies is the process name (e.g. / p2ps)
    • Parameter process > args specifies the command line arguments passed to this process (e.g. -install .gw). This is an optional parameter. If ARGS is not set, the aguments are not checked.

  • Timing Record Check Mode

    • This is an alternate mode of operation to check if the publishing process is up and the data provided is current, in the case where an existing shared memory segment may have gone stale and is no longer being updated. Most but not all of the publishing services provide Shared Memory Statistics information. This data is updated on a 5 second interval. By setting the correct Timing Record parameters in the Sampler Descriptor, we can use this information to detect if the shared memory is running:
    • Timing Record > Use timing record specifies if we want to enable this mode of shared memory detection.
    • Timing Record > Time out specifies the maximum amount of time in seconds before declaring the shared memory is invalid since the last update received.
    • Timing Record > Variable name specifies the variable that holds the timing record information.
    • Example:
      $host.$instance.src_dist.admin.MemoryStats/time = value: 1336501974
    • Using this mode alone, the plugin may not be able to recover and resync with a restarted RMC component if the previous instance of the component crashed or was forcibly killed (kill -9), and the shared memory resources were manually removed (ipcrm). It is advisable to use this mode together with either the default mode or the process check mode.