REST Extractor

Overview Copied

The REST Extractor plug-in enables you to parse and extract fields from JSON content and turn these into dataviews in Geneos.

The plug-in supports JSON responses coming from a web server or URL.

Dataviews are configured through the use of jq queries. The REST Extractor plug-in uses the jq for Jackson library. For more information, see Geneos Compatibility Matrix.

Intended audience Copied

This guide is directed towards Geneos users who want to set up a plug-in to display JSON content in a dataview.

As a user, you should be familiar with REST API. Knowledge of jq is required in order to create specific dataviews.

Prerequisites Copied

Java requirements Copied

The Java installation and environment configuration is a common source of errors for users setting up Java-based components and plug-ins. It is recommended to read Configure the Java environment to help you understand your Java installation.

Source URL and credentials Copied

You need the following to set up the REST Extractor plug-in:

Setup and configuration Copied

Setup involves the following tasks:

  1. Create the REST Extractor sampler.
  2. Associate the REST Extractor sampler with a managed entity.

Note

If you are using this plugin with Gateway Hub, you must create a user defined data schema. For instructions, see Create a data schema.

Create the REST Extractor sampler Copied

  1. In the Gateway Setup Editor, create a new sampler by right-clicking the Samplers folder and selecting New Sampler.

  2. Enter a name for this sampler. For example, enter “rest-extractor” in the Name field.

  3. Under the Plugin field, click the drop-down list and select rest-extractor.

  4. Under Dataviews, enter the information in the fields that appear:

    • In the Name field, enter the name you want for your dataview.

    • In the Url field, enter the source URL from which the REST Extractor plug-in expects a JSON response.

    • In the Http method field, select your preferred method. By default, the GET method is used.

    • In the Query field, enter the jq query you want to use to transform the JSON response into your desired dataview.

      Note

      For guidance in using jq for Jackson in queries, see Example queries, responses, and dataviews.

    • In the Row limit field, enter the number of rows you would like the dataview to display.

      Note

      You can toggle between data and var for the Row limit field. This toggle option allows you to define either a text or numerical value (data) or variable (var) for this field.

  5. If you want the fields from the JSON response to appear in a certain order on the dataview, click the Columns button. Then, click Add new to add the fields in the order you want on the dataview.

    Note

    The REST Extractor plug-in does not require you to define the columns for the sampler to run. However, we recommend that you establish an order for the columns. For guidance on how to use the columns feature, see Column order.

  6. If you want to add more dataviews to the sampler, click Add new under the Query box. Then, enter the information in the fields that appear, similar to the previous step.

  7. If you want to include any HTTP header in the HTTP request, click the Headers button. Then, click Add new to provide the headers in key-value pairs.

  8. If authentication is needed to access the source URL, click the Authentication button.

    Note

    The REST Extractor plug-in supports basic and bearer authentication, as well as no authentication. The authentication setting depends on the authentication setup in the system that you are monitoring, if any. For guidance in setting up basic or bearer authentication, see Plugin configuration.

  9. Click Save current document to apply your changes.

    Success

    The sampler can now be associated with a managed entity. This sampler provides the dataviews you have defined.

Associate the REST Extractor sampler with a managed entity Copied

  1. In the Gateway Setup Editor, create a new managed entity by right-clicking the Managed entities folder and selecting New Managed entity.
  2. Enter a name for this managed entity. For example, enter “rest-extractor” in the Name field.
  3. In the Options field, select the probe on which you want the sampler to run.
  4. Under the Sampler field, click Add new.
  5. In the text field under Ref, select the sampler you just created from the drop-down list.
  6. Click Save current document to apply your changes.

    Success

    The REST Extractor dataviews appear under the managed entity on the Active Console state tree.

Dataviews Copied

The REST Extractor plug-in works by accepting queries. Each query creates a new dataview on the sampler.

For more information on creating queries, see Example queries, responses, and dataviews.

Response format Copied

The REST Extractor plug-in expects the following JSON format for query responses:

[
 {
   "column1": "dataA1",
   "column2": "dataA2",
   "column3": "dataA3"
 },
 {
   "column1": "dataB1",
   "column2": "dataB2",
   "column3": "dataB3"
 },
 {
   "column1": "dataC1",
   "column2": "dataC2",
   "column3": "dataC3"
 }
]

Note

The REST Extractor plug-in expects a well-formed query response from the source URL. If the query response is invalid, then the REST Extractor plug-in returns an error.

Complex structures Copied

The REST Extractor plug-in supports simple structures. If JSON structures exist within an object, then the entire structure falls within one cell. For example:

{
 "name": "gateway-5-deploy",
 "creation": "2019-02-05T23:06:03Z",
 "annotations": {
  "kubernetes.io/limit-ranger":
  "LimitRanger plugin set: cpu, memory request for container deployment; cpu, memory limit for container deployment",
  "openshift.io/deployment-config.name": "gateway",
  "openshift.io/deployment.name": "gateway-5",
  "openshift.io/scc": "restricted"
 }
}

In the query response above, the annotations field includes the entire sub-structure as its value:

name creation annotations
gateway-5-deploy 2019-02-05T23:06:03Z { "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu, memory request for container deployment; cpu, memory limit for container deployment", "openshift.io/deployment-config.name": "gateway", "openshift.io/deployment.name": "gateway-5", "openshift.io/scc": "restricted" }

Note

We recommend that you write queries that return simple data types, such as strings, numbers, and Boolean for each field value.

Plugin configuration Copied

The REST Extractor plug-in requires your inputs in order to provide meaningful dataviews.

Note

You can safely update the configuration of this plug-in without causing the Netprobe to restart.

Note

If you are using this plugin with Gateway Hub, you must create a user defined data schema. For instructions, see Create a data schema.
Field Description
Name Name of the dataview.
URL

Source URL accessible to the Netprobe from which the REST Extractor plug-in expects a JSON response.

Note: The REST Extractor plug-in supports HTTP and HTTPS. Other protocols, such as FTP, are not supported and result in an error if invoked.
HTTP method

Specifies the HTTP method that the sampler uses to query the URL. By default, the GET method is used.

The POST and PUT methods enable you to add a request body to your query.

The request body is optional, and supports the following content types:

  • Application/json
  • Application/javascript
  • Application/xml
  • Application/x-www-form-urlencoded
  • text/html
  • text/plain

For the Application/x-www-form-urlencoded content type, you can input key-value pairs for the body. The value can be a string or a password-type variable (stdAES or extPwd).

For the other content types, you can specify a string for the body. Password-type variables (stdAes or extPwd) can be used within the string, which will then be decrypted before the body is sent in the HTTP request.

By default, the JSON content type is used.

Note: If you specify a different Content-Type in the Headers configuration, then that configuration overrides the content type in this option.
Columns

Allows you to define the order of the columns on the dataview, based on the fields you define.

For guidance in defining columns, see Columns.
Query

jq query used to transform the JSON response from URL into a format that can be rendered as a dataview.

An empty query string is equivalent to a query of ".". For this case, the query response includes the full JSON content.

Note: The REST Extractor plug-in requires a well-formed query operation. If the query is invalid, then the REST Extractor plug-in returns an error. No rows or columns appear in the dataview.
Mode

Reporting mode for the REST Extractor plug-in.

The REST Extractor plug-in supports the following query mode:

  • complete — for each sampling interval, the dataview refreshes to show the latest query result.
Row limit

Limits the dataview to the number of rows specified.

Default value:

200

Headers HTTP headers to include.

Provide headers in key-value pairs.

Note: If you specify an HTTP Content-Type in the key-value pairs, then this content type overrides the content type you might specify in the HTTP method for POST and PUT.
Authentication

The REST Extractor plug-in supports the following authentication types:

  • None — requires no authentication. This is the default setting.
  • Basic — requires basic authentication. If you choose this type, then you must provide a username and password.
  • Bearer — requires bearer authentication. If you choose this type, then you must provide the requisite fields. For more information, see Bearer authentication.
  • Certificate — uses the KeyStore and TrustStore files. If you choose this type, then you must provide the path and the password to the file. For more information, see Certificate authentication.

    • Note: If your source URL requires authentication, check that the correct authentication type is set and that your username and password are correct. Otherwise, the REST Extractor plug-in will show an empty dataview and return an error.
Timeout

This field is found in the Advanced tab.

Number of seconds the REST Extractor plug-in waits for a response from a source URL.

If the source URL takes longer than the timeout value, then the REST Extractor plug-in reports a timeout and displays no data in the affected dataview.

You can toggle between entering numerical value (data) or a variable (var).

Default value: 60

Note: If no timeout is specified, then this plugin follows the default value. If the platform (for example, Windows) where the sampler is running has its own timeout setting for establishing connections, then this takes precedence over the configured timeout of the plugin.

However, if a connection to the URL has already been established, the plugin will use the configured timeout when waiting for a response from the URL.

Columns Copied

The REST Extractor plug-in allows you to arrange a dataview into an order of columns based on the fields on the JSON response.

Columns

When you define columns for your sampler dataview, note the following behaviours:

Note

When you update the column order for an existing dataview, the dataview does not reflect the change immediately. To see the updated column order, click away from the Metrics dockable on the Active Console, then click the Metrics dockable again.

Empty dataviews Copied

The REST Extractor plugin creates dataviews based on column information. This can affect the resulting dataview when the plugin receives an empty JSON response. Therefore, the REST Extractor plugin responds to empty JSON responses as follows:

Empty JSON response occurs REST Extractor handling
On the first sample

Only the values specified in Columns, with no rows, are displayed.

If there are no values set, then the resulting dataview is empty.
After an empty JSON response

Columns from the previous sample, with no rows, are displayed.

If there were no values set from the previous sample, then the resulting dataview is empty.
After a non-empty JSON response Columns from the previous sample, with no rows, are displayed.
Before a non-empty JSON response

Columns from the previous sample and their corresponding values from the non-empty JSON response are displayed.

If a new field appears in the JSON response, it will not be displayed in the dataview unless defined in Columns. If you define a field in Columns, and this field does not appear in the JSON response, the missing field is displayed but with empty rows. In addition, the Netprobe logs a WARN message only in the first instance for every missing field.

Bearer authentication Copied

When you choose bearer authentication for your sampler dataview, you have the option to choose between GET, POST or PUT methods. The method depends on how bearer authentication is implemented in the system that you are monitoring.

For either method, you need to specify the following fields:

Bearer authentication

Field Description
Service url

Authentication URL or endpoint where the sampler retrieves the token.
Headers HTTP headers to include.

Provide headers in key-value pairs.

Note: If you specify an HTTP Content-Type in the key-value pairs, then this content type overrides the content type you might specify in the HTTP method for POST and PUT.
HTTP method

Specifies the HTTP method that the sampler uses to query the URL. By default, the GET method is used.

The POST and PUT methods enable you to add a request body to your query.

The request body is optional, and supports the following content types:

  • Application/json
  • Application/javascript
  • Application/xml
  • Application/x-www-form-urlencoded
  • text/html
  • text/plain

For the Application/x-www-form-urlencoded content type, you can input key-value pairs for the body. The value can be a string or a password-type variable (stdAES or extPwd).

For the other content types, you can specify a string for the body. Password-type variables (stdAes or extPwd) can be used within the string, which will then be decrypted before the body is sent in the HTTP request.

By default, the JSON content type is used.
Response token

Defines how the response token is handled:

  • fieldName — name of the field in your service URL response that carries the token value.

  • jq — jq query that will be applied to the service URL JSON data. The result will be used as the bearer authorization header.

The sampler expects the response to be in JSON. If you do not specify this field, then the sampler treats the entire response from the service URL as the token, in plaintext.

For guidance in defining the response token, see Response token examples.

Response token examples Copied

Consider an example service URL response in JSON:

{
   "token":"correct-token"
}

The following table shows examples of configured Response token options, and the resulting authorization header:

Response token option Value Authorization header
Field name token Bearer correct-token
JQ expression "Bearer " + .token Bearer correct-token
JQ expression .token | "Bearer " + . Bearer correct-token
JQ expression "vRealizeOpsToken " + .token vRealiseOpsToken correct-token

GET method Copied

You can configure bearer authentication with the GET method as an authentication option for a dataview on a REST Extractor sampler.

Field Description
Username

Username of the user authenticated by the service URL to retrieve a token.
Password

Password of the user authenticated by the service URL to retrieve a token.

Choose the appropriate field when specifying the password:

  • stdAES — use this to input your plaintext password. If you select stdAES, you can define your password directly in the sampler and store it in standard AES encryption hash in the Gateway.
  • var — use this to pass the password as a variable. The variable is defined in Managed entity > Advanced > Var. This is useful for situations where you have multiple samplers that use the same credentials.
  • extPwd — use this for passwords provided by an external provider. For more information, see

    Securing your Gateway with an external password provider in Secure Passwords.

POST method Copied

You can configure bearer authentication with the POST method as an authentication option for a dataview on a REST Extractor sampler.

REST Extractor bearer authentication POST method

Field Description
Post

The POST method enables you to add a request body to your query.

The request body is optional, and supports the following content types:

  • Application/json
  • Application/javascript
  • Application/xml
  • Application/x-www-form-urlencoded
  • text/html
  • text/plain

By default, the JSON content type is used.

Body HTTP body to include which the service URL uses to return a token. Typically, the body contains the user credentials.

For Application/x-www-form-urlencoded content type, the user can input key-value pairs for the body. The value can be a string or a password (stdAES or extPwd).

For the other content types, the user can specify a string for the body. Password-type variables (stdAes or extPwd) are allowed to be used within the string, which will then be decrypted before the body is sent in the HTTP request.

Provide the body in key-value pairs.

PUT method Copied

You can configure bearer authentication with the PUT method as an authentication option for a dataview on a REST Extractor sampler.

Field Description
Put

The PUT method enables you to add a request body to your query.

The request body is optional, and supports the following content types:

  • Application/json
  • Application/javascript
  • Application/xml
  • Application/x-www-form-urlencoded
  • text/html
  • text/plain

By default, the JSON content type is used.

Body HTTP body to include which the service URL uses to return a token. Typically, the body contains the user credentials.

For Application/x-www-form-urlencoded content type, the user can input key-value pairs for the body. The value can be a string or a password (stdAES or extPwd).

For the other content types, the user can specify a string for the body. Password-type variables (stdAes or extPwd) are allowed to be used within the string, which will then be decrypted before the body is sent in the HTTP request.

Provide the body in key-value pairs.

Certificate authentication Copied

When you choose certificate authentication for your sampler dataview, you must have a KeyStore file containing the key and certificate of the client. You also have an option to have a server authentication by providing a TrustStore file containing the trusted server certificate chain.

REST Extractor certificate authentication

Note

The supported keyStore and trustStore types are JKS and PKCS12.

Whether you are specifying the KeyStore or the TrustStore file, you need to specify the following fields:

Field Description
Path

Absolute or relative path that points to the location of the file.
Password

Password to the file.

Choose the appropriate field when specifying the password:

  • stdAES — use this to input your plaintext password. If you select stdAES, you can define your password directly in the sampler and store it in standard AES encryption hash in the Gateway.
  • var — use this to pass the password as a variable. The variable is defined in Managed entity > Advanced > Var. This is useful for situations where you have multiple samplers that use the same credentials.
  • extPwd — use this for passwords provided by an external provider. For more information, see

    Securing your Gateway with an external password provider in Secure Passwords.

Example queries, responses, and dataviews Copied

This section contains basic query examples to help you set up the dataviews for the REST Extractor sampler.

Note

jq for Jackson aims to be compatible with the original jq implementation. However, not all features are available. For in-depth information on using jq for Jackson, refer to the jq for Jackson Java JSON Processor repository on GitHub.

Basic structure — query a field Copied

Consider the example JSON data:

{
 "rows": [
    {
  "column1": "dataA1",
  "column2": "dataA2",
  "column3": "dataA3"
    },
    {
  "column1": "dataB1",
  "column2": "dataB2",
  "column3": "dataB3"
    },
    {
  "column1": "dataC1",
  "column2": "dataC2",
  "column3": "dataC3"
    }
 ]
}

Query applied Copied

The following query fetches the value within the rows field:

.rows

Query result Copied 

[
 {
   "column1": "dataA1",
   "column2": "dataA2",
   "column3": "dataA3"
 },
 {
   "column1": "dataB1",
   "column2": "dataB2",
   "column3": "dataB3"
 },
 {
   "column1": "dataC1",
   "column2": "dataC2",
   "column3": "dataC3"
 }
]

Resulting dataview Copied

column1 column2 column3
dataA1 dataA2 dataA3
dataB1 dataB2 dataB3
dataC1 dataC2 dataC3

Complex structure — query a field Copied

Consider the example JSON data:

{
 "rows": [{
  "column1": "dataA1",
  "column2": "dataA2",
  "column3": [{
   "column_name": "name1",
   "column_value": "value1",
   "column_description": "description1"
   },
   {
   "column_name": "name2",
   "column_value": "value2",
   "column_description": "description2"
   },
   {
   "column_name": "name3",
   "column_value": "value3",
   "column_description": "description3"
   }]
  },
  {
  "column1": "dataB1",
  "column2": "dataB2",
  "column3": "dataB3"
  },
  {
  "column1": "dataC1",
  "column2": "dataC2",
  "column3": "dataC3"
 }]
}

Query applied Copied

The following query fetches the value within the rows field:

.rows

Query result Copied 

[
 {
  "column1": "dataA1",
  "column2": "dataA2",
  "column3": [
   {
   "column_name": "name1",
   "column_value": "value1",
   "column_description": "description1"
   },
   {
   "column_name": "name2",
   "column_value": "value2",
   "column_description": "description2"
   },
   {
   "column_name": "name3",
   "column_value": "value3",
   "column_description": "description3"
   }
  ]
 },
 {
  "column1": "dataB1",
  "column2": "dataB2",
  "column3": "dataB3"
 },
 {
  "column1": "dataC1",
  "column2": "dataC2",
  "column3": "dataC3"
 }
]

Resulting dataview Copied

column1 column2 column3
dataA1 dataA2 [ { "column_name": "name1", "column_value": "value1", "column_description": "description1" }, { "column_name": "name2", "column_value": "value2", "column_description": "description2" }, { "column_name": "name3", "column_value": "value3", "column_description": "description3" } ]
dataB1 dataB2 dataB3
dataC1 dataC2 dataC3

The REST Extractor plug-in supports simple structures. If JSON structures exist within an object, then the entire structure falls within one cell.

Note

We recommend that you write queries that return simple data types, such as strings, numbers, and Boolean for each field value.

Complex structure — query a structure within an object Copied

Consider the example JSON data:

{
 "rows": [{
  "column1": "dataA1",
  "column2": "dataA2",
  "column3": [{
   "column_name": "name1",
   "column_value": "value1",
   "column_description": "description1"
   },
   {
   "column_name": "name2",
   "column_value": "value2",
   "column_description": "description2"
   },
   {
   "column_name": "name3",
   "column_value": "value3",
   "column_description": "description3"
   }]
  },
  {
  "column1": "dataB1",
  "column2": "dataB2",
  "column3": "dataB3"
  },
  {
  "column1": "dataC1",
  "column2": "dataC2",
  "column3": "dataC3"
 }]
}

Query applied Copied

The following query fetches the first object (with index 0) and returns value inside the column3 field:

.rows[0].column3

Query result Copied 

[
 {
 "column_name": "name1",
 "column_value": "value1",
 "column_description": "description1"
 },
 {
 "column_name": "name2",
 "column_value": "value2",
 "column_description": "description2"
 },
 {
 "column_name": "name3",
 "column_value": "value3",
 "column_description": "description3"
 }
]

Resulting dataview Copied

column_name column_value column_description
name1 value1 description1
name2 value2 description2
name3 value3 description3

Complex structure — query specific fields Copied

Consider the example JSON data:

[
 {
  "sha": "9fa2e51099c55af56e3e541dc4b399f11de74abe",
  "node_id": "MDY6Q29tbWl0NTEwMTE0MTo5ZmEyZTUxMDk5YzU1YWY1NmUzZTU0MWRjNGIzOTlmMTFkZTc0YWJl",
  "commit": {
   "committer": {
    "name": "GitHub",
    "email": "noreply@github.com",
    "date": "2019-02-09T22:32:20Z"
    },
   "message": "Merge pull request #1818 from muhmuhten/free_in_index",
   "tree": {
    "sha": "0fc0322e726f9a6eb8637defbff27b28c316cce0",
    "url": "https://api.github.com/repos/stedolan/jq/git/trees/0fc0322e726f9a6eb8637defbff27b28c316cce0"
    },
   "url": "https://api.github.com/repos/stedolan/jq/git/commits/9fa2e51099c55af56e3e541dc4b399f11de74abe",
   "comment_count": 0
  },
  "url": "https://api.github.com/repos/stedolan/jq/commits/9fa2e51099c55af56e3e541dc4b399f11de74abe",
  "html_url": "https://github.com/stedolan/jq/commit/9fa2e51099c55af56e3e541dc4b399f11de74abe",
  "comments_url": "https://api.github.com/repos/stedolan/jq/commits/9fa2e51099c55af56e3e541dc4b399f11de74abe/comments",
  "committer": {
   "login": "web-flow",
   "id": 19864447,
   "node_id": "MDQ6VXNlcjE5ODY0NDQ3"
  },
  "parents": [
   {
    "sha": "88613f546926a7737af7542bf0a74a22d935cd14",
    "url": "https://api.github.com/repos/stedolan/jq/commits/88613f546926a7737af7542bf0a74a22d935cd14",
    "html_url": "https://github.com/stedolan/jq/commit/88613f546926a7737af7542bf0a74a22d935cd14"
   },
   {
    "sha": "3f59bee3e080c4b1ee2ae94244ac0263a69999d2",
    "url": "https://api.github.com/repos/stedolan/jq/commits/3f59bee3e080c4b1ee2ae94244ac0263a69999d2",
    "html_url": "https://github.com/stedolan/jq/commit/3f59bee3e080c4b1ee2ae94244ac0263a69999d2"
   }
  ]
 },
 {
  "sha": "3f59bee3e080c4b1ee2ae94244ac0263a69999d2",
  "node_id": "MDY6Q29tbWl0NTEwMTE0MTozZjU5YmVlM2UwODBjNGIxZWUyYWU5NDI0NGFjMDI2M2E2OTk5OWQy",
  "commit": {
   "committer": {
    "name": "Muh Muhten",
    "email": "muh.muhten@gmail.com",
    "date": "2019-02-09T22:17:59Z"
    },
   "message": "Make sure to free in jv_string_indexes",
   "tree": {
    "sha": "0fc0322e726f9a6eb8637defbff27b28c316cce0",
    "url": "https://api.github.com/repos/stedolan/jq/git/trees/0fc0322e726f9a6eb8637defbff27b28c316cce0"
   },
   "url": "https://api.github.com/repos/stedolan/jq/git/commits/3f59bee3e080c4b1ee2ae94244ac0263a69999d2",
   "comment_count": 0
  },
  "url": "https://api.github.com/repos/stedolan/jq/commits/3f59bee3e080c4b1ee2ae94244ac0263a69999d2",
  "html_url": "https://github.com/stedolan/jq/commit/3f59bee3e080c4b1ee2ae94244ac0263a69999d2",
  "comments_url": "https://api.github.com/repos/stedolan/jq/commits/3f59bee3e080c4b1ee2ae94244ac0263a69999d2/comments",
  "committer": {
   "login": "muhmuhten",
   "id": 9712278,
   "node_id": "MDQ6VXNlcjk3MTIyNzg="
  },
  "parents": [
   {
    "sha": "88613f546926a7737af7542bf0a74a22d935cd14",
    "url": "https://api.github.com/repos/stedolan/jq/commits/88613f546926a7737af7542bf0a74a22d935cd14",
    "html_url": "https://github.com/stedolan/jq/commit/88613f546926a7737af7542bf0a74a22d935cd14"
   }
  ]
 },
 {
  "sha": "88613f546926a7737af7542bf0a74a22d935cd14",
  "node_id": "MDY6Q29tbWl0NTEwMTE0MTo4ODYxM2Y1NDY5MjZhNzczN2FmNzU0MmJmMGE3NGEyMmQ5MzVjZDE0",
  "commit": {
   "committer": {
    "name": "Nico Williams",
    "email": "nico@cryptonector.com",
    "date": "2019-02-09T04:41:31Z"
   },
   "message": "Improve linking time by marking subtrees with unbound symbols",
   "tree": {
    "sha": "f3076767b391b45c83f665231188b689786de463",
    "url": "https://api.github.com/repos/stedolan/jq/git/trees/f3076767b391b45c83f665231188b689786de463"
   },
   "url": "https://api.github.com/repos/stedolan/jq/git/commits/88613f546926a7737af7542bf0a74a22d935cd14",
   "comment_count": 0
  },
  "url": "https://api.github.com/repos/stedolan/jq/commits/88613f546926a7737af7542bf0a74a22d935cd14",
  "html_url": "https://github.com/stedolan/jq/commit/88613f546926a7737af7542bf0a74a22d935cd14",
  "comments_url": "https://api.github.com/repos/stedolan/jq/commits/88613f546926a7737af7542bf0a74a22d935cd14/comments",
  "committer": {
   "login": "nicowilliams",
   "id": 604851,
   "node_id": "MDQ6VXNlcjYwNDg1MQ=="
  },
  "parents": [
   {
    "sha": "2660b04a731568c54eb4b91fe811d81cbbf3470b",
    "url": "https://api.github.com/repos/stedolan/jq/commits/2660b04a731568c54eb4b91fe811d81cbbf3470b",
    "html_url": "https://github.com/stedolan/jq/commit/2660b04a731568c54eb4b91fe811d81cbbf3470b"
   }
  ]
 },
 {
  "sha": "2660b04a731568c54eb4b91fe811d81cbbf3470b",
  "node_id": "MDY6Q29tbWl0NTEwMTE0MToyNjYwYjA0YTczMTU2OGM1NGViNGI5MWZlODExZDgxY2JiZjM0NzBi",
  "commit": {
   "committer": {
    "name": "Nicolas Williams",
    "email": "nico@cryptonector.com",
    "date": "2019-02-09T02:40:55Z"
   },
   "message": "Fix infinite loop in index(\"\") (fix #1815)",
   "tree": {
    "sha": "a3f99840574d3bf3c1fc2b4947d73215f25c6c68",
    "url": "https://api.github.com/repos/stedolan/jq/git/trees/a3f99840574d3bf3c1fc2b4947d73215f25c6c68"
   },
   "url": "https://api.github.com/repos/stedolan/jq/git/commits/2660b04a731568c54eb4b91fe811d81cbbf3470b",
   "comment_count": 0
  },
  "url": "https://api.github.com/repos/stedolan/jq/commits/2660b04a731568c54eb4b91fe811d81cbbf3470b",
  "html_url": "https://github.com/stedolan/jq/commit/2660b04a731568c54eb4b91fe811d81cbbf3470b",
  "comments_url": "https://api.github.com/repos/stedolan/jq/commits/2660b04a731568c54eb4b91fe811d81cbbf3470b/comments",
  "committer": {
    "login": "nicowilliams",
    "id": 604851,
    "node_id": "MDQ6VXNlcjYwNDg1MQ=="
  },
  "parents": [
   {
    "sha": "18ac789d6d74f198053c547fba4e8c70cb99d55a",
    "url": "https://api.github.com/repos/stedolan/jq/commits/18ac789d6d74f198053c547fba4e8c70cb99d55a",
    "html_url": "https://github.com/stedolan/jq/commit/18ac789d6d74f198053c547fba4e8c70cb99d55a"
   }
  ]
 },
 {
  "sha": "18ac789d6d74f198053c547fba4e8c70cb99d55a",
  "node_id": "MDY6Q29tbWl0NTEwMTE0MToxOGFjNzg5ZDZkNzRmMTk4MDUzYzU0N2ZiYTRlOGM3MGNiOTlkNTVh",
  "commit": {
   "committer": {
    "name": "Nico Williams",
    "email": "nico@cryptonector.com",
    "date": "2019-02-07T17:08:44Z"
   },
   "message": "FIX if-then-else example\n\n\"(.+)\" doesn't match new line characters so the link generated for [if-then-else](https://github.com/stedolan/jq/blob/90bc29c1b544c0436ec44246e180fdbb6d6384df/docs/content/3.manual/v1.5/manual.yml#L1804) doesn't work",
   "tree": {
    "sha": "8fb2e369d0606b8773f5ae5040c5963c87c09da7",
    "url": "https://api.github.com/repos/stedolan/jq/git/trees/8fb2e369d0606b8773f5ae5040c5963c87c09da7"
   },
   "url": "https://api.github.com/repos/stedolan/jq/git/commits/18ac789d6d74f198053c547fba4e8c70cb99d55a",
    "comment_count": 0
  },
  "url": "https://api.github.com/repos/stedolan/jq/commits/18ac789d6d74f198053c547fba4e8c70cb99d55a",
  "html_url": "https://github.com/stedolan/jq/commit/18ac789d6d74f198053c547fba4e8c70cb99d55a",
  "comments_url": "https://api.github.com/repos/stedolan/jq/commits/18ac789d6d74f198053c547fba4e8c70cb99d55a/comments",
  "committer": {
   "login": "nicowilliams",
   "id": 604851,
   "node_id": "MDQ6VXNlcjYwNDg1MQ=="
  },
  "parents": [
   {
    "sha": "956d40d9dd7cc69227af817009ad47373e709e5f",
    "url": "https://api.github.com/repos/stedolan/jq/commits/956d40d9dd7cc69227af817009ad47373e709e5f",
    "html_url": "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e709e5f"
   },
   {
    "sha": "956d40d9dd7cc69227af817009ad47373e70xxxx",
    "url": "https://api.github.com/repos/stedolan/jq/commits/956d40d9dd7cc69227af817009ad47373e709e5f",
    "html_url": "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e7xxxxx"
   }
  ]
 }
]

Query applied Copied

The following query fetches certain fields of each element in the structure, defines the columns used, then further filters the response with a value under the name field:

[.[] | {message: .commit.message, name: .commit.committer.name, parents: [.parents[].html_url]} | select(.name == "Nico Williams") ]

Query result Copied 

[
 {
 "message": "Improve linking time by marking subtrees with unbound symbols",
 "name": "Nico Williams",
 "parents": [
  "https://github.com/stedolan/jq/commit/2660b04a731568c54eb4b91fe811d81cbbf3470b"
 ]
 },
 {
 "message": "FIX if-then-else example\n\n\"(.+)\" doesn't match new line characters so the link generated for [if-then-else](https://github.com/stedolan/jq/blob/90bc29c1b544c0436ec44246e180fdbb6d6384df/docs/content/3.manual/v1.5/manual.yml#L1804) doesn't work",
 "name": "Nico Williams",
 "parents": [
  "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e709e5f",
  "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e7xxxxx"
 ]
 }
]

Resulting dataview Copied

message name parents
Improve linking time by marking subtrees with unbound symbols Nico Williams [ "https://github.com/stedolan/jq/commit/2660b04a731568c54eb4b91fe811d81cbbf3470b" ]
FIX if-then-else example\n\n\"(.+)\" doesn't match new line characters so the link generated for if-then-else doesn't work Nico Williams [ "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e709e5f", "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e7xxxxx" ]

Debug options Copied

The REST Extractor plug-in has a few debug options on the probe. When set, these options provide additional information in the Netprobe log:

Module Setting Log contents
REST_EXTRACTOR DATAVIEW_CONFIG
  • Sampler name
  • Dataview name
  • URL
  • Configured headers
  • Method
  • Content type
  • Request body
  • Query
  • Timeout
  • Row limit
REST_EXTRACTOR DATA
  • Sampler name
  • Dataview name
  • Raw data
  • Transformed data
REST_EXTRACTOR_AUTHENTICATION None

For all authentication types:

  • Sampler name
  • Dataview name
  • Authentication type

For basic authentication:

  • Username

For bearer authentication, GET method:

  • Bearer method
  • Username used for retrieving the token
  • Service URL
  • Service URL response body
  • Token field
  • Token value

For bearer authentication, POST method:

  • Bearer method
  • Service URL
  • Service URL response body
  • Token field
  • Token

For bearer authentication, PUT method:

  • Bearer method
  • Service URL
  • Service URL response body
  • Token field
  • Token
["Geneos"] ["Geneos > Netprobe"] ["User Guide"]

Was this topic helpful?