Geneos
Geneos
Recommended Reads

REST Extractor Plug-in Technical Reference

Overview

The REST Extractor plug-in enables you to parse and extract fields from JSON content and turn these into dataviews in Geneos.

The plug-in supports JSON responses coming from a web server or URL.

Dataviews are configured through the use of jq queries. The REST Extractor plug-in uses the jq for Jackson library. For more information, see Application and plug-in specific information in 5.x Compatibility Matrix.

Dataviews

The REST Extractor plug-in works by accepting queries. Each query creates a new dataview on the sampler.

For more information on creating queries, see Example queries, responses, and dataviews.

Response format

The REST Extractor plug-in expects the following JSON format for query responses: 

[
	{
	  "column1": "dataA1",
	  "column2": "dataA2",
	  "column3": "dataA3"
	},
	{
	  "column1": "dataB1",
	  "column2": "dataB2",
	  "column3": "dataB3"
	},
	{
	  "column1": "dataC1",
	  "column2": "dataC2",
	  "column3": "dataC3"
	}
]
  • Each object represents a new row on the dataview.
  • Each field represents a column.
  • If a field is missing in one of the objects, then its value is "null" on the dataview.

Note: The REST Extractor plug-in expects a well-formed query response from the source URL. If the query response is invalid, then the REST Extractor plug-in returns an error.

Complex structures

The REST Extractor plug-in supports simple structures. If JSON structures exist within an object, then the entire structure falls within one cell. For example:

{
	"name": "gateway-5-deploy",
	"creation": "2019-02-05T23:06:03Z",
	"annotations": {
		"kubernetes.io/limit-ranger":
		"LimitRanger plugin set: cpu, memory request for container deployment; cpu, memory limit for container deployment",
		"openshift.io/deployment-config.name": "gateway",
		"openshift.io/deployment.name": "gateway-5",
		"openshift.io/scc": "restricted"
	}
}

In the query response above, the annotations field includes the entire sub-structure as its value:

name creation annotations
gateway-5-deploy 2019-02-05T23:06:03Z {
"kubernetes.io/limit-ranger":
"LimitRanger plugin set: cpu, memory request for container deployment; cpu, memory limit for container deployment",
"openshift.io/deployment-config.name": "gateway",
"openshift.io/deployment.name": "gateway-5",
"openshift.io/scc": "restricted"
}

Note: We recommend that you write queries that return simple data types, such as strings, numbers, and Boolean for each field value.

Plug-in configuration

Caution: When you update the configuration of certain Java-based plug-ins, including this one, the Netprobe running it has to restart. Be aware of other potentially unrelated plug-ins that can be affected when you configure this particular plug-in.

The REST Extractor plug-in requires your inputs in order to provide meaningful dataviews. If you are setting up the REST Extractor plug-in for the first time, see the REST Extractor Plug-in User Guide.

Field Description
Name Name of the dataview.
URL

Source URL accessible to the Netprobe from which the REST Extractor plug-in expects a JSON response.

Note: The REST Extractor plug-in supports HTTP and HTTPS. Other protocols, such as FTP, are not supported and result in an error if invoked.
Columns

Allows you to define the order of the columns on the dataview, based on the fields you define.

For guidance in defining columns, see Column order.
Query

jq query used to transform the JSON response from URL into a format that can be rendered as a dataview.

An empty query string is equivalent to a query of ".". For this case, the query response includes the full JSON content.

Note: The REST Extractor plug-in requires a well-formed query operation. If the query is invalid, then the REST Extractor plug-in returns an error. No rows or columns appear in the dataview.
Mode

Reporting mode for the REST Extractor plug-in.

The REST Extractor plug-in supports the following query mode:

  • complete — for each sampling interval, the dataview refreshes to show the latest query result.
Row limit

Limits the dataview to the number of rows specified.

Default value:

200

Headers

HTTP headers to include.

Provide headers in key-value pairs.
Authentication

The REST Extractor plug-in supports the following authentication types:

  • None — requires no authentication. This is the default setting.
  • Basic — requires basic authentication. If you choose this type, then you must provide a username and password.
  • Bearer — requires bearer authentication. If you choose this type, then you must provide the requisite fields. For more information, see Bearer authentication.

    • Note: If your source URL requires authentication, check that the correct authentication type is set and that your username and password are correct. Otherwise, the REST Extractor plug-in will show an empty dataview and return an error.
Timeout

This field is found in the Advanced tab.

Number of seconds the REST Extractor plug-in waits for a response from a source URL.

If the source URL takes longer than the timeout value, then the REST Extractor plug-in reports a timeout and displays no data in the affected dataview.

You can toggle between entering numerical value (data) or a variable (var).

Default value: 60

Note: If no timeout is specified, the REST Extractor plug-in follows the default value. However, if the platform (for example, Windows) where the sampler is running has its own timeout setting, then this takes precedence over the plug-in default.

Column order

The REST Extractor plug-in allows you to arrange a dataview into an order of columns based on the fields on the JSON response.

REST Extractor plug-in columns feature

When you define columns for your sampler dataview, note the following behaviours:

  • The fields appear on the dataview in the order that you define. That is, the first field appears as the first column, the second field appears as the second column, and so on.
  • The first column you define becomes the row key. We recommend that your first column be a field that has a unique value per JSON object. Otherwise, the Gateway Setup Editor marks recurring values as duplicates.
  • You do not need to define all possible fields when using the columns feature. However, only those fields that you define are guaranteed to follow an order.
  • If you define a field in the column window, and this field does not appear in the JSON response, then the JSON response displays normally on the dataview, but the column for the missing field is not displayed. In addition, the Netprobe returns a warning message in the log.

Note: When you update the column order for an existing dataview, the dataview does not reflect the change immediately. To see the updated column order, click away from the Metrics dockable on the Active Console, then click the Metrics dockable again.

Bearer authentication

When you choose bearer authentication for your sampler dataview, you have the option to choose between GET or POST methods. The method depends on how bearer authentication is implemented in the system that you are monitoring.

For either method, you need to specify the following fields:

Field Description
Service url

Authentication URL or endpoint where the sampler retrieves the token.
Response token field

Name of the field in your service URL response that carries the token value.

This field is optional:

  • If you specify this field, then the sampler expects the response to be in JSON.
  • if you do not specify this field, then the sampler treats the entire response from the service URL as the token, in plaintext.

GET method

You can configure bearer authentication with the GET method as an authentication option for a dataview on a REST Extractor sampler.

Bearer authentication — GET method

Field Description
Username

Username of the user authenticated by the service URL to retrieve a token.
Password

Password of the user authenticated by the service URL to retrieve a token.

Choose the appropriate field when specifying the password:

  • stdAES — use this to input your plaintext password. If you select stdAES, you can define your password directly in the sampler and store it in standard AES encryption hash in the Gateway.
  • var — use this to pass the password as a variable. The variable is defined in Managed entity > Advanced > Var. This is useful for situations where you have multiple samplers that use the same credentials.
  • extPwd — use this for passwords provided by an external provider. For more information, see Secure Passwords in Secure Passwords.

POST method

You can configure bearer authentication with the POST method as an authentication option for a dataview on a REST Extractor sampler.

Bearer authentication — POST method

Field Description
Body

HTTP body to include which the service URL uses to return a token. Typically, the body contains the user credentials.

Provide the body in key-value pairs.

Example queries, responses, and dataviews

This section contains basic query examples to help you set up the dataviews for the REST Extractor sampler.

Note: jq for Jackson aims to be compatible with the original jq implementation. However, not all features are available. For in-depth information on using jq for Jackson, refer to the jq for Jackson Java JSON Processor repository on GitHub.

Basic structure — query a field

Consider the example JSON data:

{
	"rows": [
	   {
		"column1": "dataA1",
		"column2": "dataA2",
		"column3": "dataA3"
	   },
	   {
		"column1": "dataB1",
		"column2": "dataB2",
		"column3": "dataB3"
	   },
	   {
		"column1": "dataC1",
		"column2": "dataC2",
		"column3": "dataC3"
	   }
	]
}

Query applied

The following query fetches the value within the rows field:

.rows

Query result

[
	{
	  "column1": "dataA1",
	  "column2": "dataA2",
	  "column3": "dataA3"
	},
	{
	  "column1": "dataB1",
	  "column2": "dataB2",
	  "column3": "dataB3"
	},
	{
	  "column1": "dataC1",
	  "column2": "dataC2",
	  "column3": "dataC3"
	}
]

Resulting dataview

column1 column2 column3
dataA1 dataA2 dataA3
dataB1 dataB2 dataB3
dataC1 dataC2 dataC3

Complex structure — query a field

Consider the example JSON data:

{
	"rows": [{
		"column1": "dataA1",
		"column2": "dataA2",
		"column3": [{
			"column_name": "name1",
			"column_value": "value1",
			"column_description": "description1"
			},
			{
			"column_name": "name2",
			"column_value": "value2",
			"column_description": "description2"
			},
			{
			"column_name": "name3",
			"column_value": "value3",
			"column_description": "description3"
			}]
		},
		{
		"column1": "dataB1",
		"column2": "dataB2",
		"column3": "dataB3"
		},
		{
		"column1": "dataC1",
		"column2": "dataC2",
		"column3": "dataC3"
	}]
}

Query applied

The following query fetches the value within the rows field:

.rows

Query result

[
	{
		"column1": "dataA1",
		"column2": "dataA2",
		"column3": [
			{
			"column_name": "name1",
			"column_value": "value1",
			"column_description": "description1"
			},
			{
			"column_name": "name2",
			"column_value": "value2",
			"column_description": "description2"
			},
			{
			"column_name": "name3",
			"column_value": "value3",
			"column_description": "description3"
			}
		]
	},
	{
		"column1": "dataB1",
		"column2": "dataB2",
		"column3": "dataB3"
	},
	{
		"column1": "dataC1",
		"column2": "dataC2",
		"column3": "dataC3"
	}
]

Resulting dataview

column1 column2 column3
dataA1 dataA2 [
{
"column_name": "name1",
"column_value": "value1",
"column_description": "description1"
},
{
"column_name": "name2",
"column_value": "value2",
"column_description": "description2"
},
{
"column_name": "name3",
"column_value": "value3",
"column_description": "description3"
}
]
dataB1 dataB2 dataB3
dataC1 dataC2 dataC3

The REST Extractor plug-in supports simple structures. If JSON structures exist within an object, then the entire structure falls within one cell.

Note: We recommend that you write queries that return simple data types, such as strings, numbers, and Boolean for each field value.

Complex structure — query a structure within an object

Consider the example JSON data:

{
	"rows": [{
		"column1": "dataA1",
		"column2": "dataA2",
		"column3": [{
			"column_name": "name1",
			"column_value": "value1",
			"column_description": "description1"
			},
			{
			"column_name": "name2",
			"column_value": "value2",
			"column_description": "description2"
			},
			{
			"column_name": "name3",
			"column_value": "value3",
			"column_description": "description3"
			}]
		},
		{
		"column1": "dataB1",
		"column2": "dataB2",
		"column3": "dataB3"
		},
		{
		"column1": "dataC1",
		"column2": "dataC2",
		"column3": "dataC3"
	}]
}

Query applied

The following query fetches the first object (with index 0) and returns value inside the column3 field:

.rows[0].column3

Query result

[
	{
	"column_name": "name1",
	"column_value": "value1",
	"column_description": "description1"
	},
	{
	"column_name": "name2",
	"column_value": "value2",
	"column_description": "description2"
	},
	{
	"column_name": "name3",
	"column_value": "value3",
	"column_description": "description3"
	}
]

Resulting dataview

column_name column_value column_description
name1 value1 description1
name2 value2 description2
name3 value3 description3

Complex structure — query specific fields

Consider the example JSON data:

[
	{
		"sha": "9fa2e51099c55af56e3e541dc4b399f11de74abe",
		"node_id": "MDY6Q29tbWl0NTEwMTE0MTo5ZmEyZTUxMDk5YzU1YWY1NmUzZTU0MWRjNGIzOTlmMTFkZTc0YWJl",
		"commit": {
			"committer": {
				"name": "GitHub",
				"email": "noreply@github.com",
				"date": "2019-02-09T22:32:20Z"
				},
			"message": "Merge pull request #1818 from muhmuhten/free_in_index",
			"tree": {
				"sha": "0fc0322e726f9a6eb8637defbff27b28c316cce0",
				"url": "https://api.github.com/repos/stedolan/jq/git/trees/0fc0322e726f9a6eb8637defbff27b28c316cce0"
				},
			"url": "https://api.github.com/repos/stedolan/jq/git/commits/9fa2e51099c55af56e3e541dc4b399f11de74abe",
			"comment_count": 0
		},
		"url": "https://api.github.com/repos/stedolan/jq/commits/9fa2e51099c55af56e3e541dc4b399f11de74abe",
		"html_url": "https://github.com/stedolan/jq/commit/9fa2e51099c55af56e3e541dc4b399f11de74abe",
		"comments_url": "https://api.github.com/repos/stedolan/jq/commits/9fa2e51099c55af56e3e541dc4b399f11de74abe/comments",
		"committer": {
			"login": "web-flow",
			"id": 19864447,
			"node_id": "MDQ6VXNlcjE5ODY0NDQ3"
		},
		"parents": [
			{
				"sha": "88613f546926a7737af7542bf0a74a22d935cd14",
				"url": "https://api.github.com/repos/stedolan/jq/commits/88613f546926a7737af7542bf0a74a22d935cd14",
				"html_url": "https://github.com/stedolan/jq/commit/88613f546926a7737af7542bf0a74a22d935cd14"
			},
			{
				"sha": "3f59bee3e080c4b1ee2ae94244ac0263a69999d2",
				"url": "https://api.github.com/repos/stedolan/jq/commits/3f59bee3e080c4b1ee2ae94244ac0263a69999d2",
				"html_url": "https://github.com/stedolan/jq/commit/3f59bee3e080c4b1ee2ae94244ac0263a69999d2"
			}
		]
	},
	{
		"sha": "3f59bee3e080c4b1ee2ae94244ac0263a69999d2",
		"node_id": "MDY6Q29tbWl0NTEwMTE0MTozZjU5YmVlM2UwODBjNGIxZWUyYWU5NDI0NGFjMDI2M2E2OTk5OWQy",
		"commit": {
			"committer": {
				"name": "Muh Muhten",
				"email": "muh.muhten@gmail.com",
				"date": "2019-02-09T22:17:59Z"
				},
			"message": "Make sure to free in jv_string_indexes",
			"tree": {
				"sha": "0fc0322e726f9a6eb8637defbff27b28c316cce0",
				"url": "https://api.github.com/repos/stedolan/jq/git/trees/0fc0322e726f9a6eb8637defbff27b28c316cce0"
			},
			"url": "https://api.github.com/repos/stedolan/jq/git/commits/3f59bee3e080c4b1ee2ae94244ac0263a69999d2",
			"comment_count": 0
		},
		"url": "https://api.github.com/repos/stedolan/jq/commits/3f59bee3e080c4b1ee2ae94244ac0263a69999d2",
		"html_url": "https://github.com/stedolan/jq/commit/3f59bee3e080c4b1ee2ae94244ac0263a69999d2",
		"comments_url": "https://api.github.com/repos/stedolan/jq/commits/3f59bee3e080c4b1ee2ae94244ac0263a69999d2/comments",
		"committer": {
			"login": "muhmuhten",
			"id": 9712278,
			"node_id": "MDQ6VXNlcjk3MTIyNzg="
		},
		"parents": [
			{
				"sha": "88613f546926a7737af7542bf0a74a22d935cd14",
				"url": "https://api.github.com/repos/stedolan/jq/commits/88613f546926a7737af7542bf0a74a22d935cd14",
				"html_url": "https://github.com/stedolan/jq/commit/88613f546926a7737af7542bf0a74a22d935cd14"
			}
		]
	},
	{
		"sha": "88613f546926a7737af7542bf0a74a22d935cd14",
		"node_id": "MDY6Q29tbWl0NTEwMTE0MTo4ODYxM2Y1NDY5MjZhNzczN2FmNzU0MmJmMGE3NGEyMmQ5MzVjZDE0",
		"commit": {
			"committer": {
				"name": "Nico Williams",
				"email": "nico@cryptonector.com",
				"date": "2019-02-09T04:41:31Z"
			},
			"message": "Improve linking time by marking subtrees with unbound symbols",
			"tree": {
				"sha": "f3076767b391b45c83f665231188b689786de463",
				"url": "https://api.github.com/repos/stedolan/jq/git/trees/f3076767b391b45c83f665231188b689786de463"
			},
			"url": "https://api.github.com/repos/stedolan/jq/git/commits/88613f546926a7737af7542bf0a74a22d935cd14",
			"comment_count": 0
		},
		"url": "https://api.github.com/repos/stedolan/jq/commits/88613f546926a7737af7542bf0a74a22d935cd14",
		"html_url": "https://github.com/stedolan/jq/commit/88613f546926a7737af7542bf0a74a22d935cd14",
		"comments_url": "https://api.github.com/repos/stedolan/jq/commits/88613f546926a7737af7542bf0a74a22d935cd14/comments",
		"committer": {
			"login": "nicowilliams",
			"id": 604851,
			"node_id": "MDQ6VXNlcjYwNDg1MQ=="
		},
		"parents": [
			{
				"sha": "2660b04a731568c54eb4b91fe811d81cbbf3470b",
				"url": "https://api.github.com/repos/stedolan/jq/commits/2660b04a731568c54eb4b91fe811d81cbbf3470b",
				"html_url": "https://github.com/stedolan/jq/commit/2660b04a731568c54eb4b91fe811d81cbbf3470b"
			}
		]
	},
	{
		"sha": "2660b04a731568c54eb4b91fe811d81cbbf3470b",
		"node_id": "MDY6Q29tbWl0NTEwMTE0MToyNjYwYjA0YTczMTU2OGM1NGViNGI5MWZlODExZDgxY2JiZjM0NzBi",
		"commit": {
			"committer": {
				"name": "Nicolas Williams",
				"email": "nico@cryptonector.com",
				"date": "2019-02-09T02:40:55Z"
			},
			"message": "Fix infinite loop in index(\"\") (fix #1815)",
			"tree": {
				"sha": "a3f99840574d3bf3c1fc2b4947d73215f25c6c68",
				"url": "https://api.github.com/repos/stedolan/jq/git/trees/a3f99840574d3bf3c1fc2b4947d73215f25c6c68"
			},
			"url": "https://api.github.com/repos/stedolan/jq/git/commits/2660b04a731568c54eb4b91fe811d81cbbf3470b",
			"comment_count": 0
		},
		"url": "https://api.github.com/repos/stedolan/jq/commits/2660b04a731568c54eb4b91fe811d81cbbf3470b",
		"html_url": "https://github.com/stedolan/jq/commit/2660b04a731568c54eb4b91fe811d81cbbf3470b",
		"comments_url": "https://api.github.com/repos/stedolan/jq/commits/2660b04a731568c54eb4b91fe811d81cbbf3470b/comments",
		"committer": {
				"login": "nicowilliams",
				"id": 604851,
				"node_id": "MDQ6VXNlcjYwNDg1MQ=="
		},
		"parents": [
			{
				"sha": "18ac789d6d74f198053c547fba4e8c70cb99d55a",
				"url": "https://api.github.com/repos/stedolan/jq/commits/18ac789d6d74f198053c547fba4e8c70cb99d55a",
				"html_url": "https://github.com/stedolan/jq/commit/18ac789d6d74f198053c547fba4e8c70cb99d55a"
			}
		]
	},
	{
		"sha": "18ac789d6d74f198053c547fba4e8c70cb99d55a",
		"node_id": "MDY6Q29tbWl0NTEwMTE0MToxOGFjNzg5ZDZkNzRmMTk4MDUzYzU0N2ZiYTRlOGM3MGNiOTlkNTVh",
		"commit": {
			"committer": {
				"name": "Nico Williams",
				"email": "nico@cryptonector.com",
				"date": "2019-02-07T17:08:44Z"
			},
			"message": "FIX if-then-else example\n\n\"(.+)\" doesn't match new line characters so the link generated for [if-then-else](https://github.com/stedolan/jq/blob/90bc29c1b544c0436ec44246e180fdbb6d6384df/docs/content/3.manual/v1.5/manual.yml#L1804) doesn't work",
			"tree": {
				"sha": "8fb2e369d0606b8773f5ae5040c5963c87c09da7",
				"url": "https://api.github.com/repos/stedolan/jq/git/trees/8fb2e369d0606b8773f5ae5040c5963c87c09da7"
			},
			"url": "https://api.github.com/repos/stedolan/jq/git/commits/18ac789d6d74f198053c547fba4e8c70cb99d55a",
				"comment_count": 0
		},
		"url": "https://api.github.com/repos/stedolan/jq/commits/18ac789d6d74f198053c547fba4e8c70cb99d55a",
		"html_url": "https://github.com/stedolan/jq/commit/18ac789d6d74f198053c547fba4e8c70cb99d55a",
		"comments_url": "https://api.github.com/repos/stedolan/jq/commits/18ac789d6d74f198053c547fba4e8c70cb99d55a/comments",
		"committer": {
			"login": "nicowilliams",
			"id": 604851,
			"node_id": "MDQ6VXNlcjYwNDg1MQ=="
		},
		"parents": [
			{
				"sha": "956d40d9dd7cc69227af817009ad47373e709e5f",
				"url": "https://api.github.com/repos/stedolan/jq/commits/956d40d9dd7cc69227af817009ad47373e709e5f",
				"html_url": "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e709e5f"
			},
			{
				"sha": "956d40d9dd7cc69227af817009ad47373e70xxxx",
				"url": "https://api.github.com/repos/stedolan/jq/commits/956d40d9dd7cc69227af817009ad47373e709e5f",
				"html_url": "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e7xxxxx"
			}
		]
	}
]

Query applied

The following query fetches certain fields of each element in the structure, defines the columns used, then further filters the response with a value under the name field:

[.[] | {message: .commit.message, name: .commit.committer.name, parents: [.parents[].html_url]} | select(.name == "Nico Williams") ]

Query result

[
	{
	"message": "Improve linking time by marking subtrees with unbound symbols",
	"name": "Nico Williams",
	"parents": [
		"https://github.com/stedolan/jq/commit/2660b04a731568c54eb4b91fe811d81cbbf3470b"
	]
	},
	{
	"message": "FIX if-then-else example\n\n\"(.+)\" doesn't match new line characters so the link generated for [if-then-else](https://github.com/stedolan/jq/blob/90bc29c1b544c0436ec44246e180fdbb6d6384df/docs/content/3.manual/v1.5/manual.yml#L1804) doesn't work",
	"name": "Nico Williams",
	"parents": [
		"https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e709e5f",
		"https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e7xxxxx"
	]
	}
]

Resulting dataview

message name parents
Improve linking time by marking subtrees with unbound symbols Nico Williams [ "https://github.com/stedolan/jq/commit/2660b04a731568c54eb4b91fe811d81cbbf3470b" ]
FIX if-then-else example\n\n\"(.+)\" doesn't match new line characters so the link generated for [if-then-else](https://github.com/stedolan/jq/blob/90bc29c1b544c0436ec44246e180fdbb6d6384df/docs/content/3.manual/v1.5/manual.yml#L1804) doesn't work Nico Williams [ "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e709e5f", "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e7xxxxx" ]

Debug options

The REST Extractor plug-in has a few debug options on the probe. When set, these options provide additional information in the Netprobe log:

Module Setting Log contents
REST_EXTRACTOR DATAVIEW_CONFIG
  • Sampler name
  • Dataview name
  • URL
  • Configured headers
  • Method
  • Content type
  • Request body
  • Query
  • Timeout
  • Row limit
REST_EXTRACTOR DATA
  • Sampler name
  • Dataview name
  • Raw data
  • Transformed data
REST_EXTRACTOR_AUTHENTICATION None

For all authentication types:

  • Sampler name
  • Dataview name
  • Authentication type

For basic authentication:

  • Username

For bearer authentication, GET method:

  • Bearer method
  • Username used for retrieving the token
  • Service URL
  • Service URL response body
  • Token field
  • Token value

For bearer authentication, POST method:

  • Bearer method
  • Service URL
  • Service URL response body
  • Token field
  • Token

Further reading

If you are interested in setting up the REST Extractor plug-in for your organisation, see REST Extractor Plug-in User Guide.