How to parse JSON output from a website using Webmon and Extractor plugins
The JSON parsing feature has been available in Netprobe version GA4.4 and above. This article will discuss how to use Webmon and Extractor plugins to monitor JSON format contents on a website.
In this example, we will be monitoring and extracting information from the following website: https://www.treasurydirect.gov/webapis/webapisecurities.html
From the API, we have picked the Announced Endpoint to obtain 30 days data with JSON format output. The full URL would become: https://www.treasurydirect.gov/TA_WS/securities/announced?format=json&days=30.
Instructions Copied
- Configure Webmon plugin and enable Data Transfer
In Web-Mon plugin, the URL should be broken down into the following options:
- Host: www.treasurydirect.gov
- Protocol: https
- URL (scenarios > scenario > stages > stage > url): TA_WS/securities/announced?format=json&days=30
Having downloaded the page in Webmon, we then need pass it as a data stream to Extractor. To enable this, you will need to implement the following setting: scenarios > scenario > stages > stage > dataTransfer.
The Data Transfer Name setting (configured to securities2) can be seen in the following screenshot:
- Configure Extractor plugin and set the source of information
Moving on to the setup of the Extractor plugin, we will need to specify the source of data, which will be from the Webmon plugin in this case. This can be enabled as follows: source > samplerDataTransfer > name.
And give the name of the Data Transfer implemented in Webmon. In this example, it would be called securities2.
- Enable showmenu debug in Extractor plugin
This is optional. It will be helpful to see how the JSON tags are converted to XML format. The showmenu option can be entered at the Debug tab for the Extractor sampler. On the Active Console, it will enable a right-click menu with a Debug > JSON to XML option.
- Extracting the JSON tags
After setting up the source to come from the Webmon plugin, we can then extract individual or recursive tags with the Extractor plugin.
In this example, we want to display the source data in tabular format on the Active Console, and use cusip as the first column. As the preceding JSON to XML debug shows, the XPath for the cusip column can be represented by /array/object/cusip. However, when we examine the raw data more closely, we encounter an issue that cusip may not be unique.
To overcome this, we have selected to concatenate it with the securityTerm column represented by /array/object/securityTerm to become the first column. Then the remaining columns (such as issueDate, MaturityDate, securityType, and more) are quite straightforward to configure.
The output on the Active Console is provided below:
Older Netprobe versions Copied
If the Netprobe is older than version GA4.4, Extractor plugin will return message ERROR: Config problem: Unknown format selected at the samplingStatus headline. Users should upgrade the Netprobe version in this case.
References Copied
For more information, refer to the Web-Mon plugin documentation and Extractor plugin documentation.
To check the XML code of the samplers, see Webmon_US_Treasury.xml.txt and Extractor_US_Treasury.xml.txt.