1. Querying Sensor Data¶
WoTKit provides flexibility in how you want to query your data. In the following section, we walk through the different ways of building a query to get sensor data out of wotkit. The queries are constructed using query parameters which you append to a URL endpoint.
Generally, the use cases of the data api is to query for the raw time-series data of a sensor or group of sensors. There are two different types of queries: Recent Queries and Time Range Queries.
Recent Queries are used to easily look at recent information. The API provides parameters for you to either:
- get n most recent sensor_data
- get sensor_data since t milliseconds in the past
- Time Range Queries are useful for going through data in the past.
- These queries allow you to page through data by specifying a start & end point in time.
The following document will walk through some examples of how to take advantage of Recent Queries and Time Range Queries
1.1. Recent Queries¶
In this section we’ll dive in quickly and briefly show an example of Recent Num Queries and Recent Time Queries.
1.1.1. Recent Num Queries¶
By default, the data endpoint will return the 100 most recent queries. Try it using a URL like this:
http://wotkit.sensetecnic.com/api/v2/sensors/sensetecnic.mule1/data
The response should look similar to the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | {
"numFound": 100564,
"data": [
{
"id": 47902511,
"timestamp": "2013-11-29T00:46:36.056Z",
"sensor_id": 1,
"sensor_name": "sensetecnic.mule1",
"value": 69,
"lng": -123.17608,
"lat": 49.14103
},
{
"id": 47902514,
"timestamp": "2013-11-29T00:46:39.556Z",
"sensor_id": 1,
"sensor_name": "sensetecnic.mule1",
"value": 52,
"lng": -123.17599,
"lat": 49.13919
},
... // more data
],
"query": {
"limit": 100,
"recent_n": 100
}
}
|
The data is returned in JSON. Generally, all list responses are returned in this container to aid paging and debugging.
Field | Description |
---|---|
numFound | The total number of elements matching this query |
data | The enclosed sensor_data. Always sorted from oldest to newest timestamp |
query | Contains the interpreted query from the request. For debugging. |
metadata | Extra information. Depends on use case. |
The query field is particularly interesting because it tells you how the query was interpreted. In this case, the query has a limit of 100 and a recent_n of 100. A recent_n query fetches the n most recent items. This is useful when API users want to peek at the recent data without having to construct complex queries.
In essence, the query we ran is a convenient default for the explicit version:
http://wotkit.sensetecnic.com/api/v2/sensors/sensetecnic.mule1/data?limit=100&recent_n=100
Next we can try a recent_t query, which looks up the timestamp
1.1.2. Recent Time Queries¶
Recent Time are very similar to Recent Num Queries. The difference is that Recent Num Queries look at data count i.e. the last 10 elements, or the last 50 elements. Recent Time queries look at the timestamp instead. So, it’s useful for where we’re interested in the elements from the last hour, or the 12 hours.
Request
http://wotkit.sensetecnic.com/api/v2/sensors/sensetecnic.mule1/data?recent_t=10000
Response
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | {
"numFound": 3,
"data": [
{
"id": 47967438,
"timestamp": "2013-11-29T18:34:09.557Z",
"sensor_id": 1,
"sensor_name": "sensetecnic.mule1",
"value": 62,
"lng": -123.14509,
"lat": 49.186
},
{
"id": 47967445,
"timestamp": "2013-11-29T18:34:13.059Z",
"sensor_id": 1,
"sensor_name": "sensetecnic.mule1",
"value": 53,
"lng": -123.1454,
"lat": 49.18565
},
{
"id": 47967446,
"timestamp": "2013-11-29T18:34:16.557Z",
"sensor_id": 1,
"sensor_name": "sensetecnic.mule1",
"value": 67,
"lng": -123.14844,
"lat": 49.18323
}
],
"query": {
"limit": 100,
"recent_t": 10000
}
}
|
Looking at the query field this time, we can see it was interpreted as a recent_t query. The query looked for items up to 10 seconds ago (10000 milliseconds). You can verify this by inspecting the timestamp of the data.
Note
When accessing WoTKit anonymously, the date string is set to UTC. If you access it an api-key, the timezone will be set based on the account’s settings.
We’ve just shown you how to run both Recent Queries. One parameter to make note of is the limit parameter. At the moment, limit is capped at 100 – which restricts how much data you get in recent_n and recent_t queries. To overcome this we will look into paging through historical data next.
1.2. Time Range Queries¶
At the end of the last section, we noted that there is a weakness in the recent queries which limit your ability to sift through historical data. So, to look through historical data, you can page through historical data using the following query parameters. For the remainder of this, we will be working with the sensor rymndhng.sdq-test.
1.2.1. Querying with Start and End¶
We’ll start with a simple practical example. We have a defined starting time and ending time where we want to get all the data in between. I want to know what data was there between start: "2013-11-21T11:00:51.000Z" to end: "2013-11-29T22:59:54.862Z"
Note
It’s important to note that start is exclusive and end is inclusive. i.e. for start=100 and end=200, then the query does the following:
start < sensor_data.timestamp <= end
Query Parameters
Query Parameter | Value |
---|---|
start | 1385031651000 (2013-11-21T11:00:51.000Z) |
end | 1385765994862 (2013-11-29T22:59:54.862Z) |
Translating those two strings to milliseconds, we end up with the request below. Execute it and follow the response.
Request
http://wotkit.sensetecnic.com/api/v2/sensors/rymndhng.sdq-test/data?start=1385031651000&end=1385765994862
Response
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | {
"numFound": 5,
"data": [
{
"id": 48232725,
"timestamp": "2013-11-29T22:59:09.472Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 81
},
{
"id": 48232726,
"timestamp": "2013-11-29T22:59:09.472Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 53
},
{
"id": 48232727,
"timestamp": "2013-11-29T22:59:19.633Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 0
},
{
"id": 48232728,
"timestamp": "2013-11-29T22:59:24.715Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 56
},
{
"id": 48232729,
"timestamp": "2013-11-29T22:59:54.862Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 97
}
],
"query": {
"end": 1385765994862,
"start": 1385031651000,
"limit": 100
}
}
|
Once again, let’s look at the query parameter in the response to see what was interpreted. We can see that start/end was interpreted in the query. Inspect the timestamps of of both data points, we can see it’s between the start/end points, specifically start < data[0].timestamp < ... < data[4].timestamp < end.
1.2.2. Paging Through Data¶
In the previous section, we gave a very naive example. In this case, only two elements were in the range and therefore all the relevent data was returned. Very often this isn’t the case – and you may want to sift through thousands of entries at a time. To do this, we enabled paging through data entries. We’ll also specify limit to 10 to make the Response more comprehenedable.
Let’s try to query all the data by choosing start: 0 and and a really large end: 2000000000000.
Query Parameters
Query Parameter | Value |
---|---|
start | 0 (1970-01-01T00:00:00.000Z) |
end | 2000000000000 (2033-05-18T03:33:20.000Z) |
limit | 3 |
Request
http://wotkit.sensetecnic.com/api/v2/sensors/rymndhng.sdq-test/data?start=0&end=2000000000000&limit=3
Response
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | {
"numFound": 9,
"data": [
{
"id": 48232722,
"timestamp": "2013-11-21T10:58:51.000Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 6.7
},
{
"id": 48232723,
"timestamp": "2013-11-21T10:59:51.000Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 6.8
},
{
"id": 48232724,
"timestamp": "2013-11-21T11:00:51.000Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 6.9
}
],
"query": {
"end": 2000000000000,
"start": 0,
"limit": 3
}
}
|
So this time, first we look at the query parameter. As mentioned previously, the limit is currently capped at 3. So how do we know if we there’s more data? Well, there is another field in the response which can help us: numFound. numFound counts all the data found within the data range from start to end. In this example, we know there’s more data because data.length < numFound.
Given this information, we can now continue paging data by setting offset. We can retrieve the next page by choosing offset = data.size, in this case, data.size is 10. Generally, we can page by specifying offset = prev_offset + data.size. We can also figure out if we’re at the end of the data range generally by testing that data.size + offset < numFound.
Now, let’s rerun the last query with an offset.
Query Parameters
Parameter | Value |
---|---|
start | 0 (same as before |
end | 2000000000000 (same as before) |
limit | 10 |
offset | 3 |
Request
http://wotkit.sensetecnic.com/api/v2/sensors/rymndhng.sdq-test/data?start=0&end=2000000000000&limit=3&offset=3
Response
{
"numFound": 9,
"data": [
{
"id": 48232725,
"timestamp": "2013-11-29T22:59:09.472Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 81
},
{
"id": 48232726,
"timestamp": "2013-11-29T22:59:09.472Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 53
},
{
"id": 48232727,
"timestamp": "2013-11-29T22:59:19.633Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 0
}
],
"query": {
"offset": 3,
"end": 2000000000000,
"start": 0,
"limit": 3
}
}
Once again, looking at the query, we can now see that offset is specfied as 3. We can also verify that an offset was used by looking at id and timestamp of the two responses. The last element of the first response has id: 48232724 and timestamp: "2013-11-21T11:00:51.000Z". The first element in the second response has id: 48232725 and timestamp: "2013-11-29T22:59:09.472Z". You can easily verify that they are in sequence.
1.2.3. Advanced Time Range Queries¶
In general, using start, end, offset provides enough flexibility. However, sensors are allowed to have multiple data on the same timestamp. This can easily happen when historical data is PUT into the system. In other words, you cannot expect timestamp to be unique for sensor data (generally they are good enough). So, we introduce the idea of start_id and end_id to allow precise selection of start and end elements.
We’ll start off with our first query .. code-block:: javascript
Response
{
"numFound": 9,
"data": [
{
"id": 48232722,
"timestamp": "2013-11-21T10:58:51.000Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 6.7
},
{
"id": 48232723,
"timestamp": "2013-11-21T10:59:51.000Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 6.8
},
{
"id": 48232724,
"timestamp": "2013-11-21T11:00:51.000Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 6.9
},
{
"id": 48232725,
"timestamp": "2013-11-29T22:59:09.472Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 81
}
],
"query": {
"start": 0,
"limit": 4
}
}
Now sometime in the future, we want to rerun the query using the information we had previously. So, we’ll use the last item’s timestamp (2013-11-29T22:59:09.472Z) as the start value.
Request
http://wotkit.sensetecnic.com/api/v2/sensors/rymndhng.sdq-test/data?start=1385765949472&limit=4
Response
{
"numFound": 4,
"data": [
{
"id": 48232727,
"timestamp": "2013-11-29T22:59:19.633Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 0
},
{
"id": 48232728,
"timestamp": "2013-11-29T22:59:24.715Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 56
},
{
"id": 48232729,
"timestamp": "2013-11-29T22:59:54.862Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 97
},
{
"id": 48232730,
"timestamp": "2013-11-29T23:00:24.862Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 6.7
}
],
"query": {
"start": 1385765949472,
"limit": 4
}
}
Everything looks fine and dandy doesn’t it? The timestamps are incremental, and therefore all is well is it? Well, no it actually isn’t. There’s a problem which we are unaware of. We’ve actually skipped an element because of duplicate timestamps.
Run this request which querys the entire range and look at the data.
Request
http://wotkit.sensetecnic.com/api/v2/sensors/rymndhng.sdq-test/data
Response
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | {
"numFound": 9,
"data": [
{
"id": 48232722,
"timestamp": "2013-11-21T10:58:51.000Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 6.7
},
{
"id": 48232723,
"timestamp": "2013-11-21T10:59:51.000Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 6.8
},
{
"id": 48232724,
"timestamp": "2013-11-21T11:00:51.000Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 6.9
},
{
"id": 48232725,
"timestamp": "2013-11-29T22:59:09.472Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 81
},
{
"id": 48232726,
"timestamp": "2013-11-29T22:59:09.472Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 53
},
{
"id": 48232727,
"timestamp": "2013-11-29T22:59:19.633Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 0
},
{
"id": 48232728,
"timestamp": "2013-11-29T22:59:24.715Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 56
},
{
"id": 48232729,
"timestamp": "2013-11-29T22:59:54.862Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 97
},
{
"id": 48232730,
"timestamp": "2013-11-29T23:00:24.862Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 6.7
}
],
"query": {
"limit": 100,
"recent_n": 10
}
}
|
The highlighted lines for id: 48232726 did not exist in either of our queries. In Querying with Start and End, we specified the second query did exactly what you asked for: Query sensor data after timestamp 1385765949472 limit 3. So, to solve this, we introduce a new parameter start_id. This parameter can be used in conjuction with start to specify specify which data element’s id to start with. Essentially, sensor_data are uniquely identified using this tuple (timestamp, id). So, let’s rerun the second query with start_id: 48232725 from the first query.
Request
http://wotkit.sensetecnic.com/api/v2/sensors/rymndhng.sdq-test/data?start=1385765949472&limit=4&start_id=48232725
Response
{
"numFound": 5,
"data": [
{
"id": 48232726,
"timestamp": "2013-11-29T22:59:09.472Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 53
},
{
"id": 48232727,
"timestamp": "2013-11-29T22:59:19.633Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 0
},
{
"id": 48232728,
"timestamp": "2013-11-29T22:59:24.715Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"valua": 56
},
{
"id": 48232729,
"timestamp": "2013-11-29T22:59:54.862Z",
"sensor_id": 531,
"sensor_name": "rymndhng.sdq-test",
"value": 97
}
],
"query": {
"start": 1385765949472,
"limit": 4,
"start_id": 48232725
}
}
There, we got the response with id: 48232726. The start_id allowed us to filter ids greater than 3. end_id works the same way as start_id if you really need fine-grained control over the range of a data query.
1.2.4. Summary of Time Range Data Query¶
With all the information given, we can really condense the query parameters into the following query. data_ts is the sensor data’s timestamp, and data_id is the data’s id element.
Without start_id or end_id, the query range is done like this.
start < data_ts <= end
With start_id and/or end_id, the query range adds extra checks near the bounds
(start < data_ts <= end)
OR (data_ts = start AND data_id > start_id)
OR (data_ts = end AND data_id <= end_id)
Below is a quicky summary of what query parameter means:
Parameter | Type | Description |
---|---|---|
start | timestamp | The absolute starting point (in milliseconds since Jan 1, 1970). |
start_id | id | The starting id of sensor_data at timestamp start. Used for paging. |
end | timestamp | The absolute ending timestamp (in milliseconds since Jan 1, 1970) |
end_id | timestamp | The end id of sensor_data with timestamp end. Used for paging. |
1.3. Sensor Data Query Recipes¶
In this section, we will highlight some novel ways of combining the information above to query the data.
1.3.1. Use start_id instead of start for start of query¶
In the documentation, we used start_id alongisde start, but actually, this is optional. If you use start_id without start, we will actually lookup the timestamp of the element with id start_id, and then use that as the starting timestamp.
1.3.2. Making Start Inclusive¶
From Summary of Time Range Data Query, it shows the start range is exclusive. But, there is a way to make this inclusive. If you set start_id: 0, it will make the data range inclusive.