113 lines
4.8 KiB
Markdown
113 lines
4.8 KiB
Markdown
# Telemetry
|
|
|
|
A simple yet functional library for capturing runtime analytic events from embedded devices.
|
|
|
|
# Build & Run
|
|
|
|
Consult the `.devcontainers/Dockerfile` for the list of project dependencies.
|
|
|
|
## VsCode Dev Containers
|
|
|
|
I am in no way a M$ fanboy, but the VsCode IDE seems to be a versatile, Swiss-army-knife tool. Long story short: you are highly advised to use the latest version of VsCode for a quick, hassle-free yet comprehensive overview of the project.
|
|
|
|
1. Upon opening the project's folder, VsCode should ask you to reopen the project as a `devcontainer`. Simply agree, and allow all necessary configuration to happen in the background.
|
|
|
|
> ⚠️ If not prompted ⚠️ <br>
|
|
> `Ctrl + Shift + P` > `Dev Containers: Reopen in Container`
|
|
|
|
2. Use the pre-installed **CMake** extension to configure and build the project.
|
|
|
|
3. Use **Test Explorer** to run and debug tests.
|
|
|
|
# Library design goals
|
|
|
|
- agnostic to payload format
|
|
- adequate runtime overhead
|
|
- non intrusive / easy to use
|
|
|
|
# Protobuf v3
|
|
|
|
Generally speaking, it is impossible to predict what kind of information would posses the most value in the future. Thus communication protocols tend to evolve with time. Not all customers are willing to update their devices on demand. As a result of such forces of nature, it is unavoidable that even devices of a same model would send Telemetry messages of different format. Endpoin servers (collectors) shall be capable of effective handling of such situation. [Protobuf](https://protobuf.dev/overview/) is well known industry solution for such problems.
|
|
|
|
A known limitation of `Protobuf` library - is an inability to effectively store multiple message in file in serial aka `one-by-one` fashion. In order to overcome such limitation, our solution makes use of simple Length-Delimited (a simplified version of [TLV](https://en.wikipedia.org/wiki/Type%E2%80%93length%E2%80%93value) format) file encoding. Where essentially first two bytes of each serialized message are used to store message length.
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
Length1
|
|
Message1
|
|
Length2
|
|
Message2
|
|
Length3
|
|
Message3
|
|
```
|
|
|
|
# Architecture
|
|
|
|
`Datapoint` aka *analytic event* representation - is a handy wrapper / utility class, aimed to ease usage of somewhat bloated autogenerated `protobuf` classes.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
subgraph .proto
|
|
AnalyticsEvent --- M{Message}
|
|
TemperatureReading --- M
|
|
ShutdownReason --- M
|
|
etc.. --- M
|
|
end
|
|
|
|
D(Datapoint) -. parse .-> M
|
|
M -. make .-> D
|
|
```
|
|
|
|
Than, a `Sink` instance shall be used to establish flow from runtime memory of an captured `Datapoints` into it's serialized form on disk. To `capture()` some `Datapoint`, an instance of `Writer` class shall be used. Each `Writer` instance is linked to it's `Sink`-paren class. `Writer` is a movable class that implements a simple, polymorphic, buffered and thread safe API for `Datapoints` (events) capturing.
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
subgraph "runtime (orbiter)"
|
|
DP1(Datapoint 1) -.capture.- W1(Writer 1)
|
|
DP2(Datapoint 2) -.capture.- W1
|
|
DP3(Datapoint 3) -.capture.- W2(Writer 2)
|
|
DP4(Datapoint 4) -.capture.- W2
|
|
W1 --- S(Sink)
|
|
W3(Writer n) --- S
|
|
W2 --- S
|
|
|
|
end
|
|
S(Sink) ---> DB[(File)]
|
|
DB[(File)] ---> R(Reader)
|
|
subgraph "runtime (server)"
|
|
R(Reader) -.parse.-> D1(Datapoint 1)
|
|
R(Reader) -.parse.-> D2(Datapoint 2)
|
|
R(Reader) -.parse.-> D3(Datapoint 3)
|
|
R(Reader) -.parse.-> D4(Datapoint 4)
|
|
end
|
|
```
|
|
`Reader` class shall be used in order to deserialize `Datapoints` from file. All `Datapoints` will be read in one-by-one fashion.
|
|
|
|
# Tests and diskussion
|
|
|
|
Tests for the project designed in such a way, so they can be used as a case study of an API usage as well as a way to ensure code quality.
|
|
|
|
## Datapoint
|
|
Shows basic API use case. As well as provides a way to reliably discriminate between multiple types of `events`.
|
|
|
|
## Serial IO
|
|
Cowers a case when all events being captured by single `reader` in serial fashion:
|
|
> write - write - write - read - read - read
|
|
|
|
Also shows a way to check if there is some data to read from file.
|
|
|
|
## Mixed IO
|
|
Slightly more complex case, where writing *into* and reading *from* file done in mixed order:
|
|
> write - read - write - write - read - read
|
|
|
|
## Buferization
|
|
Writing data to disk (even SSD) is notoriously slow operation. Storing several messages in RAM and then writing all of them on disk as single batch - is a way to speed things up.
|
|
|
|
> [!NOTE] Trade off
|
|
> In case of sudden power los - all cached data (aka not stored to disk) will be irretrievably lost
|
|
|
|
## Multithreading
|
|
Yes, all `Writer` instances are thread safe. Altho, so further improvement can be made here. See comments in `telemetry\sink.cpp Sink::Writer::capture()`.
|
|
|
|
## Shared
|
|
`RAII` and `std::shared_pointer` is exactly that type of magic, that provides ease of mind. No need to worry about dangling pointers, leaky memory and problems alike. |