telemetry/README.md

Telemetry

A simple yet functional library for capturing runtime analytic events from embedded devices.

Build & Run

Consult the .devcontainers/Dockerfile for the list of project dependencies.

VsCode Dev Containers

I am in no way a M$ fanboy, but the VsCode IDE seems to be a versatile, Swiss-army-knife tool. Long story short: you are highly advised to use the latest version of VsCode for a quick, hassle-free yet comprehensive overview of the project.

1. Upon opening the project's folder, VsCode should ask you to reopen the project as a devcontainer. Simply agree, and allow all necessary configuration to happen in the background.

   ⚠️ If not prompted ⚠️
   Ctrl + Shift + P > Dev Containers: Reopen in Container

2. Use the pre-installed CMake extension to configure and build the project.

3. Use Test Explorer to run and debug tests.

Library design goals

• agnostic to payload format
• adequate runtime overhead
• non intrusive / easy to use

Protobuf v3

Generally speaking, it is impossible to predict what kind of information would posses the most value in the future. Thus communication protocols tend to evolve with time. Not all customers are willing to update their devices on demand. As a result of such forces of nature, it is unavoidable that even devices of a same model would send Telemetry messages of different format. Endpoin servers (collectors) shall be capable of effective handling of such situation. Protobuf is well known industry solution for such problems.

A known limitation of Protobuf library - is an inability to effectively store multiple message in file in serial aka one-by-one fashion. In order to overcome such limitation, our solution makes use of simple Length-Delimited (a simplified version of TLV format) file encoding. Where essentially first two bytes of each serialized message are used to store message length.

flowchart TB
	Length1
	Message1
	Length2
	Message2
	Length3
	Message3

Architecture

Datapoint aka analytic event representation - is a handy wrapper / utility class, aimed to ease usage of somewhat bloated autogenerated protobuf classes.

flowchart LR
    subgraph .proto
	    AnalyticsEvent --- M{Message}
	    TemperatureReading --- M  
	    ShutdownReason --- M  
	    etc.. --- M 
    end

	D(Datapoint) -. parse .-> M
	M -. make .-> D

Than, a Sink instance shall be used to establish flow from runtime memory of an captured Datapoints into it's serialized form on disk. To capture() some Datapoint, an instance of Writer class shall be used. Each Writer instance is linked to it's Sink-paren class. Writer is a movable class that implements a simple, polymorphic, buffered and thread safe API for Datapoints (events) capturing.

flowchart TB
	subgraph "runtime (orbiter)"
		DP1(Datapoint 1) -.capture.- W1(Writer 1)
		DP2(Datapoint 2) -.capture.- W1
		DP3(Datapoint 3) -.capture.- W2(Writer 2)
		DP4(Datapoint 4) -.capture.- W2
	    W1 --- S(Sink)
	    W3(Writer n) --- S
	    W2 --- S

	end
	S(Sink) ---> DB[(File)]
	DB[(File)] ---> R(Reader)
	subgraph "runtime (server)"
		R(Reader) -.parse.-> D1(Datapoint 1)
		R(Reader) -.parse.-> D2(Datapoint 2)
		R(Reader) -.parse.-> D3(Datapoint 3)
		R(Reader) -.parse.-> D4(Datapoint 4)
	end

Reader class shall be used in order to deserialize Datapoints from file. All Datapoints will be read in one-by-one fashion.

Tests and diskussion

Tests for the project designed in such a way, so they can be used as a case study of an API usage as well as a way to ensure code quality.

Datapoint

Shows basic API use case. As well as provides a way to reliably discriminate between multiple types of events.

Serial IO

Cowers a case when all events being captured by single reader in serial fashion:

write - write - write - read - read - read

Also shows a way to check if there is some data to read from file.

Mixed IO

Slightly more complex case, where writing into and reading from file done in mixed order:

write - read - write - write - read - read

Buferization

Writing data to disk (even SSD) is notoriously slow operation. Storing several messages in RAM and then writing all of them on disk as single batch - is a way to speed things up.

[!NOTE] Trade off
In case of sudden power los - all cached data (aka not stored to disk) will be irretrievably lost

Multithreading

Yes, all Writer instances are thread safe. Altho, so further improvement can be made here. See comments in telemetry\sink.cpp Sink::Writer::capture().

Shared

RAII and std::shared_pointer is exactly that type of magic, that provides ease of mind. No need to worry about dangling pointers, leaky memory and problems alike.