# Telemetry 

A simple yet functional library for capturing runtime analytic events from embedded devices. 

# Build & run

`Dockerfile` with two helper scripts were added into project's root folder. Buy doing so, two useful goals were achieved:
- **Infrastructure as a code**:
	All project dependancies and as well as all installation / configuration steps are easily documented as groups of handy scripts inside `Dockerfile`. 
- **Containerization**:
	Almost instant ability to jump into app development, testing and/or deployment with zero footprint (pollution) on main (host) PC. 

## IDE

[VsCode](https://code.visualstudio.com/) IDE was used for in-container development. There is two different ways how one might approach this task:
- [Attach](https://code.visualstudio.com/docs/devcontainers/attach-container) VsCode to already [[#Build & run|running]] container
- Manually [install](https://github.com/ahmadnassri/docker-vscode-server/blob/master/Dockerfile) VsCode IDE into container and use [X11 window forwarding](https://goteleport.com/blog/x11-forwarding/), to automagically *spawn* VsCode IDE window on host PC 

# Design goals

- agnostic to payload format
- adequate runtime overhead
- non intrusive / easy to use

# Protobuf v3

Generally speaking, it is impossible to predict what kind of information would posses the most value in the future. Thus communication protocols tend to evolve with time. Not all customers are willing to update their devices on demand. As a result of such forces of nature, it is unavoidable that even devices of a same model would send Telemetry messages of different format. Endpoin servers (collectors) shall be capable of effective handling of such situation. [Protobuf](https://protobuf.dev/overview/) is well known industry solution for such problems.

A known limitation of `Protobuf` library - is an inability to effectively store multiple message in file in serial aka `one-by-one` fashion. In order to overcome such limitation, our solution makes use of simple Length-Delimited (a simplified version of [TLV](https://en.wikipedia.org/wiki/Type%E2%80%93length%E2%80%93value) format) file encoding. Where essentially first two bytes of each serialized message are used to store message length.

```mermaid
flowchart TB
	Length1
	Message1
	Length2
	Message2
	Length3
	Message3
```

# Architecture

`Datapoint` aka *analytic event* representation - is a handy wrapper / utility class, aimed to ease usage of somewhat bloated  autogenerated `protobuf` classes.

```mermaid
flowchart LR
    subgraph .proto
	    AnalyticsEvent --- M{Message}
	    TemperatureReading --- M  
	    ShutdownReason --- M  
	    etc.. --- M 
    end

	D(Datapoint) -. parse .-> M
	M -. make .-> D
```

Than, a `Sink` instance shall be used to establish flow from runtime memory of an captured `Datapoints` into it's serialized form on disk.  To `capture()` some `Datapoint`, an instance of `Writer` class shall be used. Each `Writer` instance is linked to it's `Sink`-paren class. `Writer` is a movable class that implements a simple, polymorphic, buffered and thread safe API for `Datapoints` (events) capturing.

```mermaid
flowchart TB
	subgraph "runtime (orbiter)"
		DP1(Datapoint 1) -.capture.- W1(Writer 1)
		DP2(Datapoint 2) -.capture.- W1
		DP3(Datapoint 3) -.capture.- W2(Writer 2)
		DP4(Datapoint 4) -.capture.- W2
	    W1 --- S(Sink)
	    W3(Writer n) --- S
	    W2 --- S

	end
	S(Sink) ---> DB[(File)]
	DB[(File)] ---> R(Reader)
	subgraph "runtime (server)"
		R(Reader) -.parse.-> D1(Datapoint 1)
		R(Reader) -.parse.-> D2(Datapoint 2)
		R(Reader) -.parse.-> D3(Datapoint 3)
		R(Reader) -.parse.-> D4(Datapoint 4)
	end
```
`Reader` class shall be used in order to deserialize `Datapoints` from file. All `Datapoints` will be read in one-by-one fashion.

# Tests and diskussion

Tests for the project designed in such a way, so they can be used as a case study of an API usage as well as a way to ensure code quality.

## Datapoint
Shows basic API use case. As well as provides a way to reliably discriminate between multiple types of `events`.
  
## Serial IO
Cowers a case when all events being captured by single `reader` in serial fashion: 
>  write - write - write - read - read - read

Also shows a way to check if there is some data to read from file.
  
## Mixed IO
Slightly more complex case, where writing *into* and reading *from* file done in mixed order:
> write - read - write - write - read - read 
  
## Buferization
Writing data to disk (even SSD) is notoriously slow operation. Storing several messages in RAM and then writing all of them on disk as single batch - is a way to speed things up.
  
> [!NOTE] Trade off  
> In case of sudden power los - all cached data (aka not stored to disk) will be irretrievably lost

## Multithreading
Yes, all `Writer` instances are thread safe. Altho, so further improvement can be made here. See comments in `telemetry\sink.cpp Sink::Writer::capture()`.
  
## Shared
`RAII` and `std::shared_pointer` is exactly that type of magic, that provides ease of mind. No need to worry about dangling pointers, leaky memory and problems alike.