Air Temperature Log Processing Using Reusable Streaming CSV and SVG Dataset Algorithms

Full abstract

Read the full abstract

Here, we propose an adaptive and easy extendable approach for collecting large-scale sensor data streams and processing the stored data with standard dataset algorithms. The practice shows that the long-term sensor monitoring produces time-ordered logs that usually exceed the practical spreadsheet and in-memory processing limits. This work continues our previous measurements with an LMT70EVM ambient-air temperature system, which generates app. 1.48 GB per log over period of 20 days, with about 35 million samples at a 50 ms period. Now, we extend that method into a reusable Python-based library tailored as a processing architecture for large sensor logs. Incremental file and CSV iterators are combined with closest-stream mapping, automatic small-shift detection, synchronization checks, direct duration aggregation, typed error samples, and SVG plotting, which are implemented as applicable processing options. The implementation presets 12.8 s detection buckets and 60 s detailed data averages are now grouped into 30 min snapshots. During processing, the stable snapshots are reduced to one weighted point, whereas the snapshots classified as containing a detected temperature drift retain 60 s detailed data values. There is a CSV output option, allowing direct reduced-sample-to-SVG processing. To reduce the overall dataset reconstruction time of the applied library in case of reuse with data from other sources, we construct a small-log equivalence test bench that produces 29 rows in both readable and direct paths, 261 compared values, and zero differences. The proposed implementation establishes reusable streaming, verification, and visualization layers required for any long-term measurement. We also demonstrate a second data stream logging from a three-axis gyroscope with CSV data through motion detection, bias compensation, quaternion integration, tabular export, and XY SVG visualization. Although large-scale gyroscope streaming is not yet demonstrated, it will be part of the future sensor fusion implementation of the library.