A quick note: If you noticed a bigger gap than usual between posts, my apologies! Things have been hectic on personal and professional fronts — in the best way possible. I recently took on a Lead Software Engineer role at Capital One and presented this work at the University for my dissertation. Both experiences served as the ultimate stress test for the ideas explored in this series. Now, back to it.

One of the most underestimated challenges in Human Activity Recognition (HAR) research is not model design — it's data acquisition at scale. For multimodal, multi-resident systems, collecting consistent, well-labeled, and privacy-aware data quickly becomes the dominant bottleneck.

In this phase of the project, I focused on designing and validating a data-gathering orchestrator mode: a system that allows researchers to boot the device, label an activity, and begin collecting synchronized multimodal data with minimal friction. This post walks through how the orchestrator was built, how data were collected over two days, and the key design decisions underlying model training.

01 — Why an Orchestrator-Centric Design?

Early prototyping revealed a familiar failure mode in HAR research: data-collection logic scattered across sensor services, ad hoc scripts, and manual synchronization steps. The result is a fragile pipeline that is difficult to reproduce, extend, or hand off to another researcher.

To address this, I designed the Edge Orchestrator as the single source of truth for everything that matters at collection time:

• Sensor coordination and stream management

• Activity labeling and buffering

• Privacy-preserving preprocessing

• Cloud upload and structured dataset generation

The key inversion of control: sensors stream continuously, and the orchestrator decides when and how to capture data. This enables two clean execution modes — Data Gathering Mode and Predictor (Inference) Mode. This entry focuses on the former.

02 — What Is the Orchestrator?

The Orchestrator is the backbone of the HAR system, coordinating data flow, preprocessing, and activity recognition across all tiers. At startup, it initiates a gRPC server that acts as the unified endpoint for all sensor services — receiving Protocol Buffer payloads and transforming them into Python objects for downstream processing. The public repository of the orchestrator can be found at the following URL: https://github.com/RodCaba/fp-orchestrator

Researchers interact with the system through a dedicated UI built with HTML, JS, and CSS, exposed via FastAPI over HTTP. A WebSocket endpoint enables real-time updates, allowing the interface to display:

• Connection status of each sensor service

• Currently identified inhabitants via RFID

• Data batches processed in gathering mode, or the active prediction label in predictor mode

Fig. 1 — The Orchestrator UI: sensor connection status, RFID presence, and real-time stream monitoring.

03 — Data Gathering Mode: From Label to Dataset

From the researcher's perspective, the data-gathering flow is intentionally simple. The complexity lives in the system, not in the workflow:

1. Select an activity label from the orchestrator UI.

2. Swipe at least one RFID tag to declare occupant presence.

3. Press the start activity button.

4. Perform the activity naturally — no scripted behaviors required.

5. Let the system handle synchronization, preprocessing, and upload.

Fig. 2 — Data flow across the edge, orchestration, and cloud tiers during a gathering session.

RFID as the Collection Trigger

RFID presence acts as the implicit trigger for data collection. When at least one unique tag is detected, and the start button is pressed, the orchestrator activates audio and IMU streams, begins buffering synchronized multimodal data, and records the number of detected users as a first-class feature in every sample.

This design choice ensures that collected data always corresponds to actual occupancy and the researcher's intentional decision that an activity has started or ended — not an automated timer or environmental trigger that could introduce label noise.

Temporal Buffering and Cloud Upload

Incoming data is streamed via gRPC into the orchestrator, where it is immediately preprocessed, anonymized (no raw audio is ever stored), and held in a temporal edge buffer. Once the buffer reaches 10,000 records, the orchestrator:

• Serializes the batch into a structured JSON file

• Uploads it to AWS S3

• Clears the local buffer and resumes collection

This batching strategy provides network efficiency, fault tolerance, and clean dataset segmentation for training. If a cloud upload fails, the system retains the batch locally to prevent data loss.

Design Candor: During the initial S3 uploads, some IMU readings were incorrectly labeled due to a configuration error. Because the system stores files as blobs, corrections cannot be made in place — the mislabeled data must be accounted for in the data-loading pipeline at training time. This is a known limitation and a reminder that error correction in blob storage requires pipeline-level handling, not in-place edits.

04 — Two Days of Real-World Data Collection

Using this orchestrated flow, I deployed the system in a three-person household kitchen over two days of naturalistic activity. The result:

Fig. 3 — Data in the Cloud Tier.

Crucially, this data was collected without modifying the environment or forcing scripted behaviors. Activities emerged naturally, including concurrent and collaborative actions — exactly the scenarios that challenge multi-resident HAR systems.

Making realistic data collection easy enough to be repeated and extended by other researchers was one of the core goals of the orchestrator. This dataset validates that it works.

05 — "Boot and Collect": Researcher Experience

From a usability standpoint, the system was designed to minimize setup overhead. A researcher can boot the edge device, connect the mobile IMU stream, open the orchestrator UI, and start collecting labeled data within minutes. In practice, the time from system startup to the availability of labeled data in S3 averaged roughly two minutes. This experience can be viewed in the following video:

https://youtu.be/XC-YxUs6ezs

• Dataset growth becomes incremental instead of painful

• New activity labels can be added without pipeline changes

• Model retraining becomes an expected, repeatable step — not a risky undertaking

The orchestrator effectively promotes data collection from an afterthought into a first-class system capability.

06 — Model Training: Design Decisions and Trade-offs

In the previous entry, I introduced the fp-orchestrator-utils package, which provides a CLI for downloading and uploading proto definitions to S3, along with a wrapper for S3 operations via boto3. This package was extended to incorporate data loading, inference modal, and training logic. The package repoitory can be found on the following URL: https://github.com/RodCaba/fp-orchestrator-utils

The DataLoader

The DataLoader class establishes a secure S3 connection via environment variables and offers two primary modes: downloading all JSON data from an S3 bucket (with optional local caching), or loading from a local directory. The class processes data into feature and label arrays, performing cleaning, transformation, and class label encoding using sklearn.LabelEncoder.

Model Architecture: Why Not One Big Network?

Rather than collapsing all inputs into a monolithic architecture, I opted for modular processors per modality. Each sensor type gets its own dedicated processor; features are then fused through a shared attention mechanism. The rationale:

• IMU sensors have varying dimensionalities (orientation has 7 dimensions; others have 3)

• Missing modalities can be handled gracefully by filling with zero tensors

• Edge inference constraints favor modular, quantizable components

• Independent processors allow targeted fine-tuning without retraining the full network

IMU Sensor Processor

Each IMU sensor (accelerometer, gyroscope, total acceleration, gravity, orientation) has its own processor instance. The pipeline: linear projection → LSTM over the time sequence → average pooling over the time dimension → final projection with dropout.

Audio Processor

Input is a tensor of mel spectrograms. The pipeline: projection from mel bands to a fixed size → two LSTM layers (first processes within-segment time, second processes across segments) → average pooling over both dimensions → final projection with dropout.

Attention as a Fusion Strategy

Not all modalities are equally informative for all activities. The attention mechanism — inspired by Nakabayashi and Saito (2024) — allows the model to learn which sensors matter most in a given context: IMU-heavy signals during motion-intensive activities, audio-dominant cues during appliance-based actions. This proved especially valuable in collaborative and overlapping activity scenarios.

# har_model.py — HARModel forward pass (simplified)

class HARModel(nn.Module):
    def forward(self, sensor_data: dict[str, torch.Tensor], n_users: torch.Tensor):
        # 1. Run each available modality through its dedicated processor
        features = []
        for modality, processor in self.processors.items():
            if modality in sensor_data:
                features.append(processor(sensor_data[modality]))
            else:
                features.append(torch.zeros(batch_size, self.imu_feature_size))

        # 2. Fuse via cross-modal attention
        attended = self.feature_attention(features)

        # 3. Concatenate fused features with RFID-derived user count
        n_users_exp = n_users.float().unsqueeze(1)
        combined = torch.cat(attended + [n_users_exp], dim=1)

        # 4. Classify
        return self.classifier(combined)

07 — Training Strategy and the HARDataset

Model training followed a conservative, reproducible workflow: data loaded directly from S3, explicit train/validation splits, best model checkpoint selected via validation accuracy, and final export to ONNX for edge deployment.

The dataset is encapsulated in a HARDataset class — a PyTorch dataset accessor designed for the open-source dataset derived from this collection effort. It includes a custom collate function to handle variable-length sequences across sensors with differing sampling frequencies. Each sample is structured as:

# har_dataset.py — sample structure

sample = {
    'features': upload_sample['features'],  # dict of sensor tensors
    'n_users':  upload_sample['n_users'],   # RFID-detected occupants
    'label':    self.labels[idx]            # encoded activity class
}

The module supports optional transformation functions that researchers can plug in per sample, making the dataset straightforward to extend without modifying the core pipeline.

Crucially, the training pipeline is repeatable: new data improves the model without architectural changes. This was a deliberate design goal from the start.

08 — What This Phase Enables

This phase marks the transition from a working prototype to a research platform — a system designed not just to recognize activities, but to support the iterative nature of HAR experimentation. Practically, this means:

• Researchers can grow the dataset incrementally without rebuilding pipelines

• New activity labels can be introduced organically as the research evolves

• Model retraining becomes routine, not risky

• The same system serves both research exploration and production deployment goals

Coming up next: Evaluation, system metrics, and edge performance trade-offs — what do they reveal about deploying multimodal HAR in real-world environments?

• Finalizing the Inertial Measurement Unit (IMU) sensor integration using Message Queuing Telemetry Transport (MQTT) for mobile streaming.

• Designing the first layer of the orchestrator, which bridges the sensor and cloud layers to build a curated dataset.

IMU Sensor Integration via MQTT

To integrate motion data from mobile devices, I used the Sensor Logger app for Android. Its premium version supports MQTT, allowing seamless publishing of IMU data from the phone to a central broker.

The following architecture exemplifies the MQTT Publish / Subscribe architecture:

Source: https://mqtt.org

MQTT, in short, hosts clients that publish to topics, and subscribers receive those messages.

I used Eclipse Mosquitto as my MQTT broker, hosted on the Raspberry Pi.

Here's a sample configuration to enable public access for testing:

# Allow connections from network
listener 1883 0.0.0.0

# Allow anonymous connections (for testing)
allow_anonymous true

# Logging
log_dest stdout
log_type all

# Persistence
persistence false

Then we start the broker:

rodrigo@raspberrypi:~/fp-imu-service $ mosquitto -c mosquitto.conf -v
1752622740: mosquitto version 2.0.11 starting
1752622740: Config loaded from mosquitto.conf.
1752622740: Opening ipv4 listen socket on port 1883.
1752622740: mosquitto version 2.0.11 running

There might already be a mosquitto process running after installation. To run the process with the desired configuration, we need to stop the current process using the command: pkill mosquitto.

Sensor Layer Architecture Overview

The following diagram depicts the architecture of the sensor layer with the MQTT broker:

MQTT Topics:

1. recording_control: instructs mobile devices when to start/stop logging.

2. data_stream: receives IMU payloads from devices.

Participants:

• RFID service → Publishes control messages (start/stop)

• Mobile device → Subscribes to control, publishes IMU data

• IMU service → Subscribes to data, then buffers and processes payloads

Creating a Reusable MQTT Client: fp-mqtt-broker

I would perform MQTT broker operations on multiple devices and services. To reduce boilerplate across services (connection, disconnection, and message handling logic), I created and published a Python package: fp-mqtt-broker on PyPI

This package simplifies and offers an easy-to-use interface for connecting a device to an MQTT broker. It includes a ready-to-use implementation of the paho-mqtt client, but it is designed to be flexible and not dependent on this client, using a factory pattern to create MQTT brokers for:

• Connection setup

• Topic subscriptions

• Message handling (via a factory + handler interface)

This is the basic example of using the package to create a new client to the MQTT broker and assign a custom message handler:

from fp_mqtt_broker import BrokerFactory
from fp_mqtt_broker.abstractions import MessageHandler

class MyHandler(MessageHandler):
    def get_subscribed_topics(self): return ['my/topic']
    def handle_message(self, topic, payload): print(f"→ {payload}")

broker = BrokerFactory.create_broker(config, [MyHandler()])
broker.connect()
broker.publish_message("my/topic", "hello world")

Buffering IMU Data

The new IMU service includes a buffer system for:

• Accelerometer

• Gyroscope

• Gravity

• Orientation (quaternions, pitch/roll/yaw)

Each reading is validated before being added to a capped buffer.

class IMUBuffer:
    """Class to manage the IMU data buffer."""

    def validate_sensor_values(self, values, name):
        """Validate the structure of sensor values."""
        if not isinstance(values, dict):
            raise ValueError("Values must be a JSON object")

        # Check for required fields in values
        required_fields = ['x', 'y', 'z']
        # Orientation requires a different structure.

        for field in required_fields:
            if field not in values:
                raise ValueError(f"Missing required field: {field}")

    def add_to_buffer(self, data, buffer):
        """Add new IMU data to the buffer."""
        if len(buffer) >= self.max_size:
            buffer.pop(0)  # Remove oldest data
        buffer.append(data)

An IMUMessageHandler subscribes to the data_stream topic and pushes incoming data into the buffer. Payloads are expected in the form:

{
  "payload": [
    { "name": "accelerometer", "values": { "x": ..., "y": ..., "z": ... } },
    ...
  ]
}

Updated RFID Service Behavior

The RFID service now:

• Connects to the MQTT broker using fp-mqtt-broker

• Publishes start or stop commands based on tag swipes

• Simultaneously triggers audio recognition via gRPC

If you don't recall the details and tasks of the RFID service, you can revisit the previous entry at https://typo.hashnode.dev/from-rfid-to-recognition. In short, it is a service that manages GPIO connections with the Raspberry Pi, including an RC522 RFID reader, and interacts with devices like an LCD screen, buzzers, and LEDs.

After the connection is set, the MQTT broker will publish a message to the “recording_control” channel to instruct the start or end of IMU data gathering, depending on the RFID swipe timing.

if not IS_READING:
          # ==================== MQTT BROKER MESSAGE PUBLICATION ===========================
          mqtt_broker.publish_message(
            topic=config['mqtt']['topics']['recording_control'],
            payload=json.dumps({"action": "start", "session_id": id})
          )

else:       
          # ==================== MQTT BROKER MESSAGE PUBLICATION ===========================
          mqtt_broker.publish_message(
            topic=config['mqtt']['topics']['recording_control'],
            payload=json.dumps({"action": "stop", "session_id": id})
          )

Integration Results

The video below shows the result of integrating MQTT with the sensor layer.

As the video shows, the sensor layer is fully connected, and each RFID tag swipe tells the system to start or stop gathering IMU and audio data.

Introducing: The Orchestrator

With the sensor layer complete, it’s time to go up the stack. The orchestrator bridges the edge and cloud layers. It will:

• Provide a User Interface (UI) for labeling activity data.

• Store sensor data in an AWS S3-based Data Lake.

• Provide a communication bridge between the sensor layer and the cloud.

The next diagram shows the planned architecture with the orchestrator:

Data Lake Design

A Data Lake is a central place to store all your data, both structured and unstructured, at any scale. It keeps data in its raw form until needed for analysis, offering flexibility and scalability. Learn more about data lakes here.

In this Human Activity Recognition system, a Data Lake will store large amounts of diverse data, like IMU and audio data, from various sensors. This data collection is vital for creating a curated, annotated dataset to train machine learning models for recognizing human activities in multi-household settings. The Data Lake's ability to handle different data types and scale makes it perfect for this system's data needs.

fp-orchestrator-utils Python Package

All services on the sensor layer need the orchestrator's new proto definitions to communicate, and duplicating these definitions on each service repository is inefficient. So, I created a public Python package, fp-orchestrator-utils, to provide utilities for the orchestrator. Unlike the fp-mqtt-broker package, this is more tailored to the project and supports only AWS S3, but the code is designed to be expandable for other use cases and cloud vendors.

Features:

• CLI for downloading, generating, and uploading gRPC protos from S3

• Programmatic S3 data operations with boto3 under the hood

fp-orchestrator-utils proto download
fp-orchestrator-utils proto generate

from fp_orchestrator_utils import S3Service

s3.save("data", "datalake/raw/data.csv")

Further Considerations

• Previously, the RFID service controlled and triggered the sensor layer. This role will be passed to the orchestrator, so further refactorization include:

  • RFID will identify household members via tag IDs.

  • The orchestrator will manage the start/stop logic via UI.

  • All services (IMU, Audio) will send data directly to the orchestrator over gRPC.

• Security is also an important consideration for the future.

  • The MQTT broker's allow_anonymous setting allows unwanted anonymous connections. To ensure privacy, we must block these connections. After testing, we should also implement TLS (Transport Layer Security) and authentication protocols for the MQTT broker.

Next steps

In the next sprint, we will focus on finalizing changes to the sensor layer so that each service can communicate directly with the orchestrator. Additionally, I will set up the data lake to collect enough data to create a curated, annotated, and open-source dataset. This dataset will be used for this project and others related to Human Activity Recognition in multi-household settings.

Wrapping Up

This sprint marked the transition from low-level integration to system-wide coordination. The sensor layer is now modular, testable, and fully interconnected. With MQTT and gRPC in place, the system is ready to scale and support richer functionality like annotation and dataset curation.

Follow the series at typo.hashnode.dev or explore the code on GitHub.

Sprint Focus (June 16th - 29th)

This sprint covered the foundational infrastructure for the sensor layer, focusing on RFID and integration work. The objectives were:

• Build and test an RFID sensor layout.

• Establish unit and end-to-end testing frameworks.

• Connect the Audio and RFID services via gRPC protocol.

Enhancements from Previous Stage

1. The full version of Raspberry Pi OS was replaced with the CLI-based Raspberry Pi OS Lite (64-bit). This version reduces disk space usage by ten times and improves access to volatile memory by reducing the number of processes.

2. The ONNX model and the best_model.pth file created by PyTorch were saved directly using Git version control. However, with each file being about 30MB, this significantly impacted the audio service repository pulls from Git. To improve this, I used Git Large File Storage, which replaces files with text pointers in version control and allows the actual files to be pulled when needed by the Raspberry Pi.

Sensor Layout Plan

The initial task was to plan the main layout of the sensor layer. This layer is responsible for collecting data from the environment for further processing. The HAR system will gather environmental data such as sound, Inertial Measurement Unit (IMU) data, and RFID tag data. The following diagram shows the user flow that will be followed to collect this sensor data for the project's development.

The RFID tag swipe process mimics how a smart home environment works when someone arrives home and uses an RFID tag to gain access. In this sequence diagram, two important changes to the initial assumptions are made:

1. The plan was to use a long-range RFID reader so that when a user enters the kitchen, data collection would begin automatically. However, long-range RFID readers are more expensive and harder to set up. For this stage of the project, I decided to use the low-range, low-cost RFID reader RC522. This reader is also easy to integrate with the General Purpose Input Output (GPIO) pins of the Raspberry Pi 4.

2. General-purpose sensors, like the MPU6050, would be used to gather IMU data. However, using these sensors requires a microcontroller to be powered on to process the data, which can be more inconvenient than simply using a mobile phone. The mobile phone uses an app that sends the IMU data through an HTTP client.

In this entry, I will focus on developing the RFID tag and its logic, which involves the first part of the sequence diagram: from the user to the buzzer.

The breadboard schematic below shows the sensor layout.

List of components

• 1 × 16 × 2 LCD screen.

• 1 x I2C backpack for LCD screen.

• 1 x RFID RC522

• 1 x Active buzzer

• 1 x Red LED

• 1 x Green LED

• 19 x Jumper wires

• 2 × 220 Ω resistors.

The following section explains the development and testing environment used to implement this sensor layer. It's important to note that this environment should be the standard for other project integrations and services.

Development Environment

To interact with the sensors, I will develop a Python package on a Windows machine using Microsoft's WSL (Windows Subsystem for Linux) with an Ubuntu distribution. This setup ensures that the resulting application is tailored to run on a Linux-based system, such as the Raspberry Pi OS Lite. You can find the code in the following repository: https://github.com/RodCaba/fp-rfid-reader-service.

The external dependencies used by the repository are listed in the README.md file.

spidev==3.7
mfrc522==0.0.7
pytest==8.4.1
pytest-mock==3.14.1
RPLCD==1.4.0
smbus2==0.5.0
coverage==7.9.1
pytest-cov==6.2.1

Code Layout

Each sensor interactor in the code includes the following components: (I’m using the LCD I2C Display interactor as an example)

1. A service that is started by external libraries, such as the script used for end-to-end testing of the interactors or any other external services.

2.    from .base import Writer
   
      class LCDService:
        """
        A service class for managing LCD operations.
   
        Attributes:
          writer: An instance of a Writer class that handles LCD writing operations.
        """
        def __init__(self, writer: Writer):
          """
          Initializes the LCDService with a specific Writer instance.
   
          Args:
            writer (Writer): An instance of a Writer class that implements the LCD writing functionality.
          """
          self.writer = writer
   
        def write(self, text: str):
          """
          Writes text to the LCD display.
   
          Args:
            text (str): The text to be displayed on the LCD.
          """
          try:
            self.writer.write(text)
          except Exception as e:
            print(f"Error writing to LCD: {e}")
   
        def clear(self):
          """
          Clears the LCD display.
   
          This method calls the clear method of the Writer instance to clear any text currently displayed.
          """
          try:
            self.writer.clear()
          except Exception as e:
            print(f"Error clearing LCD: {e}")
   

3. An abstract class for the sensor interactor is passed to the service. This abstraction is designed for the dependency inversion principle (part of the SOLID principles) and allows the services to run unit tests on devices other than the intended hardware, like the Raspberry Pi. For example, importing the GPIO module on a device other than a Raspberry Pi would cause a runtime error: from RPi._GPIO import * RuntimeError: This module can only be run on a Raspberry Pi!

    from abc import ABC, abstractmethod
   
    class Writer(ABC):
        """
        Abstract base class for LCD writers.
   
        This class defines the interface that all concrete LCD writer 
        implementations must follow.
        """
        def __init__(
                self,
                i2c_expander="PCF8574",
                address=0x27,
                port=1,
                cols=16,
                rows=2,
                dotsize=8,
            ):
            """
            Initialize the LCD writer.
   
            This method can be overridden by subclasses to perform any necessary
            setup for the LCD display.
            """
            self.i2c_expander = i2c_expander
            self.address = address
            self.port = port
            self.cols = cols
            self.rows = rows
            self.dotsize = dotsize
   
        def __del__(self):
            """
            Clean up resources when the LCD writer is deleted.
   
            The LCD display should be cleared to ensure no residual text
            remains when the writer is no longer in use.
            """
            try:
                self.clear()
            except Exception as e:
                print(f"Error during cleanup: {e}")
   
        @abstractmethod
        def write(self, text: str):
            """
            Write text to the LCD display.
   
            Args:
                text (str): The text to display on the LCD.
   
            Raises:
                Exception: If there's an error writing to the display.
            """
            pass
   
        @abstractmethod
        def clear(self):
            """
            Clear the LCD display.
   
            This method should be called to clear any text currently displayed
            on the LCD.
            Raises:
                Exception: If there's an error clearing the display.
            """
            pass
   

4. One or more concrete implementations of the sensor interaction abstraction. This involves importing external modules and implementing the abstraction functions.

    from ..base import Writer
    from RPLCD.i2c import CharLCD
   
    class CharLCDWriter(Writer):
        """
        Concrete implementation of the Writer interface for character LCD displays.
        """
   
        def __init__(
                self,
                i2c_expander="PCF8574",
                address=0x27,
                port=1,
                cols=16,
                rows=2,
                dotsize=8,
            ):
            super().__init__(
                i2c_expander=i2c_expander,
                address=address,
                port=port,
                cols=cols,
                rows=rows,
                dotsize=dotsize,
            )
            self.lcd = CharLCD(
                i2c_expander=i2c_expander,
                address=address,
                port=port,
                cols=cols,
                rows=rows,
                dotsize=dotsize,
            )
   
        def write(self, text: str):
            """
            Write text to the LCD display.
   
            Args:
                text (str): The text to display on the LCD.
   
            Raises:
                Exception: If there's an error writing to the display.
            """
            try:
                self.lcd.write_string(text)
            except Exception as e:
                raise Exception(f"Error writing to LCD: {e}")
   
        def clear(self):
            """
            Clear the LCD display.
   
            Raises:
                Exception: If there's an error clearing the display.
            """
            try:
                self.lcd.clear()
            except Exception as e:
                raise Exception(f"Error clearing LCD: {e}")
   

In summary, each hardware interaction is abstracted and injected into a service to:

• Avoid hardware, such as RPi.GPIO , errors on non-Pi machines

• Enable full mocking and testing

• Follow Dependency Inversion and Open/Closed principles

Testing Environment

Unit Testing

Under the tests folder, the layout of the src folder is copied so that each application service has its own set of unit tests. This suite includes unit tests for the sensor service and each specific implementation of the sensor interactor.

The project runs unit tests on every push or pull request to the master branch using a GitHub action workflow. This ensures that everything added to the master branch passes the unit test suites.

Furthermore, one of the project's goals is to achieve over 80% statement test coverage in unit tests. The current coverage report from pytest-cov shows a 95% test coverage.

======================================================================= tests coverage =======================================================================
______________________________________________________ coverage: platform linux, python 3.10.12-final-0 ______________________________________________________

Name                                           Stmts   Miss  Cover
------------------------------------------------------------------
src/gpio/gpio_controller.py                       23      0   100%
src/lcd/base.py                                   20      2    90%
src/lcd/implementations/charlcd_writer.py         16      0   100%
src/lcd/lcd_service.py                            14      0   100%
src/reader/base.py                                 8      2    75%
src/reader/implementations/mfrc522_reader.py      16      1    94%
src/reader/reader_service.py                      11      0   100%
------------------------------------------------------------------
TOTAL                                            108      5    95%

Integration Tests

In addition to the unit test setup, the repository includes an integration folder with integration tests. These tests ensure that the integration between services works as expected. They are labeled with an "integration" tag using the pytest framework and, like unit tests, are run in the GitHub action workflow when there is a push or pull request to the master branch.

End to End Tests

Finally, a set of End to End tests was set up for the User to Buzzer interaction. A format was developed, which can be found at this URL. This format describes the test scenarios, execution log, and the software and hardware specifications of the tests. The end-to-end script implements the intended functionality and is run to test the scenarios.

Building the Sensor Layout

LCD Display I2C

The first step is to solder the I2C backpack to the LCD display. Some versions of the LCD display come with the backpack already soldered, which can save you this step (especially if you're not great at soldering like me). Next, I enabled the I2C interface using the raspi-config module and installed the necessary tools with sudo apt-get install i2c-tools python3-smbus.

After connecting the LCD to the Raspberry Pi, you need to detect the I2C bus. Run the following command and take note of the address given:

$ sudo i2cdetect 1

WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-1.
I will probe address range 0x08-0x77.
Continue? [Y/n] Y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- 27 -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

This matrix shows that there is an I2C bus detected at address 0x27. You will need to add this address to the code when starting the LCD service.

RFID Reader.

Once the RFID reader is soldered, I connect it to the GPIOs of the Raspberry Pi, enable the SPI interface using the raspi-config tool and then reboot the device. The MFRC522 Python package includes a SimpleMFRC522 class that makes it easier to initialize and communicate with the reader. However, there is a known compatibility issue with newer RFID tags that do not support authentication protocols. When you run the reader function and detect a new RFID tag (like an NTAG215), the following runtime error will occur:

"AUTH ERROR!!
AUTH ERROR(status2reg & 0x08) != 0"

To solve this, you need to use the MFRC522 class instead. You can find more details about the issue on this GitHub page, where the solution was sourced: https://github.com/pimylifeup/MFRC522-python/issues/31

gRPC Services Connection

To establish a connection between both services, I decided to use the gRPC protocol. This choice was driven by the need for enhanced performance and the compatibility with the solution's microservices architecture. The gRPC protocol is well-suited for this task because it allows for efficient communication between distributed systems. In this setup, the audio service will be made accessible, and the RFID service will send a request to initiate the audio recognition loop whenever an RFID tag is detected.

The initial step in this process involves defining the Protocol Buffer, which serves as the interface definition language for the service. This definition will specify the structure of the data and the methods that the services will use to communicate.

syntax = "proto3";

package audio_service;

// Audio processing service
service AudioService {
    // Start audio recording and processing
    rpc StartAudioProcessing(AudioRequest) returns (AudioResponse);

    // Get the status of audio processing
    rpc GetProcessingStatus(StatusRequest) returns (StatusResponse);

    // Health check
    rpc HealthCheck(HealthCheckRequest) returns (HealthCheckResponse);
}

// Request message for audio processing
message AudioRequest {
    string session_id = 1;           // Unique session identifier
    int32 recording_duration = 2;    // Duration in seconds
    string output_format = 3;        // Output format (wav, mp3, etc.)
}

In this last extract of the definition file, I'm defining the Protocol Buffer to serialize the data that the service will request and return as a response. I can also set up a set of functions that the RFID service can call. Using the grpc_tools Python package, I've compiled the Protocol Buffer definition into Python classes to be used in the code.

from src.grpc_generated import audio_service_pb2, audio_service_pb2_grpc
from src.predictor.predict import AudioPredictor


class AudioService(audio_service_pb2_grpc.AudioServiceServicer):
    def __init__(self):
        # Initialize the predictor
        model_path = os.path.join(
                                  "exported_models", "model.onnx")
        print(f"Loading model from: {model_path}")
        self.predictor = AudioPredictor(model_path, feature_type="melspectrogram")

       ...

    def StartAudioProcessing(self, request, context):
        """Start audio recording and processing"""
        session_id = request.session_id or str(uuid.uuid4())

        # Process audio code with AudioPredictor...

        return audio_service_pb2.AudioResponse(
                session_id=session_id,
                success=True,
                predicted_class=predicted_class,
                confidence=float(confidence),
                top_predictions=top_predictions
            )

Each function defined in the Protocol Buffer must be implemented in the class that extends the AudioServiceServicer class. These functions should accept and return the expected Protocol Buffer types.

The same Protocol Buffer definition and gRPC auto-generated Python classes are used in the RFID service. A client is set up to connect and call the functions to communicate with the audio service.

class AudioServiceClient:
    """gRPC client for Audio Service"""

    def __init__(self, server_address: str = None, timeout: int = 30):
        # Use environment variable or default to Docker service name
        if server_address is None:
            server_address = os.environ.get('AUDIO_SERVICE_URL', 'localhost:50051')

        self._connect()

    def _connect(self):
        """Establish connection to audio service"""
            self.logger.info(f"Attempting to connect to audio service at {self.server_address}")
            self.channel = grpc.insecure_channel(self.server_address)
            self.stub = audio_service_pb2_grpc.AudioServiceStub(self.channel)

    def start_audio_processing(self, duration: int = 5, session_id: Optional[str] = None) -> Optional[Dict]:
        """
        Start audio recording and processing
        """
            if session_id is None:
                session_id = str(uuid.uuid4())

            request = audio_service_pb2.AudioRequest(
                session_id=session_id,
                recording_duration=duration,
                output_format="wav"
            )

            response = self.stub.StartAudioProcessing(request, timeout=self.timeout)

Final Integrated Result

The RFID service will wait for the RFID swipe. Once this happens, it will create a new thread using the threading package. This thread requests the Audio service to process audio through gRPC until the RFID tag is swiped again. These separate threads are created to ensure that RFID tag swipe detection is not blocked by the main thread.

The following video demonstrates the integrated result. You can find the end-to-end testing execution in the End to end testing format.

As the test execution log explains, the RFID swipe stress test is failing. Swiping the RFID reader repeatedly creates new threads and causes overlaps in predictions. Ideally, the audio processing thread should finish before starting a new one.

Known Limitations

1. The plan was to use Docker containers for the services, with the Raspberry Pi managing the initialization of these containers. However, because containers operate separately from the Raspberry Pi environment, setting up the audio hardware and GPIO pins in the containers proved to be complicated. As a result, the idea was abandoned in favor of running the Python scripts and starting the servers on the local machine.

What’s Next?

1. To enhance the quality and maintainability of the codebase, the next step involves implementing a linting tool integrated with a GitHub Actions (GHA) workflow.

   1. This tool will automatically check the code for adherence to coding standards, ensuring consistency and promoting best practices across the entire project.

2. Begin by implementing the IMU (Inertial Measurement Unit) Service, which is designed to capture detailed IMU data from the mobile device. This service will be responsible for collecting various types of sensor data, including accelerometer, gyroscope, and other readings.

3. As shown by E2E testing, we need a stronger RFID swipe logic to pass the stress execution tests.

Wrapping Up

This sprint marked a pivotal step in bridging the hardware and software layers of the HAR system. From soldering and assembling the sensor layout to implementing abstraction layers and wiring services with gRPC, the project is now equipped with a solid, extensible foundation. The successful integration between RFID and audio modules shows that even in constrained environments, it's possible to build smart, responsive systems with strong architectural principles.

The upcoming sprints will push this foundation further—by incorporating mobile-based IMU data, refining audio recognition, and reinforcing system performance under load.

Thanks for following along. Feel free to explore the repositories, share feedback, or connect if you're navigating similar challenges in edge AI, IoT, or smart environments.

📬 Follow this series on typo.hashnode.dev to see how this HAR system evolves from prototype to production-ready.

As part of my Computer Science dissertation, I’m developing a multi-resident, multimodal Human Activity Recognition (HAR) system tailored for privacy-sensitive environments like shared kitchens. This system is designed to operate on resource-constrained edge devices, using a mix of low-resolution audio, IMU sensors, and RFID tags.

You can find a more detailed literature review and the motivation behind the project at this link.

In this first installment of the series, I’ll walk you through the prototype implementation of one of the most challenging features: audio-based activity recognition on a Raspberry Pi.

Prototype Architecture

The feature prototype focuses on processing environmental audio to predict activity using a CNN model trained on the Kitchen20 dataset. Kitchen20 is a rich dataset for environmental audio in kitchen settings.

Key components:

• Raspberry Pi 4 Model B (1GB RAM)

  • Although this model of the Raspberry Pi can come in higher versions of RAM, I decided to use the lowest version possible so the project is aware and built-upon a resource-constrained device to aim for efficiency and cost-effective solutions on HAR.

• USB omnidirectional microphone

• ONNX exported model

• Google TTS for audio output

Model Training with Kitchen20

The original PyTorch implementation of Kitchen20 relied on outdated torchaudio APIs. This is evident in the following extraction of the code:

audio_set = Kitchen20(
        root='/media/data/dataest/kitchen20/',
        folds=[1, 2, 3, 4],
        transforms=transforms.Compose([ # transforms.Compose is no longer available in torchaudio
            transforms.RandomStretch(1.25),
            transforms.Scale(2 ** 16 / 2),
            transforms.Pad(input_length // 2),
            transforms.RandomCrop(input_length),
            transforms.RandomOpposite()]),
        overwrite=False,
        use_bc_learning=False,
        audio_rate=audio_rate)

    audio_loader = DataLoader(audio_set, batch_size=2,
                              shuffle=True, num_workers=4)

The use of the Compose function for audio transformations was available in the very first release version of torchaudio, which was removed as a breaking change on the next release of torchaudio, version 0.3.0

I re-implemented the dataset accessor using modern torch and torchaudio versions, which you can find on the Github public repository:🔗 Kitchen20 PyTorch Accessor (Updated)

I trained a four-layer CNN, then exported the model to ONNX format for lightweight inference on the edge device. The current results are:

• Approximately 30% accuracy on the training data: The current model achieves around 30% accuracy when evaluated on the training dataset. This indicates that while the model is learning to some extent, there is significant room for improvement. The relatively low accuracy suggests that the model may not be capturing all the necessary patterns in the data effectively.

• Plans for improvement include integrating IMU and RFID modalities, as well as refining preprocessing: To enhance the model's performance, we plan to incorporate additional data sources, such as IMU (Inertial Measurement Unit) and RFID (Radio Frequency Identification) modalities. These additional data streams can provide more context and features for the model to learn from, potentially leading to better accuracy.

Preprocessing Pipeline

The preprocessing pipeline of audio includes:

1. Sample rate adjustment

    def preprocess_audio(
          waveform,
          original_sample_rate,
          target_sample_rate=16000,
          target_length=4,
    ):
        """
        Preprocess the audio waveform to have a consistent length and sample rate.
   
        Args:
            waveform (Tensor): The audio waveform.
            original_sample_rate (int): The original sample rate of the audio.
            target_sample_rate (int, optional): The target sample rate. Defaults to 16000.
            target_length (int, optional): Target number of samples.
   
        Returns:
            Tensor: The preprocessed audio waveform.
        """
        # Convert to mono if stereo
        if waveform.size(0) > 1:
            waveform = torch.mean(waveform, dim=0, keepdim=True)
   
        # Resample the audio if the sample rate is different
        if original_sample_rate != target_sample_rate:
            waveform = torchaudio.transforms.Resample(
                orig_freq=original_sample_rate,
                new_freq=target_sample_rate
            )(waveform)
   
        # Adjust length
        current_length = waveform.shape[1]
        # Trim or pad the waveform to the target length
        if current_length > target_length:
            waveform = waveform[:, :target_length]
        else:
            waveform = torch.nn.functional.pad(waveform, (0, target_length - current_length))
   
        return waveform
   

2. Feature extraction using Pytorch (as opposed to Moreaux, 2019 work that used Librosa library):

   • MFCC

   • MelSpectrogram

if feature_type == 'melspectrogram':
            self.transform = torchaudio.transforms.MelSpectrogram(
                sample_rate=sample_rate,
                n_fft=n_fft,
                hop_length=hop_length,
                n_mels=n_mels,
                f_min=f_min,
                f_max=f_max,
            )
            self.db_transform = torchaudio.transforms.AmplitudeToDB()
elif feature_type == 'mfcc':
            self.transform = torchaudio.transforms.MFCC(
                sample_rate=sample_rate,
                n_mfcc=n_mfcc,
                melkwargs={
                    'n_fft': n_fft,
                    'hop_length': hop_length,
                    'n_mels': n_mels,
                    'f_min': f_min,
                    'f_max': f_max,
                }
            )
            self.db_transform = None

This preprocessing service is shared between the cloud (for training) and the edge (for prediction) layers to ensure consistent input formats. Some of the work that is considered to improve the preprocessing pipeline includes:

• Refactor the existing codebase to function as a standalone service. This involves decoupling the preprocessing logic from the main application, allowing it to operate independently.

• Extend the current preprocessing capabilities by adding support for IMU and RFID data. This will involve developing new transformation pipelines tailored to the specific characteristics of IMU and RFID data.

• Conduct a thorough parameter tuning process to optimize the system's performance. This is probably the most important improvement needed during the project's development phase. Due to time limits, the feature prototype was created quickly to show that the ONNX model could work with the edge device and kitchen environment, without focusing much on the preprocessing service details. The first goal was to copy the work of Moreaux et al. (2019) with minimal changes to make it run on modern libraries, mainly fixing compile and runtime errors. I believe this is why the model's accuracy and confidence are low. I need to explore and find the best audio preprocessing practices for CNN to greatly improve this key part of the project.

Edge Device Inference Loop

The Raspberry Pi (edge device layer) performs the following in a loop:

1. Records 5 seconds of audio

2. Uses the shared preprocessing pipeline

3. Feeds features to the ONNX model using onnxruntime

4. Outputs:

   • Prediction label

   • Confidence score

5. Plays the outputs using an audible feedback using gTTS + playsound

The following short video, showcase the result of the prototype.

This setup confirms feasibility of real-time HAR prediction on low-power hardware, crucial for assisted living scenarios where privacy and efficiency are essential. The improvements to be made on the edge device layer includes:

• Replace the full operating system with a lightweight Linux distribution like Alpine Linux or Raspberry Pi OS Lite. This change will use fewer system resources, speed up boot times, and improve overall performance by cutting down on unnecessary background processes and services.

• Automate updating the model from the cloud using SSH and ONNX replacement. This will keep the device running the latest model version, improving prediction accuracy and reliability. By securely connecting to the cloud, updates can happen automatically without needing manual work, reducing downtime and maintenance efforts.

Conclusion

Developing a privacy-focused Human Activity Recognition (HAR) system on limited-resource edge devices is a promising way to improve privacy and efficiency in shared spaces. By using low-resolution audio, IMU sensors, and RFID tags, this system aims to recognize activities accurately while keeping user privacy intact. The prototype on a Raspberry Pi shows that real-time HAR prediction is possible on low-power hardware. Although the current model's accuracy is moderate, ongoing improvements, like adding more data types and improving preprocessing, should boost performance. Future work will focus on optimizing system design, organizing datasets, and automatic deployments on edge devices to make the system practical and usable in real-world situations. This project aims as a step toward creating energy-efficient and privacy-aware HAR solutions for edge devices.

Code & Resources

• 🔗 GitHub Repo: fp-audio-service - Modern Pytorch accessor for the Kitchen20 dataset.

• 🔗 Kitchen20 Dataset - Moreaux et al. (2019) original Kitchen20 implementation.

• 🧠 Literature Review and project motivations

Follow the Series

This post is part of a series documenting my dissertation project on Energy-Efficient and Privacy-Aware Multimodal HAR for Edge Devices. Future posts will cover:

1. System Design & Architecture: Tiers, sensors, and design principles

2. Dataset Structuring: From Kitchen20 to custom multi-resident datasets

3. Edge Device Deployment: Optimization, updates, and latency analysis

4. Final Evaluation & Results: Precision, recall, and real-world usability

Subscribe to typo.hashnode.dev and follow along!

In my first article, I want to explore a Computer Science topic: the heap data structure.

These kinds of theoretical topics are pretty common in technical interviews, and although I have never used them in my day-to-day job, the applications of these data structures are exciting and worth studying to improve your problem-solving skills.

In this article, I will explain what a heap is, its uses and how to implement it in C++.

What is a heap?

A heap is a tree data structure that satisfies 2 properties:

• It is a complete binary tree

• The value of a node is greater (max heap) or smaller (min-heap) than or equal to the value of its children

An example of a max heap would be the following:

The example above is both:

• A complete binary tree:

  • All levels are filled except possibly the last level.

  • The last level has all keys as left as possible.

• A max heap:

  • The value of a node is greater than or equal to the value of its children.

Uses of a heap

A useful property of the heap is that the root node is always the tree's maximum (max heap) or minimum (min-heap) value.

This property becomes handy to implement a priority queue, where the root node is always the next element to be processed.

Priority queues are useful in many applications, for example:

• A CPU scheduler, where the tasks with the highest priority are processed first.
• A web server, where the requests with the highest priority are processed first.

Another use of the heap is to implement a sorting algorithm. The idea is to insert all the elements in the heap and then extract them individually. The result will be a sorted ascending list (min-heap) or descending list (max heap).

Heap Representation

The most common way to represent a heap is through an array.

Thanks to the properties of a complete binary tree, the root node is the first element of the array, and its children are the second and third elements. Then, the children of the second element are the fourth and fifth elements, and so on.

We get the children of a node with the following formulas, assuming that the array is 0-indexed:

$$left(i) = (2 * i) + 1$$

$$right(i) = (2 * i) + 2$$

To get the parent of a node we use the following formula:

$$parent(i) = FLOOR(\frac{i-1}{2})$$

Implementation

We will assume a max heap implementation in this article. Let's imagine we are constructing a priority queue for an application that processes tasks. Each task has a priority and we want to process the tasks with the highest priority first.

This will be our Heap class structure:

// MaxHeap.h
#inclde <string>
#include <vector>
#include <cmath>

struct Task {
    int priority;
    std::string id;
};

class MaxHeap {
  public:
    MaxHeap(){
        heap = std::vector<Task>();
    }
    ~MaxHeap();
    void insert(Task task);
    bool isEmpty();
    Task extractMax();
    static int getLeft(int i);
    static int getRight(int i);
    static int getParent(int i);
  private:
    std::vector<Task> heap;
    void swap(int i, int j);
    unsigned int highestIndex(unsigned int i);
    void heapify(unsigned int i);
}

In the above code, we have a Task struct containing the task's priority and id.

The MinHeap class consists of the following:

• 3 public methods that allow other classes to insert a task, extract the task with the highest priority and check if the heap is empty.

• 3 static methods, that allow to get the children and parent of a node.

  • These methods are static because they don't need to access any state of the Heap class, they just need the index of the node.

• A private vector of tasks will be used to store the tasks in the heap.

  • This vector is initialized in the constructor of the class.

• 3 private methods, that the public methods will use to insert and extract tasks from the heap.

We also import the vector, string and cmath libraries. The vector library will be used to store the tasks in the heap, the string library to store the ID of the task and the cmath library to use the floor function.

Let's start the implementation of static methods to get the children and parent of a node, this will be pretty straightforward, as we just need to apply the formulas that we saw before:

// MaxHeap.cpp
#include "MaxHeap.h"

int MaxHeap::getLeft(int i) {
    return (2 * i) + 1;
}

int MaxHeap::getRight(int i) {
    return (2 * i) + 2;
}

int MaxHeap::getParent(int i) {
    return floor((i - 1) / 2);
}

Now, let's take a look at the private functions that will be used for the insertion and extraction of the tasks:

// MaxHeap.cpp

void MaxHeap::swap(int i, int j) {
    Task temp = heap[i];
    heap[i] = heap[j];
    heap[j] = temp;
}

unsigned int MaxHeap::highestIndex(unsigned int i) {
    unsigned int highest = i;
    unsigned int left = MinHeap::getLeft(i);

    // No left child, hence no right child, return parent
    if (left >= heap.size()) {
        return highest;
    }

    if (heap[left].priority > heap[highest].priority) {
        highest = left;
    }

    unsigned int right = MinHeap::getRight(i);

    if (right < heap.size() && heap[right].priority > heap[highest].priority) {
        highest = right;
    }

    return highest;
}

void MaxHeap::heapify(unsigned int i) {
    unsigned int highest = highestIndex(i);
    if (highest != i) {
        swap(i, highest);
        heapify(highest);
    }
}

Let's go through the code above:

The swap function swaps the position of two elements in the heap, given their indexes.

The highestIndex function returns the index of the highest priority element between the node with index i and its children. If the node with index i has no children, it returns the index of the node with index i.

Finally, the heapify function takes a node with an index i and swaps it with its highest-priority child if the child has a higher priority than the node. It then calls itself recursively with the index of the child, this will ensure that the heap property is satisfied from the node with index i all the way to the leaf nodes.

Believe it or not, that's as complicated as it gets. Now that we have the static methods and the private methods, we can implement the public methods that will be used to insert and extract tasks from the heap.

// MaxHeap.cpp

void MaxHeap::insert(Task task){
  unsigned int pos = heap.size();
  heap.push_back(task);
  while (pos > 0 && heap[MaxHeap::getParent(pos)].priority < heap[pos].priority) {
    swap(pos, MaxHeap::getParent(pos));
    pos = MaxHeap::getParent(pos);
  }
}

bool MaxHeap::isEmpty() {
  return heap.size() == 0;
}

Task MaxHeap::extractMax() {
  if (isEmpty()) {
    throw "Heap is empty";
  }
  Task max{ heap[0].priority, heap[0].id };
  heap.erase(heap.begin());

  if (isEmpty()) return max; 

  heapify(0);
  return max;
}

The insert method takes a task and inserts it in the heap. It first inserts the task at the end of the heap and then swaps it with its parent until the heap property is satisfied.

The isEmpty method returns true if the heap is empty, and false otherwise.

The extractMax method returns the task with the highest priority and removes it from the heap. It first checks if the heap is empty, and if it is, it throws an exception. Then it stores the task with the highest priority in a variable, removes it from the heap and calls the heapify method with the root node.

Now that we have implemented the heap, we can use it in our application:

// main.cpp

#include <iostream>
#include "MaxHeap.h"

int main() {
    MaxHeap heap = MaxHeap();
    heap.insert(Task{ 1, "Task 1" });
    heap.insert(Task{ 2, "Task 2" });
    heap.insert(Task{ 3, "Task 3" });

    while (!heap.isEmpty()) {
        Task task = heap.extractMax();
        std::cout << "Task with id " << task.id << " and priority " << task.priority << " extracted" << std::endl;
    }

    // Output:
    // Task with id Task 3 and priority 3 extracted
    // Task with id Task 2 and priority 2 extracted
    // Task with id Task 1 and priority 1 extracted

    return 0;
}

Conclusion

In this article, we have seen what a heap is, its uses and how to implement a max heap in C++.

As I stated, these topics allow you to improve your problem-solving skills. It allows you to think about simple solutions to complex problems, and it is a good way to practice your programming skills.

I hope you enjoyed this article, and if you have any questions or suggestions, please let me know in the comments below.