Mixed TechnologiesCompletedIntense Violet theme

Interactive Piano HCI

A multidisciplinary piano-learning system combining connected hardware, a web interface and an augmented-reality learning experience. The project explores ESP32 communication, Angular/TypeScript, Unity, Vuforia and cloud-assisted recognition.

C++ESP32AngularTypeScriptUnityVuforiaAzure
IPInteractive Piano HCI
C++Main technology
1505Tracked files*
27.cs files
19.json files

*File metrics count tracked files after excluding common generated build folders. The project status is an editable draft classification.

01 · What it does

What the application does

Capabilities evidenced by project documentation and source structure; some behaviours require platform configuration.

01

ESP32-linked musical input and feedback.

ESP32-linked musical input and feedback.

02

Angular/TypeScript application for interactive pract

Angular/TypeScript application for interactive practice.

03

Unity/Vuforia augmented-reality training components.

Unity/Vuforia augmented-reality training components.

04

Exploratory music recognition and connected services

Exploratory music recognition and connected services.

02 · Technical structure

How the system works

Technical architecture drawn from the repository and supplied documentation.

01

The project uses C++, ESP32, Angular, TypeScript as its documented technical foundation.

02

The source tree and supplied README describe the primary modules; individual files and runtime behaviour may vary by platform.

04 · Source and design

Implementation examples and code walkthrough

Live ESP32 serial input driving a Unity AR piano

Piano-AR/Assets/Scripts/EspSerialReader.csVerified GitHub source excerpt (abridged)
void Update() {
    if (serial == null || !serial.IsOpen || piano == null) return;
    try {
        while (serial.BytesToRead > 0) {
            string raw = serial.ReadLine();
            string note = Normalize(raw);
            if (string.IsNullOrEmpty(note)) continue;
            piano.SimulatePressByName(note);
        }
    } catch (TimeoutException) { }
}

Azure assistant request lifecycle

C#

The cloud assistant creates a thread, submits the user message, launches a run and polls for a finished response. Secret validation occurs before the client is created.

Piano-Azure/Azure_API/ai-assistant-api/src/HCI.AIAssistant.API/Services/AIAssistantService.csSelected source excerpt
AssistantThread thread = await _assistantsClient.CreateThreadAsync();
await _assistantsClient.CreateMessageAsync(thread.Id, MessageRole.User, message);
ThreadRun run = await _assistantsClient.CreateRunAsync(thread.Id, new CreateRunOptions(_id));
do {
    run = await _assistantsClient.GetRunAsync(thread.Id, run.Id);
    await Task.Delay(TimeSpan.FromMilliseconds(_DELAY_IN_MS));
} while (run.Status == RunStatus.Queued || run.Status == RunStatus.InProgress);
06 · Technology

Programming languages & stack

C++
ESP32
Angular
TypeScript
Unity
Vuforia
Azure

Technology overview: C++, ESP32, Angular, TypeScript, Unity, Vuforia, Azure. The list reflects documented tools, dependencies and proposed components; for planned projects it does not prove implementation.

File types identified: 27 .cs, 19 .json, 10 .ts, 10 .c

07 · Reality check

Status, evidence & limitations

The project status is based on repository sources and supplied documentation. It may be updated as the project evolves. A published case study is not a certification of production readiness.
  • Features are based on the supplied README and code inventory; no end-to-end runtime certification is implied.
08 · Resources

Project resources

Explore the public repository, source code and project documentation on GitHub.