Color & Math Game (Ionic)

Background

Mild Cognitive Impairment (MCI) is a condition that causes noticeable problems with memory, thinking, and reasoning, but not severe enough to significantly interfere with daily life or independent function. People with MCI often experience difficulties such as forgetting appointments, losing track of conversations, or struggling with decision-making. While these symptoms are less severe than those seen in dementia, they can still reduce quality of life, create frustration, and increase reliance on family members or caregivers. Importantly, individuals with MCI remain aware of many of their challenges, which can also contribute to feelings of anxiety, stress, and social withdrawal.

As the global population ages, the number of people affected by MCI is expected to grow. This has created an urgent need for supportive tools and interventions that can help people maintain independence, reduce caregiver burden, and improve overall well-being. One promising approach is the use of socially assistive robots that provide not only practical reminders and guidance but also emotional support and companionship.

During my internship with the PHAST Lab at SFSU, our lab is developing Carmen, a robot designed specifically to assist people with MCI in their daily lives. I helped to build the robot hardware and translated a set of cognitive games from an iOS app into a responsive web app. I used Ionic (Angular, HTML, CSS, TypeScript) to make the games more accessible across devices. Furthermore, we want to have the web app send and receive messages from the robot. For example: If the users input the correct answer, the robot will make some movement or show some colors that tell them they did it correctly.

Our final goal is to make a Carmen robot that aims to support memory through reminders for medications and appointments, guide users through everyday tasks, and provide conversational interaction to reduce loneliness. By combining assistive functions with social engagement, Carmen has the potential to enhance the autonomy and confidence of people living with MCI, while offering caregivers reassurance that their loved ones are supported.

What I Built

• Math Game — I created a numeric input game with a large on-screen keypad and clear prompts.
• Color Memory Game — I developed a sequence-recall game where users remember and select colors in the correct order.
• Toggle Feature — I added a switcher so users can move easily between games.

Why It Matters

• Encourages engagement through play while supporting cognitive training goals.
• Strengthens memory, attention, and visual sequencing—aligned with home-based interventions.
• Accessibility-first UI with large, clear components and responsive layouts.

My Contributions & Challenges

After I finished building the game, I pushed my code to GitHub so I could pull it onto Carmen, our robot platform, for testing. This testing phase helped me adjust the scaling of the text prompts, number pad, and response pop-up. I intentionally made these UI elements larger to accommodate users with cognitive impairments, ensuring that everything is easy to read and interact with.

When I did this work, there were a lot of problems setting up the environment and managing GitHub. My mentor, Mathew, and another teammate named Daniel worked together with me to solve those issues.

One of the challenges I encountered was configuring the correct scaling and resolution when deploying to Carmen. The Raspberry Pi didn’t automatically detect the default resolution of our tablet display. To fix this, we had to manually set the HDMI output through the Pi terminal. This step was important; without it, applications would overflow the screen and become unreadable.

Another issue occurred when updating our code on GitHub. I didn’t realize at the time that I needed to pull the latest changes before pushing my updates. As a result, I encountered a merge conflict while backing up and pushing the code, which ultimately required us to reinstall everything from scratch.

During this internship, I learned how to collaborate and communicate effectively with my teammates, mentor, and professor. It also helped me improve my self-study skills, problem-solving abilities, and gain more hands-on experience with coding.

Screens & UI

Next Steps

• Currently, the web app runs on the Raspberry Pi but there is no connection between the web app and the robot software. I have built cognitive games, and the next step is to enable the web app to send and receive messages from the robot. For example: the robot could respond dynamically to a user’s answer in the math or color game. Additionally, I also attempted to add a login page to enhance the system’s security.
• Add more levels and variety to the color game.
• Continue improving cross-device compatibility and performance.
• Work on deeper integration with robotic therapy (CARMEN) workflows.