Personal Recommendation System Based On Emotion Recognition

- National Health Insurance Administration, MOHW

Scientific research highlights the health benefits of regular fruit consumption, particularly in reducing chronic disease risks due to the antioxidants and vitamins in fruits. However, the belief that vegetables can replace fruits is incorrect, as fruits offer distinct nutritional advantages. Addressing the lack of tools to monitor fruit intake, we propose a facial recognition-based recommendation system. This technology-driven app will analyze user data to offer personalized fruit consumption advice, aimed at reducing sugar intake and fostering healthy eating habits. By leveraging facial recognition, the system ensures precise and tailored recommendations, filling a critical gap in the current health tech landscape.

July 2021 - Jan 2022

In this project, I mainly worked as a Back end Developer and also responsible for doing preliminary research to validate prototypes. I used the research data and analyzed their needs to design our function of system and the structure.

Driven by a keen interest in machine learning (ML), cloud computing, and contemporary consumer preferences for sugary beverages, I spearheaded the development of a personalized recommendation system utilizing emotion recognition on Microsoft Azure. My team leveraged open data on fruit nutrition to build this system, deploying it as a React Native app via Azure DevOps. The resulting application offers users personalized fruit tea recommendations based on their emotional state, inferred from a selfie, along with detailed guides on fruit tea preparation.

The absence of a centralized platform to store and manage detailed nutritional information for various fruits posed significant challenges in aggregating, accessing, and utilizing the data required for the system. Thus, we were compelled to visit multiple websites to collect, store, and integrate this information, which added complexity to the development process.

The limited accuracy of our image recognition models and the complexities involved in adapting the deep learning models to Azure Custom Vision. To effectively address these obstacles and increase our flexibility in responding to unforeseen changes, we implemented Scrum methodologies. Moreover, the ongoing GPU shortage necessitated a swift adaptation to cloud computing services, prompting us to engage in thorough research, seek insights on platforms like Stack Overflow, and collaborate with industry experts. These efforts were crucial in navigating the technical hurdles we faced.

In this project, we utilized a variety of advanced tools and platforms to achieve our goals. The Line Bot SDK was employed in Visual Studio as our primary back-end development tool, enabling us to write code that effectively responds to user input. For deployment, we leveraged Azure Web Services to host our Line Chatbot in the cloud, seamlessly connecting it to Azure's infrastructure. Through Line Developers, we were able to upload and manage our custom-developed chatbots in the cloud, allowing for easy integration with Line's official platform.

On the front-end, we used React Native as the development framework for the Tang system. This framework was crucial in incorporating various functional packages, such as react-native-image-picker and react-native-stack, to enhance the app's capabilities. To handle data storage, Azure Cosmos DB served as our back-end database, where we stored both the nutritional composition of fruits and users' personal information. The nutritional data was meticulously gathered through web scraping from reputable sources, including the Food Nutritional Composition Database by the Ministry of Health and Welfare (MOHW) and the Food Ingredient Registration Platform of the Taipei City Government (TMG).

For food and drink identification, Azure Custom Vision was employed to analyze items, assess sugar content, and calculate essential nutrients like calories, vitamins, and minerals. Azure Cognitive Services played a critical role in facial recognition, allowing us to analyze users' emotional intensity and assess their physical and mental conditions, which we then factored into our fruit recommendation system.

To streamline development and deployment, we utilized Azure DevOps to deploy our React Native apps to Azure, while Azure App Center was instrumental in opening multiple simulators and enabling rapid deployment and testing. Additionally, the Google Maps Platform and Google Place Platform were integrated to provide users with location-based services, including recommendations for nearby stores where they could purchase fruits.

This comprehensive approach ensured that our system was robust, user-centric, and capable of delivering accurate and personalized recommendations.

Through image recognition technology, a more convenient food recording process is designed to allow users to develop the habit of food recording. You can also know the approximate sweetness of the fruit before eating it.

Personalized diet recommendations allow users to enjoy delicious food, improve their mood, and achieve goals while focusing on a healthy diet to nutritional balance.

Nutrient intake record statistics create a bar chart, and users can immediately check the nutritional intake status.

Combined with Google Maps, users can obtain the best stored information and real-time navigation to achieve the ultimate goal of healthy eating mark.

From a government perspective, since the system uses Line as an interactive platform, the government can use it to promote healthy eating concepts and save labor costs.

From the perspective of local businesses, marketing advertising can be carried out through the system to promote business information.

Provide more channels for purchasing fruits through the delivery platform and merchants, and provide users with consumption feedback, which increases users’ stickiness and brings further business opportunities to merchants. Furthermore, Integrate local tourism to promote local fruits and assist farmers in selling fruits to various parts of Taiwan.