Enhancing Pharmacovigilance with Optical Character Recognition

The project achieved prominent recognition and was featured on the Ministry of Science and Technology.

With the rapid development of information, technological information is also widely used in medical care and management. However, most of the medical-related application nowadays requires users to set the initial data through the mobile phone keyboard that lead to be a crop of questions, for the elderly not only don’t know how to operate the mobile cellphone, but also have difficulties in using it due to illiteracy or font size.

This group of people tends to forget to take medicine because no one reminds them. In some areas in Taiwan, illiterate elderly people account for more than 30% of the local population (Statistics Office of the Ministry of the Interior, 2021). With such a high proportion, it is impossible to use traditional application software to assist medication and reduce medical burden. Furthermore, with the increasing demand for medical care for the elderly, it appears to the importance of health care and long-term care. 

This research project is based on the "Active Aging" policy proposed by the World Health Organization (WHO) to address global aging.

In recent years, the population in Taiwan has a negative growth trend and the population structure has entered an aging society in 2018. According to the WHO, when the population over the age of 65 accounts for 14% of the population in a region, it means that the region has entered an aging society (World Health Organization, 2010). Moreover, based on the Ministry of the Interior Statistical Bulletin in Taiwan, the elderly population (over 65 years old) had reached 3.804 million, accounting for 16.2% of the total population by the end of January in 2010 (Ministry of Interior Statistical Bulletin, 2021). This situation highlights the needs of the elderly in terms of family care, physical and mental health. It also puts a huge pressure on medical care, which shows the importance of medical care and long-term care issues. 

Aging Society in Taiwan 

The Seniors' medication problems

Optical Character Recognition, OCR

Image recognition of drugs

Personalized healthy database

Chatbot2.7 Text-to-Speech

The importance of family support 

System development architecture and tools

Personalized medicine label

Data collection and classification

Processing the government open data

Text-to-speech

Personal information: When users enter the system in the first time, they need to fill in their personal information. They can use the voice function to input, change and delete data and also use voice to confirm the information.

Drug record: After taking a picture of the prescriptions, the system will identify the name of the medicine that also connect to the database of the Ministry of Health and Welfare Food and Drug Administration. When it compares the name of the medicine in the system database, the information will be automatically stored in the system database with allow the users and user’s family members to manage if they need in the future.

Medication Assistant: When it is close to the time that the user needs to take medicine, the Chatbot in the system will actively send a message to remind the user to take medicine. It will also require users to take photos of medicine that they take and guide them to take the medicine, which can prevents them from taking the wrong medicine. If the user is inconvenient to take medicine immediately, he can choose to delay the reminder. After taking the medicine, the Chatbot will play the message from the family in the form of voice - if the text message can use the voice playback function - the user can also use the voice recording to send message to the family. If the user feels unwell after taking medicine, these records will be automatically stored in the "Health Report”.

Information of return appointment: Users are able to use voice to inquire about return visit information, and the "Medication Assistant" will also remind users before seeing a doctor.

Clinic record: After recording the medical staff’s diagnoses during the consultation period, they information is stored in the system for playback and query. This record offers users to listen to the consultation records at any time after returning to the clinic, which solves the problem of memory loss and forgetfulness of the elderly instructions from medical personnel.

Health Report: When users take medicine or they feel bad, the system will let their family members know immediately by posting. Their family members have the ability to encourage and care through voice and message. This record also allows doctors to know the user's usual condition during the return visit.

In this project, Xcode was employed as the integrated development environment, playing a central role in the development of system programs. For deployment and resource management, we utilized Amazon Lambda, a key component of AWS, which facilitated the seamless delivery and management of system resources. Additionally, Amazon Transcribe, another AWS service, provided automatic speech recognition (ASR). This technology leveraged machine learning models to convert audio into text, enabling elderly users to view and transmit messages with ease. The system also allowed for voice playback, helping users understand text displayed on the screen.

To enhance the Chatbot's capabilities, I integrated Google’s Dialogflow, which provided natural language processing (NLP) to analyze dialogues and generate appropriate responses. The system further employed Jieba for text segmentation, particularly in analyzing conversations with elderly users. This allowed the system to provide more accurate responses by better understanding the text inputs.

On the back-end, MongoDB was utilized for data management, handling operations such as adding, querying, modifying, and storing a variety of data, including personal information, medication records, and images. The data on medication ingredients was sourced from government platforms and the Ministry of Health and Welfare (MOHW) through web scraping techniques. For image recognition tasks, Word2Vec was applied to process images of medication bills and follow-up visit sheets, converting them into digital text for storage in the database.

Finally, to assess the effectiveness of the medical management system, I conducted a questionnaire survey among senior users after the system's development was complete. This survey aimed to evaluate the system's usability, usefulness, user attitudes, and willingness to adopt the system, providing valuable feedback for further refinement.