Solution for internal usage in the financial organization.

Purpose: import personal data which is under GDPR regulation for potential anti-money laundry investigation and reporting to the court.

data comes in "dirty" format, may be with duplicates or personal information with extra spaces or not ideal

huge file volumes, each file can contain million users

import should be done only if user data/balance been changed or new users

Bank staff have a management console where can do investigation for each case and review user data. System periodically import data from the shared file storage where all data represented in the CSV files with millions of rows. After processing those files and updating user data in the DB bank personal can perform due-diligence process.

Management console: ReactJS application with simple search where all data represented in table format mainly

Management service: responsible for data searching and processing personal data

SFTP Server: external server where other departments upload their data files

Import service: on the daily basis doing import of the data files, searching for a daily delta and inserting only new data to the MySql DB