Projects using KafkaProjects using KafkaProblem:
Batch data processing was not suitable for real-time analytics and scalable cloud-based data ingestion.
Solution:
Created a real-time streaming pipeline using Kafka on AWS EC2, stored processed data in S3, cataloged it with AWS Glue, and queried it with Amazon Athena.
Tools:
Python, Apache Kafka, AWS EC2, Amazon S3, AWS Glue, Amazon Athena, Pandas
Result:
Built an end-to-end cloud data streaming workflow that supports real-time ingestion, storage, cataloging, and SQL-based analytics. Smartport IoT Integrated Platform Management
• Took ownership of a Smartport IoT integration platform originally developed through the IPLT national R&D project, maintaining and extending it for continued operational and research use.
• Operated the platform as a real-time data hub, providing port equipment data to external companies and research partners through MQTT, Kafka, and HTTP-based interfaces.
• Managed equipment integration • Operated the platform as a real-time data hub, providing port equipment data to external companies and research partners through MQTT, Kafka, and HTTP-based interfaces.and data availability by investigating terminal devices, communication methods, data formats, and operational status across connected port equipment.
• Maintained and improved data pipelines, interface configurations, and platform components to support stable data delivery across multiple downstream systems.
• Contributed to a commissioned operational analysis project by integrating Smartport equipment data with Terminal Operating System (TOS) data to analyze terminal workflows and equipment activities.
• Supported ongoing collaboration with external vendors, terminal stakeholders, and R&D participants by validating data flows, resolving integration issues, and coordinating interface requirements.• Operated the platform as a real-time data hub, providing port equipment data to external companies and research partners through MQTT, Kafka, and HTTP-based interfaces. Problem:
Many organizations still process invoices manually by reading PDF documents and entering key details (invoice number, vendor, amount, etc.) into systems. This process is slow, error-prone, and difficult to scale, and it also makes it harder to detect duplicate invoices or incorrect totals.
Solution:
This project builds an automated invoice processing pipeline that converts uploaded invoice PDFs into structured data. It uses OCR to extract text, LLMs to identify invoice fields, validation checks to ensure correctness, and Kafka-based event streaming to manage the processing pipeline. The extracted data is stored in PostgreSQL and visualized through a dashboard, enabling faster, scalable, and more reliable invoice processing. 
