Strategic Overhaul of AAK Architecture – Scalable Microservices

As the technical lead and member of the board of directors at AAK Tele-Science, I spearheaded the strategic overhaul of the company’s legacy platform architecture. The project involved transitioning from a monolithic Elgg-based system to a highly scalable microservices architecture to resolve existing limitations and set the stage for future growth. This initiative required a deep restructuring of both the technical infrastructure and the development processes, fostering a scalable, flexible, and agile environment. The overhaul was critical to enabling AAK Tele-Science to improve platform performance, streamline team collaboration, and support new features such as blockchain integration, better AI-driven analytics, and real-time collaborations.

The previous Elgg-based platform faced significant limitations that restricted both the technical scalability of the system and the productivity of the development teams:

Monolithic structure: The tightly coupled nature of the Elgg platform created bottlenecks between the frontend and backend teams, stalling feature development and deployment.

Inflexible Design: Built-in Elgg UI and codebase made frontend developers struggle incorporating custom UI/UX mockups.

Scalability and performance concerns: It became increasingly clear that as the platform’s user base grows, the monolithic architecture will struggle to support future growth.

Developer onboarding challenges: The complexity and poor documentation of Elgg made it difficult to hire new developers and onboard them effectively, further hampering team productivity.

As the technical lead, I was responsible for driving the architectural transformation of the platform. This project involved the following key areas:

1. Architectural Restructuring:

Led the complete overhaul of the system’s architecture, transitioning from a monolithic structure to a modular microservices-based infrastructure.

Separation of concerns: We decoupled the frontend from the backend, adopting a REST API-based architecture. The backend was restructured using Django REST Framework, while the frontend was implemented with ReactJS. This separation eliminated dependencies that previously caused development bottlenecks.

2. Development Process Redesign:

Implemented Agile/Scrum methodologies, introducing structured sprints and better communication among the teams. I also introduced GitHub Projects with Kanban boards & roadmaps to track tasks and milestones for more effective project management.

Each complex big feature had its own collaborative Kanban board and was broken down to epic cards, which would further get broken down into smaller tasks for different teams (frontend, backend, data science, blockchain, etc.). This approach provided clarity on the integration of different components and ensured smooth collaboration between teams.

3. Multi-Environment Workflow and CI/CD Pipelines:

Reimagined the development lifecycle by introducing multi-environment workflows: development, testing, and production. This allowed teams to work on separate environments and maintain stability in production.

Implemented CI/CD pipelines that automated testing and deployment processes, minimizing human errors and accelerating feature releases.

Integrated Git and version control best practices, ensuring that every pull request was reviewed and tested before merging into the main branch. Only team leads were authorized to push to production, significantly reducing the risk of untested code being deployed.

4. Team Scalability and Productivity Improvements:

The new architecture and processes allowed teams to work independently, removing bottlenecks between frontend and backend.

We overcame the steep learning curve associated with Elgg for frontend developers by creating comprehensive Swagger API documentation for all backend services, empowering frontend developers to utilize the backend without delays.

Streamlined hiring and onboarding by eliminating the complex and outdated Elgg infrastructure, making it easier to recruit and integrate new developers into the team.

5. Infrastructure Migration:

Migrated the platform’s infrastructure from a single-server setup to scalable AWS EC2 instances. We utilized Docker to containerize services, ensuring portability, and Kubernetes for orchestration and scaling.

The new cloud infrastructure enabled automatic scaling of services based on user demand, improving platform availability and reducing operational costs.

300% improvement in platform performance: The shift from Elgg to microservices, coupled with cloud migration and infrastructure optimization, resulted in a dramatic improvement in navigation speed and overall performance.

Decoupled development: The separation of frontend and backend allowed teams to work in parallel, increasing development speed by more than 200% and removing previous blockers.

Agile transformation: The introduction of Agile methodologies led to improved communication across teams and more predictable, on-time feature delivery. The Kanban board structure streamlined the task-tracking process, aligning individual team efforts with the platform’s long-term goals.

Improved scalability and flexibility: The microservices architecture allowed the platform to scale horizontally, supporting the growing user base without performance degradation.

Robust CI/CD pipeline: Automated testing and deployment workflows ensured that features were tested thoroughly in the development and test environments before reaching production, improving code quality and reducing bugs in live releases.

The transition from a monolithic to microservices-based architecture required significant changes in both technical implementation and team workflow. The primary challenges we overcame included:

Breaking down the monolithic architecture into smaller, more manageable services that could be developed and deployed independently.

Redefining team structures and processes: Shifting from an ad-hoc development process to a well-structured Agile methodology required substantial change management and continuous monitoring to ensure its success.

Ensuring backward compatibility with existing features during the migration to the new architecture, while also preparing the platform for future enhancements.

This architectural overhaul not only addressed the limitations of the legacy Elgg platform but also enabled AAK Tele-Science to grow and scale its operations. The success of this project paved the way for the development of new, advanced platform features, including blockchain integration, AI-driven analytics, and sophisticated collaboration tools. It also significantly enhanced the efficiency and productivity of the development teams, reducing feature delivery times and increasing platform stability. The new infrastructure laid a strong foundation for future projects, ensuring that AAK Tele-Science was positioned to continue innovating and growing without the limitations of outdated technologies.