Open Source | Data Source Control

The Data Version Control (DVC) project is an open-source version control system specifically designed for machine learning projects. DVC enables data scientists and machine learning engineers to efficiently manage data, models, and experiments.

The DVC team aimed to fulfill user requests by making their tool accessible via the Anaconda Package Manager. Additionally, the goal was to expand DVC’s capabilities by integrating support for Google Drive cloud storage.

Anaconda DVC Package: Developed the dvc-feedstock repository, enabling DVC and all relevant dependencies to be available through the Anaconda Package Manager.

Google Drive Integration: Implemented the highly requested Google Drive support, allowing users to store data seamlessly using Google's cloud storage service. Beyond software development, I personally handled Google's verification process, successfully collaborating with Google's API team to activate necessary Google Drive API services for DVC.

Documentation Enhancement: Updated and expanded DVC documentation with comprehensive, step-by-step guides for setting up and utilizing the Google Drive integration.

Python: Developed new features and resolved bugs within the DVC codebase.

Google Drive API: Managed user data interactions aligned with DVC’s functionality.

Git: Utilized for effective version control and collaborative development.

Markdown: Created and maintained clear, user-friendly documentation.

Continuous Integration: Established CI processes to maintain high-quality standards and ensure the reliability of the DVC project.

Integrating DVC with Anaconda Package Manager required comprehensive management of all DVC dependencies, updating them to compatible versions, and maintaining a dedicated GitHub repository listed within the conda-forge recipe database. Technical challenges were systematically resolved, successfully making DVC accessible to conda users. Adding Google Drive support necessitated reliable interaction with Google Drive APIs. After thorough research, the decision was made to revive and maintain an open-source Python wrapper for Google Drive APIs, named PyDrive2. This wrapper was integrated into DVC, providing seamless API usage from Python. Through extensive testing and iterative improvements, an optimized solution was developed that respected Google's API usage limits.

The DVC project was successfully published on the Anaconda Package Manager, greatly enhancing its accessibility. Additionally, the integration of Google Drive support significantly expanded DVC’s data storage capabilities, benefiting its growing community. The PyDrive2 library was revitalized and actively maintained to ensure reliable and efficient interaction with Google Drive APIs from Python-based applications.