Fine tuning Domain Specific dataset (Personal Dataset) on Gemma 2B Model

This repository contains the code and data for fine-tuning a small language model (SLM) for the specific domain of Indian history. The project demonstrates how to adapt a pre-trained language model to better understand and generate text relevant to this historical context.

Kaggle/collab Account

GPU access in Kaggle/collab

Hugging Face API Key

Large language models (LLMs) have shown remarkable capabilities in natural language processing tasks. However, fine-tuning them for specific domains remains crucial to unlock their full potential. This project explores the adaptation of the GEMMA model for analyzing Indian history, utilizing a dedicated Indian history dataset. We employ techniques like BitsAndBytes quantization and LoraConfig customization to optimize the model for causal language modeling tasks within this domain.

clone the repository

git clone https://github.com/AjayK47/Gemma-Model-Finetuning-Using-Lora.git

Run the Jupyter Notebooks in the following order:

You can Try Out My Hugging Face Model from here - https://huggingface.co/Ajayk/indian-history-gemma-instruction-finetuned

Install the dependencies using the following command:

The project follows these key steps:

Data Preprocessing: Select and clean a specific Domain dataset. Format the data into the GEMMA model chat template.

Fine-tuning:

Evaluation and Deployment:

Potential future directions include:

Contributions to this project are welcome!

This project is licensed under the MIT License.