D3 05 PEFT
Parameter-Efficient Fine-Tuning (PEFT)¶
Fine-tuning large pre-trained language models can be computationally expensive and require significant memory, especially when all model parameters are updated during training. Parameter-Efficient Fine-Tuning (PEFT) techniques aim to reduce the number of trainable parameters, allowing for efficient fine-tuning with less memory and computational cost.
PEFT is especially useful if the pre-trained model is loaded as a quantized model to save GPU memory and thus cannot be trained.
In this example, we will fine-tune a pre-trained model using Low-Rank Adapters (LoRA) to demonstrate parameter-efficient fine-tuning.
Bazzite-AI Setup Required
RunD0_00_Bazzite_AI_Setup.ipynbfirst to verify GPU access.
In [29]:
Copied!
# Import libraries
import torch
from datasets import load_dataset
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTTrainer, SFTConfig
torch.__version__
# Import libraries import torch from datasets import load_dataset from peft import LoraConfig, get_peft_model from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig from trl import SFTTrainer, SFTConfig torch.__version__
Out[29]:
'2.9.1+cu130'
In [30]:
Copied!
# Use TinyLlama for consistency with D3_02/D3_03/D3_04 notebooks
HF_LLM_MODEL = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
tokenizer = AutoTokenizer.from_pretrained(HF_LLM_MODEL)
# Set padding token and side for causal LMs
tokenizer.pad_token_id = tokenizer.eos_token_id
tokenizer.padding_side = 'left'
model = AutoModelForCausalLM.from_pretrained(
HF_LLM_MODEL,
quantization_config=BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_compute_dtype=torch.bfloat16),
device_map='cuda:0',
)
print(f"Model: {HF_LLM_MODEL}")
print(f"Tokenizer: pad_token_id={tokenizer.pad_token_id}, padding_side={tokenizer.padding_side}")
# Use TinyLlama for consistency with D3_02/D3_03/D3_04 notebooks HF_LLM_MODEL = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" tokenizer = AutoTokenizer.from_pretrained(HF_LLM_MODEL) # Set padding token and side for causal LMs tokenizer.pad_token_id = tokenizer.eos_token_id tokenizer.padding_side = 'left' model = AutoModelForCausalLM.from_pretrained( HF_LLM_MODEL, quantization_config=BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_compute_dtype=torch.bfloat16), device_map='cuda:0', ) print(f"Model: {HF_LLM_MODEL}") print(f"Tokenizer: pad_token_id={tokenizer.pad_token_id}, padding_side={tokenizer.padding_side}")
Out[30]:
Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0 Tokenizer: pad_token_id=2, padding_side=left
In [ ]:
Copied!
In [31]:
Copied!
# Load the guanaco dataset from HuggingFace Hub
guanaco_train = load_dataset('timdettmers/openassistant-guanaco', split='train')
guanaco_test = load_dataset('timdettmers/openassistant-guanaco', split='test')
# Load the guanaco dataset from HuggingFace Hub guanaco_train = load_dataset('timdettmers/openassistant-guanaco', split='train') guanaco_test = load_dataset('timdettmers/openassistant-guanaco', split='test')
Out[31]:
Repo card metadata block was not found. Setting CardData to empty.
Out[31]:
Repo card metadata block was not found. Setting CardData to empty.
In [32]:
Copied!
guanaco_train
guanaco_train
Out[32]:
Dataset({
features: ['text'],
num_rows: 9846
}) In [33]:
Copied!
guanaco_test
guanaco_test
Out[33]:
Dataset({
features: ['text'],
num_rows: 518
}) In [34]:
Copied!
guanaco_train[3]
guanaco_train[3]
Out[34]:
{'text': "### Human: I want to start doing astrophotography as a hobby, any suggestions what could i do?### Assistant: Getting started in astrophotography can seem daunting, but with some patience and practice, you can become a master of the craft. To begin, you'll need a good camera and lens, a tripod, and a dark sky location free of light pollution. You will also need to learn about the basics of astrophotography, such as what camera settings to use, how to capture star trails, and the best techniques for tracking celestial objects. You can also purchase or rent different types of telescopes, depending on what types of objects you want to capture. Additionally, it's important to keep up with the latest astrophotography news and trends. Once you have the necessary equipment and knowledge, you can start shooting and experimenting with different techniques to get the best results.### Human: Can you tell me more? What would you recommend as a basic set of equipment to get started with? How much will it cost?### Assistant: Astrophotography can be a fun and rewarding hobby, and here are some more in depth suggestions for you to get started:\n\n Equipment: As a beginner, you will need a camera that is capable of taking long exposures and manual control over the settings. A good starting point would be a digital SLR (DSLR) camera or a mirrorless camera. You will also need a sturdy tripod, a fast wide-angle lens (f/2.8 or faster), and a remote shutter release cable to minimize camera shake during long exposures.\n\n Location: Look for a location with minimal light pollution and a clear view of the night sky. Check online maps to find the best spots near you.\n\n Settings: Start with a wide-angle lens and set your camera to manual mode. Set your aperture to f/2.8 or wider, ISO to 800 or 1600, and experiment with different exposure times (30 seconds to 2 minutes).\n\n Focus: Focus the lens on the stars using live view and manual focus. You may also need to use a Bahtinov mask to achieve accurate focus.\n\n Editing: After you've taken your photos, use editing software such as Adobe Lightroom or Photoshop to adjust the exposure, contrast, and color balance to get the best results.\n\nThe cost of equipment will depend on the type of camera and lens you choose, but a basic setup can cost anywhere from $500 to $3000. However, you don't need to start with the most expensive equipment to get good results. As you improve, you can upgrade your equipment and add additional accessories such as filters and tracking mounts.\n\nRemember, astrophotography is a hobby that requires patience and practice, so don't be discouraged if your first few attempts don't turn out as well as you had hoped. Keep experimenting and learning, and you'll soon be able to capture stunning images of the night sky!### Human: What is a good course of instruction to avoid years of wasted effort?"} In [35]:
Copied!
question1 = guanaco_train[3]['text'].split('###')[1].removeprefix(' Human: ')
question1
question1 = guanaco_train[3]['text'].split('###')[1].removeprefix(' Human: ') question1
Out[35]:
'I want to start doing astrophotography as a hobby, any suggestions what could i do?'
In [36]:
Copied!
answer1 = guanaco_train[3]['text'].split('###')[2].removeprefix(' Assistant: ')
answer1
answer1 = guanaco_train[3]['text'].split('###')[2].removeprefix(' Assistant: ') answer1
Out[36]:
"Getting started in astrophotography can seem daunting, but with some patience and practice, you can become a master of the craft. To begin, you'll need a good camera and lens, a tripod, and a dark sky location free of light pollution. You will also need to learn about the basics of astrophotography, such as what camera settings to use, how to capture star trails, and the best techniques for tracking celestial objects. You can also purchase or rent different types of telescopes, depending on what types of objects you want to capture. Additionally, it's important to keep up with the latest astrophotography news and trends. Once you have the necessary equipment and knowledge, you can start shooting and experimenting with different techniques to get the best results."
In [37]:
Copied!
messages = [
{'role': 'user', 'content': question1},
{'role': 'assistant', 'content': answer1}
]
messages
messages = [ {'role': 'user', 'content': question1}, {'role': 'assistant', 'content': answer1} ] messages
Out[37]:
[{'role': 'user',
'content': 'I want to start doing astrophotography as a hobby, any suggestions what could i do?'},
{'role': 'assistant',
'content': "Getting started in astrophotography can seem daunting, but with some patience and practice, you can become a master of the craft. To begin, you'll need a good camera and lens, a tripod, and a dark sky location free of light pollution. You will also need to learn about the basics of astrophotography, such as what camera settings to use, how to capture star trails, and the best techniques for tracking celestial objects. You can also purchase or rent different types of telescopes, depending on what types of objects you want to capture. Additionally, it's important to keep up with the latest astrophotography news and trends. Once you have the necessary equipment and knowledge, you can start shooting and experimenting with different techniques to get the best results."}] In [38]:
Copied!
tokenizer.apply_chat_template(messages, tokenize=False)
tokenizer.apply_chat_template(messages, tokenize=False)
Out[38]:
"<|user|>\nI want to start doing astrophotography as a hobby, any suggestions what could i do?</s>\n<|assistant|>\nGetting started in astrophotography can seem daunting, but with some patience and practice, you can become a master of the craft. To begin, you'll need a good camera and lens, a tripod, and a dark sky location free of light pollution. You will also need to learn about the basics of astrophotography, such as what camera settings to use, how to capture star trails, and the best techniques for tracking celestial objects. You can also purchase or rent different types of telescopes, depending on what types of objects you want to capture. Additionally, it's important to keep up with the latest astrophotography news and trends. Once you have the necessary equipment and knowledge, you can start shooting and experimenting with different techniques to get the best results.</s>\n"
In [39]:
Copied!
def reformat_text(text, include_answer=True):
question1 = text.split('###')[1].removeprefix(' Human: ')
answer1 = text.split('###')[2].removeprefix(' Assistant: ')
if include_answer:
messages = [
{'role': 'user', 'content': question1},
{'role': 'assistant', 'content': answer1}
]
else:
messages = [
{'role': 'user', 'content': question1}
]
reformatted_text = tokenizer.apply_chat_template(messages, tokenize=False)
return reformatted_text
def reformat_text(text, include_answer=True): question1 = text.split('###')[1].removeprefix(' Human: ') answer1 = text.split('###')[2].removeprefix(' Assistant: ') if include_answer: messages = [ {'role': 'user', 'content': question1}, {'role': 'assistant', 'content': answer1} ] else: messages = [ {'role': 'user', 'content': question1} ] reformatted_text = tokenizer.apply_chat_template(messages, tokenize=False) return reformatted_text
Out[39]:
[No output generated]
In [40]:
Copied!
reformat_text(guanaco_train[3]['text'])
reformat_text(guanaco_train[3]['text'])
Out[40]:
"<|user|>\nI want to start doing astrophotography as a hobby, any suggestions what could i do?</s>\n<|assistant|>\nGetting started in astrophotography can seem daunting, but with some patience and practice, you can become a master of the craft. To begin, you'll need a good camera and lens, a tripod, and a dark sky location free of light pollution. You will also need to learn about the basics of astrophotography, such as what camera settings to use, how to capture star trails, and the best techniques for tracking celestial objects. You can also purchase or rent different types of telescopes, depending on what types of objects you want to capture. Additionally, it's important to keep up with the latest astrophotography news and trends. Once you have the necessary equipment and knowledge, you can start shooting and experimenting with different techniques to get the best results.</s>\n"
In [41]:
Copied!
# Now, apply reformat_train(..) to both datasets:
guanaco_train = guanaco_train.map(lambda entry: {
'reformatted_text': reformat_text(entry['text'])
})
guanaco_test = guanaco_test.map(lambda entry: {
'reformatted_text': reformat_text(entry['text'])
})
# Now, apply reformat_train(..) to both datasets: guanaco_train = guanaco_train.map(lambda entry: { 'reformatted_text': reformat_text(entry['text']) }) guanaco_test = guanaco_test.map(lambda entry: { 'reformatted_text': reformat_text(entry['text']) })
Out[41]:
[No output generated]
In [42]:
Copied!
guanaco_train
guanaco_train
Out[42]:
Dataset({
features: ['text', 'reformatted_text'],
num_rows: 9846
}) In [43]:
Copied!
guanaco_test
guanaco_test
Out[43]:
Dataset({
features: ['text', 'reformatted_text'],
num_rows: 518
}) In [ ]:
Copied!
In [44]:
Copied!
# Add low-rank adapters (LORA) to the model:
peft_config = LoraConfig(
task_type='CAUSAL_LM',
r=16,
lora_alpha=32, # thumb rule: lora_alpha should be 2*r
lora_dropout=0.05,
bias='none',
target_modules='all-linear',
)
# model = get_peft_model(model, peft_config)
# Add low-rank adapters (LORA) to the model: peft_config = LoraConfig( task_type='CAUSAL_LM', r=16, lora_alpha=32, # thumb rule: lora_alpha should be 2*r lora_dropout=0.05, bias='none', target_modules='all-linear', ) # model = get_peft_model(model, peft_config)
Out[44]:
[No output generated]
In [45]:
Copied!
training_arguments = SFTConfig(
output_dir='output/tinyllama-1.1b-chat-guanaco',
per_device_train_batch_size=2, # Reduced from 8 for memory efficiency
gradient_accumulation_steps=4, # Increased to compensate
gradient_checkpointing=True,
gradient_checkpointing_kwargs={'use_reentrant': False},
optim='adamw_torch',
learning_rate=2e-4,
logging_strategy='steps',
logging_steps=10,
save_strategy='no',
max_steps=100,
bf16=True,
report_to='none',
max_length=512, # Reduced from 1024 for memory efficiency
dataset_text_field='reformatted_text',
)
training_arguments = SFTConfig( output_dir='output/tinyllama-1.1b-chat-guanaco', per_device_train_batch_size=2, # Reduced from 8 for memory efficiency gradient_accumulation_steps=4, # Increased to compensate gradient_checkpointing=True, gradient_checkpointing_kwargs={'use_reentrant': False}, optim='adamw_torch', learning_rate=2e-4, logging_strategy='steps', logging_steps=10, save_strategy='no', max_steps=100, bf16=True, report_to='none', max_length=512, # Reduced from 1024 for memory efficiency dataset_text_field='reformatted_text', )
Out[45]:
[No output generated]
In [46]:
Copied!
trainer = SFTTrainer(
model=model,
peft_config=peft_config,
args=training_arguments,
train_dataset=guanaco_train,
eval_dataset=guanaco_test,
processing_class=tokenizer,
)
trainer = SFTTrainer( model=model, peft_config=peft_config, args=training_arguments, train_dataset=guanaco_train, eval_dataset=guanaco_test, processing_class=tokenizer, )
Out[46]:
[No output generated]
In [47]:
Copied!
eval_result = trainer.evaluate()
print("Evaluation on test dataset before finetuning:")
print(eval_result)
eval_result = trainer.evaluate() print("Evaluation on test dataset before finetuning:") print(eval_result)
Out[47]:
<IPython.core.display.HTML object>
Out[47]:
Evaluation on test dataset before finetuning:
{'eval_loss': 1.6349399089813232, 'eval_model_preparation_time': 0.0019, 'eval_runtime': 15.3591, 'eval_samples_per_second': 33.726, 'eval_steps_per_second': 4.232, 'eval_entropy': 1.5628409018883338, 'eval_num_tokens': 0.0, 'eval_mean_token_accuracy': 0.635103637438554}
In [48]:
Copied!
train_result = trainer.train()
print("Training result:")
print(train_result)
train_result = trainer.train() print("Training result:") print(train_result)
Out[48]:
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'pad_token_id': 2}.
Out[48]:
<IPython.core.display.HTML object>
Out[48]:
Training result:
TrainOutput(global_step=100, training_loss=1.459467601776123, metrics={'train_runtime': 60.3472, 'train_samples_per_second': 13.257, 'train_steps_per_second': 1.657, 'total_flos': 1924738659975168.0, 'train_loss': 1.459467601776123, 'epoch': 0.08125126955108673})
In [49]:
Copied!
eval_result = trainer.evaluate()
print("Evaluation on test dataset after finetuning:")
print(eval_result)
eval_result = trainer.evaluate() print("Evaluation on test dataset after finetuning:") print(eval_result)
Out[49]:
Evaluation on test dataset after finetuning:
{'eval_loss': 1.495754361152649, 'eval_model_preparation_time': 0.0019, 'eval_runtime': 15.2864, 'eval_samples_per_second': 33.886, 'eval_steps_per_second': 4.252, 'eval_entropy': 1.5434014815550583, 'eval_num_tokens': 235882.0, 'eval_mean_token_accuracy': 0.6537831517366263, 'epoch': 0.08125126955108673}
In [50]:
Copied!
# trainer.save_model()
# model.save_pretrained('output_dir')
# tokenizer.save_pretrained('output_dir')
# model.push_to_hub("your-name/model-name")
# trainer.save_model() # model.save_pretrained('output_dir') # tokenizer.save_pretrained('output_dir') # model.push_to_hub("your-name/model-name")
Out[50]:
[No output generated]
In [51]:
Copied!
## To load fine-tuned model, use:
# from peft import AutoPeftModelForCausalLM
# from transformers import AutoTokenizer
# model_name = 'phi-3.5-mini-instruct-medmcqa'
# model = AutoPeftModelForCausalLM.from_pretrained(
# model_name,
# dtype=torch.bfloat16,
# device_map='cuda'
# )
# tokenizer = AutoTokenizer.from_pretrained(model_name)
## To load fine-tuned model, use: # from peft import AutoPeftModelForCausalLM # from transformers import AutoTokenizer # model_name = 'phi-3.5-mini-instruct-medmcqa' # model = AutoPeftModelForCausalLM.from_pretrained( # model_name, # dtype=torch.bfloat16, # device_map='cuda' # ) # tokenizer = AutoTokenizer.from_pretrained(model_name)
Out[51]:
[No output generated]
In [52]:
Copied!
# model = AutoModelForCausalLM.from_pretrained(base_model_id)
# model.load_adapter(peft_model_id)
# model = AutoModelForCausalLM.from_pretrained(base_model_id) # model.load_adapter(peft_model_id)
Out[52]:
[No output generated]
In [53]:
Copied!
# model.enable_adapters()
# model.disable_adapters()
# model.merge_and_unload()
# model.enable_adapters() # model.disable_adapters() # model.merge_and_unload()
Out[53]:
[No output generated]
In [ ]:
Copied!
Conclusion¶
In this notebook, we explored Parameter-Efficient Fine-Tuning (PEFT) techniques using adapter layers to efficiently fine-tune pre-trained models. You learned how to configure and apply adapters, perform fine-tuning, and evaluate the model's performance.
Next Steps:¶
In the next notebook, we will explore a technique to speed up model inference and training.
In [ ]:
Copied!
In [26]:
Copied!
# Shut down the kernel to release memory
import IPython
app = IPython.Application.instance()
app.kernel.do_shutdown(restart=False)
# Shut down the kernel to release memory import IPython app = IPython.Application.instance() app.kernel.do_shutdown(restart=False)
Out[26]:
{'status': 'ok', 'restart': False} In [ ]:
Copied!