OSS

Libraries

• Axolotl
• Torch-Tune
• Training:
  • FSDP
  • DeepSpeed

Recipes

• Axolotl examples
• Torch-Tune examples

Papers

Parameter-Efficient Fine-tuning

• LoRA: LoRA: Low-Rank Adaptation of Large Language Models
• SoRA: Sparse Low-rank Adaptation of Pre-trained Language Models
  • Adapter is linear combination of: original weights, LoRA weights, and sparse weights.
• DoRA: Weight-Decomposed Low-Rank Adaptation