A PyTorch Lightning extension that enhances model experimentation with flexible fine-tuning schedules.

---

Docs • Setup • Examples • Community

---

FinetuningScheduler is simple to use yet powerful, offering a number of features that facilitate model research and exploration:

• easy specification of flexible fine-tuning schedules with explicit or regex-based parameter selection
  • implicit schedules for initial/naive model exploration
  • explicit schedules for performance tuning, fine-grained behavioral experimentation and computational efficiency
• automatic restoration of best per-phase checkpoints driven by iterative application of early-stopping criteria to each fine-tuning phase
• composition of early-stopping and manually-set epoch-driven fine-tuning phase transitions

---

Setup

Step 0: Install from PyPI

pip install finetuning-scheduler

Additional installation options

Install Optional Packages

To install additional packages required for examples:

pip install finetuning-scheduler['examples']

or to include packages for examples, development and testing:

pip install finetuning-scheduler['all']

Source Installation Examples

To install from (editable) source (includes docs as well):

# FTS pins Lightning to a specific commit for CI and development
# This is similar to PyTorch's approach with Triton.
export USE_CI_COMMIT_PIN="1"

git clone https://github.com/speediedan/finetuning-scheduler.git
cd finetuning-scheduler
python -m pip install -e ".[all]" -r requirements/docs.txt

Install a specific FTS version from source using the standalone pytorch-lighting package:

export FTS_VERSION=2.6.0
export PACKAGE_NAME=pytorch
git clone -b v${FTS_VERSION} https://github.com/speediedan/finetuning-scheduler
cd finetuning-scheduler
python -m pip install -e ".[all]" -r requirements/docs.txt

Latest Docker Image

Note, publishing of new finetuning-scheduler version-specific docker images was paused after the 2.0.2 patch release. If new version-specific images are required, please raise an issue.

Step 1: Import the FinetuningScheduler callback and start fine-tuning!

import lightning as L
from finetuning_scheduler import FinetuningScheduler

trainer = L.Trainer(callbacks=[FinetuningScheduler()])

Get started by following the Fine-Tuning Scheduler introduction which includes a CLI-based example or by following the notebook-based Fine-Tuning Scheduler tutorial.

---

Installation Using the Standalone pytorch-lightning Package

applicable to versions >= 2.0.0

Now that the core Lightning package is lightning rather than pytorch-lightning, Fine-Tuning Scheduler (FTS) by default depends upon the lightning package rather than the standalone pytorch-lightning. If you would like to continue to use FTS with the standalone pytorch-lightning package instead, you can still do so as follows:

Install a given FTS release (for example v2.0.0) using standalone pytorch-lightning:

export FTS_VERSION=2.0.0
export PACKAGE_NAME=pytorch
wget https://github.com/speediedan/finetuning-scheduler/releases/download/v${FTS_VERSION}/finetuning-scheduler-${FTS_VERSION}.tar.gz
pip install finetuning-scheduler-${FTS_VERSION}.tar.gz

Dynamic Versioning

FTS (as of version 2.6.0) now enables dynamic versioning both at installation time and via CLI post-installation. Initially, the dynamic versioning system allows toggling between Lightning unified and standalone imports. The two conversion operations are individually idempotent and mutually reversible.

Toggling Between Unified and Standalone Lightning Imports

FTS provides a simple CLI tool to easily toggle between unified and standalone import installation versions post-installation:

# Toggle from unified to standalone Lightning imports
toggle-lightning-mode --mode standalone

# Toggle from standalone to unified Lightning imports (default)
toggle-lightning-mode --mode unified

Note: If you have the standalone package (pytorch-lightning) installed but not the unified package (lightning), toggling to unified mode will be prevented. You must install the lightning package first before toggling.

This can be useful when:

• You need to adapt existing code to work with a different Lightning package
• You're switching between projects using different Lightning import styles
• You want to test compatibility with both import styles

---

Examples

Scheduled Fine-Tuning For SuperGLUE

• Notebook-based Tutorial
• CLI-based Tutorial
• FSDP Scheduled Fine-Tuning
• LR Scheduler Reinitialization (advanced)
• Optimizer Reinitialization (advanced)

---

Continuous Integration

Fine-Tuning Scheduler is rigorously tested across multiple CPUs, GPUs and against major Python and PyTorch versions. Each Fine-Tuning Scheduler minor release (major.minor.patch) is paired with a Lightning minor release (e.g. Fine-Tuning Scheduler 2.0 depends upon Lightning 2.0).

To ensure maximum stability, the latest Lightning patch release fully tested with Fine-Tuning Scheduler is set as a maximum dependency in Fine-Tuning Scheduler's requirements.txt (e.g. <= 1.7.1). If you'd like to test a specific Lightning patch version greater than that currently in Fine-Tuning Scheduler's requirements.txt, it will likely work but you should install Fine-Tuning Scheduler from source and update the requirements.txt as desired.

Current build statuses for Fine-Tuning Scheduler

System / (PyTorch/Python ver)	2.3.1/3.9	2.7.0/3.9, 2.7.0/3.12
Linux [GPUs**]	-
Linux (Ubuntu 22.04)
OSX (14)
Windows (2022)

• ** tests run on one RTX 4090 and one RTX 2070

Community

Fine-Tuning Scheduler is developed and maintained by the community in close communication with the Lightning team. Thanks to everyone in the community for their tireless effort building and improving the immensely useful core Lightning project.

PR's welcome! Please see the contributing guidelines (which are essentially the same as Lightning's).

---

Citing Fine-Tuning Scheduler

Please cite:

@misc{Dan_Dale_2022_6463952,
    author       = {Dan Dale},
    title        = {{Fine-Tuning Scheduler}},
    month        = Feb,
    year         = 2022,
    doi          = {10.5281/zenodo.6463952},
    publisher    = {Zenodo},
    url          = {https://zenodo.org/record/6463952}
    }

Feel free to star the repo as well if you find it useful or interesting. Thanks 😊!