Skip to main content
Telescope saves checkpoints during training so you can resume interrupted runs and export trained models. Checkpoints include model weights and optionally optimizer/scheduler state, and are saved in the backend’s native format (PyTorch DCP for FSDP, dist_checkpointing for Megatron).

​
Configuration

When checkpointing is enabled, a checkpoint is always saved at the final training step regardless of the checkpoint_every interval.

​
Checkpoint directory structure

Checkpoints are saved under checkpoints/ in your run directory (or the path set by checkpoint_dir):

​
Cleanup rules

By default, all checkpoints are kept. Use checkpoint_keep_last and checkpoint_keep_every to manage disk space:
A checkpoint is kept if it matches either rule. With the config above, after step 120 you’d have: step_50, step_100, step_110, step_120.

​
Resuming training

Set resume_from_checkpoint to continue from where training left off:
You can also resume from a specific step:
Resume restores model weights, optimizer state, learning rate schedule, orchestrator counters, and the dataset sampling order — so training continues exactly where it stopped. This requires checkpoint_save_training_state: true (the default) on the original run.

​
Weights-only checkpoints

Set checkpoint_save_training_state: false to save only model weights without optimizer or scheduler state. This produces smaller checkpoints that can be converted to HuggingFace format but cannot be used for resume.

​
Converting to HuggingFace format

Checkpoints are saved in the training backend’s native format. To use a checkpoint with vLLM, HuggingFace, or other tools, convert it to standard HuggingFace format (safetensors + config.json): Single checkpoint:
All checkpoints in a directory:
The converter handles both FSDP (DCP) and Megatron checkpoints automatically based on the meta.json backend field. Already-converted checkpoints are skipped in batch mode.