|
|
--- |
|
|
title: CoDA Fine-tuning |
|
|
emoji: 🚀 |
|
|
colorFrom: blue |
|
|
colorTo: purple |
|
|
sdk: gradio |
|
|
sdk_version: 4.44.0 |
|
|
app_file: app.py |
|
|
pinned: false |
|
|
license: apache-2.0 |
|
|
hf_oauth: true |
|
|
hf_oauth_scopes: |
|
|
- read-repos |
|
|
- write-repos |
|
|
--- |
|
|
|
|
|
# CoDA Model Fine-tuning Space |
|
|
|
|
|
This Space allows you to fine-tune the **Salesforce/CoDA-v0-Instruct** text generation diffusion model on the **baseten-admin/gpt-oss120b-generated-perfectblend** dataset. |
|
|
|
|
|
## Features |
|
|
|
|
|
- 🎯 **Full Fine-tuning**: Complete parameter fine-tuning (not LoRA) |
|
|
- 💬 **ChatML Format**: Processes conversation data with question-answer pairs |
|
|
- 🔄 **Auto Upload**: Automatically uploads trained model to your Hugging Face account |
|
|
- 📊 **Progress Tracking**: Real-time training progress updates |
|
|
- 🔐 **OAuth Integration**: Secure authentication via Hugging Face login |
|
|
|
|
|
## How to Use |
|
|
|
|
|
1. **Login**: Click the "Sign in with Hugging Face" button |
|
|
2. **Configure**: Adjust training parameters (epochs, batch size, learning rate) |
|
|
3. **Train**: Click "Start Training" (requires GPU - upgrade Space to GPU tier) |
|
|
4. **Resume**: If training is interrupted, check "Resume from last checkpoint" and restart |
|
|
5. **Upload**: After training completes, click "Upload to Hugging Face Hub" |
|
|
|
|
|
### Persistence |
|
|
|
|
|
This Space supports checkpoint persistence: |
|
|
- Training checkpoints are saved every 500 steps |
|
|
- If interrupted, you can resume from the last checkpoint |
|
|
- For Docker deployment: Mount `/data` volume for full persistence |
|
|
- On Spaces: Checkpoints persist within the same session and across rebuilds if using persistent storage tier |
|
|
|
|
|
## Requirements |
|
|
|
|
|
- **Hardware**: GPU (T4, A10G, or better) strongly recommended |
|
|
- **Account**: Hugging Face account with write permissions |
|
|
- **Time**: Training takes several hours depending on configuration |
|
|
|
|
|
## About the Model |
|
|
|
|
|
**CoDA (Code Diffusion with Autoregressive)** is a 1.7B parameter bidirectional diffusion model developed by Salesforce AI Research. Unlike traditional autoregressive models, CoDA uses discrete denoising for text generation. The Instruct version is pre-tuned for instruction following, making it ideal for fine-tuning on conversational data. |
|
|
|
|
|
### Model Configuration |
|
|
|
|
|
```json |
|
|
{ |
|
|
"architectures": ["CoDALanguageModel"], |
|
|
"hidden_size": 2048, |
|
|
"num_hidden_layers": 28, |
|
|
"num_attention_heads": 16, |
|
|
"vocab_size": 151936, |
|
|
"max_position_embeddings": 40960 |
|
|
} |
|
|
``` |
|
|
|
|
|
## Dataset |
|
|
|
|
|
The training uses the **baseten-admin/gpt-oss120b-generated-perfectblend** dataset: |
|
|
- **Format**: Conversational data in ChatML format |
|
|
- **Column**: `conversations` (list of role-content pairs) |
|
|
- **Split**: Uses `train` split with 90/10 train/eval split |
|
|
|
|
|
## Training Details |
|
|
|
|
|
- **Optimizer**: AdamW |
|
|
- **Precision**: FP16 (on GPU) |
|
|
- **Gradient Accumulation**: 4 steps |
|
|
- **Gradient Checkpointing**: Enabled for memory efficiency |
|
|
- **Max Sequence Length**: 2048 tokens |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use this Space or the CoDA model, please cite: |
|
|
|
|
|
```bibtex |
|
|
@article{coda2023, |
|
|
title={CoDA: Bidirectional Code Diffusion}, |
|
|
author={Salesforce AI Research}, |
|
|
journal={arXiv preprint}, |
|
|
year={2023} |
|
|
} |
|
|
``` |
|
|
|
|
|
## License |
|
|
|
|
|
Apache 2.0 |
|
|
|
