distil-bumble-bert

This model was trained from scratch on the None dataset. It achieves the following results on the evaluation set:

Model description

More information needed

More information needed

More information needed

The following hyperparameters were used during training:

learning_rate: 2e-07
train_batch_size: 16
eval_batch_size: 16
seed: 42
gradient_accumulation_steps: 4
total_train_batch_size: 64
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 10
mixed_precision_training: Native AMP

Training Loss	Epoch	Step	Accuracy	F1	Validation Loss	Precision	Recall	Roc Auc
0.4599	0.5204	500	0.7726	0.5497	0.4772	0.6237	0.4914	0.7998
0.4529	1.0416	1000	0.7869	0.5670	0.4523	0.6653	0.4940	0.8199
0.4542	1.5621	1500	0.7893	0.5823	0.4455	0.6614	0.5201	0.8268
0.4428	2.0833	2000	0.7898	0.5943	0.4428	0.6533	0.5451	0.8306
0.4999	2.6037	2500	0.7896	0.6095	0.4451	0.6403	0.5816	0.8318
0.4168	3.1249	3000	0.7941	0.5967	0.4378	0.6677	0.5393	0.8338
0.4143	3.6453	3500	0.7918	0.6114	0.4413	0.6463	0.5800	0.8353
0.4528	4.1665	4000	0.7950	0.6125	0.4333	0.6569	0.5737	0.8395
0.3911	4.6870	4500	0.7941	0.6278	0.4374	0.6413	0.6149	0.8397
0.4225	5.2082	5000	0.7935	0.6270	0.4360	0.6401	0.6144	0.8404
0.3861	5.7286	5500	0.7946	0.6259	0.4338	0.6442	0.6087	0.8412
0.4541	6.2498	6000	0.7965	0.6354	0.4356	0.6430	0.6279	0.8417
0.4086	6.7702	6500	0.7916	0.6332	0.4367	0.6296	0.6368	0.8417
0.4318	7.2914	7000	0.7935	0.6385	0.4372	0.6315	0.6456	0.8421
0.4483	7.8119	7500	0.7957	0.6305	0.4324	0.6445	0.6170	0.8429
0.4059	8.3331	8000	0.7931	0.6306	0.4329	0.6359	0.6253	0.8431
0.4138	8.8535	8500	0.7968	0.6279	0.4296	0.6501	0.6071	0.8439
0.427	9.3747	9000	0.7960	0.6314	0.4309	0.6448	0.6186	0.8439
0.4185	9.8951	9500	0.4346	0.7931	0.6309	0.6441	0.6374	0.8438

Safetensors

Model size

67M params

Tensor type

F32

Finetunes