isy-thl/multilingual-e5-base-learning-outcome-skill-tuned

SentenceTransformer based on intfloat/multilingual-e5-base

This is a sentence-transformers model finetuned from intfloat/multilingual-e5-base. It maps learning outcomes, course descriptions etc. & ESCO skill descriptions to a 768-dimensional dense vector space and can be used for semantic search, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: intfloat/multilingual-e5-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset: course-esco-skill-retrieval

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Custom Skill Retrieval Finetuning

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'query: Erweitern Sie Ihr Beratungsprofil: Lernen Sie, wie Sie Betriebe bei der Reduzierung von Chemikalien nach aktuellen Umweltvorschriften unterstützen – für mehr Nachhaltigkeit und Sicherheit!',
    'passage: Zur Verringerung des Chemikalieneinsatzes beraten: Zur Verringerung des Einsatzes von Chemikalien wie Pestiziden und der Emissionen verschiedener chemischer Stoffe beraten, um deren Auswirkungen auf die Umwelt zu begrenzen und die Risiken für den Menschen zu verringern. Bezüglich geltender Vorschriften auf dem Laufenden bleiben.',
    'passage: Sägetechniken: Verschiedene Sägetechniken zur Verwendung manueller und elektrischer Sägen.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.6994, -0.0105],
#         [ 0.6994,  1.0000, -0.0066],
#         [-0.0105, -0.0066,  1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: learning_outcome_esco_pairs
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8582
cosine_accuracy@3	0.9546
cosine_accuracy@5	0.9716
cosine_accuracy@10	0.983
cosine_precision@1	0.8582
cosine_precision@3	0.3963
cosine_precision@5	0.2556
cosine_precision@10	0.1359
cosine_recall@1	0.7239
cosine_recall@3	0.8962
cosine_recall@5	0.9324
cosine_recall@10	0.9639
cosine_ndcg@5	0.8942
cosine_ndcg@10	0.9054
cosine_mrr@10	0.9092
cosine_map@100	0.8753

Training Details

Training Dataset

course-esco-skill-retrieval

• Size: 9,562 training samples

• Columns: anchor, positive, and negative

• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	list
  details	min: 21 tokens mean: 82.15 tokens max: 299 tokens	min: 16 tokens mean: 51.09 tokens max: 152 tokens	size: 3 elements

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 32,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• gradient_accumulation_steps: 2
• learning_rate: 2e-05
• weight_decay: 0.01
• num_train_epochs: 8
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• save_only_model: True
• fp16: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 2
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 8
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: True
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	cosine_ndcg@10
0	50	4.287	0.8414
1	100	0.7882	0.8823
1	150	0.6427	0.8970
2	200	0.4974	0.8960
3	250	0.4099	0.8987
3	300	-	0.9054

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.1.1
• Transformers: 4.52.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.9.0
• Datasets: 4.1.1
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

Model Card Authors

• Pascal Hürten pascal.huerten@th-luebeck.de Technische Hochschule Lübeck Institut für Interaktive Systeme (ISy)