prueba

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+**.spm filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,119 @@
----
-license: apache-2.0
----
+---
+license: Apache 2.0
+language:
+- eu
+pipeline_tag: translation
+tags:
+- Translation
+- Capitalization-and-punctuation
+- Transformer
+---
+
+# HiTZ's Capitalization & Punctuation model for Basque
+
+## Model description
+This model was trained from scratch usign Marian NMT. The dataset used in training contains 9,784,905 sentences. The model was evaluated on the Flores-101 dev and devtest datasets and some randomly picked CommonVoice dataset sentences.  
+* **Developed by**: HiTZ Research Center (University of the Basque Country EHU)
+* **Model type**: Capitalization and Punctuation
+* **Language**: Basque  
+
+## Intended uses and limitations  
+You can use this model for Punctuation and Capitalization in Basque  
+## Usage
+Required packages:
+- torch
+- transformers
+- sentencepiece
+
+### Capitalizing using python:
+Clone repository to download the model:
+
+```bash
+git clone https://huggingface.co/HiTZ/marianmt-cap-punct-eu
+```
+
+Given `MODEL_PATH` is the path that points to the downloaded `marianmt-cap-punct-eu` folder.
+
+```python
+from transformers import pipeline
+
+device = 0 # 0-->GPU, -1-->CPU
+
+segment_list = ["kaixo egun on guztioi", "faktoria e i te beko irratian entzuten da", "gutxi gora behera ehuneko berrogeita bikoa","lau zortzi hamabost hamasei hogeita hiru berrogeita bi", "nire jaio urtea mila bederatziehun eta laurogeita hamasei da", "informazio gehiago hitz puntu e hatxe u puntu eus web horrian"]
+
+translator = pipeline(task="translation", model=MODEL_PATH, tokenizer=MODEL_PATH, device=device)    
+result_list = translator(segment_list)
+cp_segment_list = [result["translation_text"] for result in result_list]
+
+for text, cp_text in zip(segment_list, cp_segment_list):
+    print(f"Normalized: {text}\n  With C&P: {cp_text}\n")
+```
+
+### Expected output:
+```bash
+Normalized: kaixo egun on guztioi
+  With C&P: Kaixo, egun on guztioi.
+
+Normalized: faktoria e i te beko irratian entzuten da
+  With C&P: Faktoria EiTBko irratian entzuten da.
+
+Normalized: gutxi gora behera ehuneko berrogeita bikoa
+  With C&P: Gutxi gora behera %42koa.
+
+Normalized: lau zortzi hamabost hamasei hogeita hiru berrogeita bi
+  With C&P: Lau, zortzi, hamabost, hamasei, hogeita hiru, berrogeita bi.
+
+Normalized: nire jaio urtea mila bederatziehun eta laurogeita hamasei da
+  With C&P: Nire jaio urtea 1996 da.
+
+Normalized: informazio gehiago hitz puntu e hatxe u puntu eus web horrian
+  With C&P: Informazio gehiago hitz.ehu.eus web horrian.
+```
+
+## Training
+
+### Data preparation
+The training data was compiled by our research group from multiple heterogeneous sources and consists of approximately 9,784,905 sentences.
+
+Prior to training, the data underwent preprocessing steps including cleaning, punctuation standardization, filtering, and the creation of aligned input–output sentence pairs for the capitalization and punctuation restoration task.
+
+To generate the input–output pairs, the target sentences were lowercased, punctuation was removed, and text normalization was applied using an in-house normalization tool.
+  
+Example:  
+```bash 
+Output--Cleaned, filetered and standarized: EHUko Estatutuetako 0. artikuluak ondorengoetarako eskumena aitortzen du: egitura bereziak sortzea unibertsitate irakaskuntza, ikasketa eta ikerketa, eta unibertsitatearen hedapenerako oinarri gisa.  
+Input--Lowercased, without punctuation and normalized: euskal herriko unibertsitateako estatutuetako zerogarren artikuluak ondorengoetarako eskumena aitortzen du egitura bereziak sortzea unibertsitate irakaskuntza ikasketa eta ikerketa eta unibertsitatearen hedapenerako oinarri gisa 
+```
+ 
+### Training procedure
+The model was trained using the official [MarianNMT](https://marian-nmt.github.io/quickstart/) implementation.  
+Training was performed on a single NVIDIA TITAN RTX GPU.
+
+## Performance
+The following table shows the model performance. We use the Word Error Rate metric. The WER-WITHOUT metric corresponds to the Word Error Rate computed on the evaluation dataset before applying capitalization and punctuation restoration. The WER metric corresponds to the output after processing with the model.
+The evaluation dataset underwent the same processing as the training dataset.  
+
+| Metric      | FLORES-101     | COMMON-VOICE   |
+|-------------|----------------|----------------|
+| WER-WITHOUT |       %19.55   |      %22.42    |
+|     WER     |        %5.99   |       %5.75    |
+
+
+# Aditional Information
+## Author
+HiTZ Basque Center for Language Technology - Aholab Signal Processing Laboratory, University of the Basque Country UPV/EHU.
+
+## Copyright
+Copyright (c) 2025 HiTZ Basque Center for Language Technology - Aholab Signal Processing Laboratory, University of the Basque Country UPV/EHU.
+
+## Licensing Information
+This work is licensed under a [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)  
+
+## Disclaimer
+<details>
+<summary>Click to expand</summary>
+The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions.
+
+When third parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of Artificial Intelligence.
+
+In no event shall the owner and creator of the models (HiTZ Basque Center for Language Technology - Aholab Signal Processing Laboratory, University of the Basque Country UPV/EHU.) be liable for any results arising from the use made by third parties of these models.

config.json

@@ -0,0 +1,39 @@
+{
+  "activation_dropout": 0.0,
+  "activation_function": "swish",
+  "architectures": [
+    "MarianMTModel"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "d_model": 512,
+  "decoder_attention_heads": 8,
+  "decoder_ffn_dim": 2048,
+  "decoder_layerdrop": 0.0,
+  "decoder_layers": 6,
+  "decoder_start_token_id": 32000,
+  "decoder_vocab_size": 32001,
+  "dropout": 0.1,
+  "dtype": "float16",
+  "encoder_attention_heads": 8,
+  "encoder_ffn_dim": 2048,
+  "encoder_layerdrop": 0.0,
+  "encoder_layers": 6,
+  "eos_token_id": 0,
+  "forced_eos_token_id": 0,
+  "init_std": 0.02,
+  "is_encoder_decoder": true,
+  "max_length": null,
+  "max_position_embeddings": 512,
+  "model_type": "marian",
+  "normalize_embedding": false,
+  "num_beams": null,
+  "num_hidden_layers": 6,
+  "pad_token_id": 32000,
+  "scale_embedding": true,
+  "share_encoder_decoder_embeddings": true,
+  "static_position_embeddings": true,
+  "transformers_version": "4.57.3",
+  "use_cache": true,
+  "vocab_size": 32001
+}

generation_config.json

@@ -0,0 +1,16 @@
+{
+  "_from_model_config": true,
+  "bad_words_ids": [
+    [
+      32000
+    ]
+  ],
+  "bos_token_id": 0,
+  "decoder_start_token_id": 32000,
+  "eos_token_id": 0,
+  "forced_eos_token_id": 0,
+  "max_length": 512,
+  "num_beams": 6,
+  "pad_token_id": 32000,
+  "transformers_version": "4.57.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6595ba4f83f1d3a5fbc0a1f9336442882810eb6d0e54b610772230feb6336d49
+size 153908130

source.spm

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:306c9fe2f6312eaaca2f28076c2f2292479ccbbe9629c6b773d6bc7efbeaf2f2
+size 842233

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "eos_token": "</s>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

target.spm

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:306c9fe2f6312eaaca2f28076c2f2292479ccbbe9629c6b773d6bc7efbeaf2f2
+size 842233

tokenizer_config.json

@@ -0,0 +1,39 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32000": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "separate_vocabs": false,
+  "source_lang": "",
+  "sp_model_kwargs": {},
+  "target_lang": "modelEuskeraConvertir",
+  "tokenizer_class": "MarianTokenizer",
+  "unk_token": "<unk>"
+}