Download model weights in Dockerfile. Large revision of README to include submission instructions. Add challenge-cli.py to make uploading via the Dyff API easier.

Dockerfile

@@ -8,6 +8,12 @@ RUN python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
 
 WORKDIR /app/
 
+# Download models during build instead of copying from local
+COPY scripts/model_download.bash /tmp/model_download.bash
+RUN python3 -m pip install --no-cache-dir huggingface-hub && \
+    bash /tmp/model_download.bash && \
+    rm /tmp/model_download.bash
+
 COPY requirements.cpu.txt ./
 RUN python3 -m pip install --no-cache-dir -r ./requirements.cpu.txt
 
@@ -15,7 +21,6 @@ COPY requirements.torch.cpu.txt ./
 RUN python3 -m pip install --no-cache-dir -r ./requirements.torch.cpu.txt
 
 COPY app ./app
-COPY models ./models
 COPY main.py ./
 
 EXPOSE 8000

README.md

@@ -2,9 +2,127 @@
 license: apache-2.0
 ---
 
-# ML Inference Service
+# Example Submission for the SAFE: Image Edit Detection and Localization Challenge 2025
 
-FastAPI service for serving ML models over HTTP. Comes with ResNet-18 for image classification out of the box, but you can swap in any model you want.
+This project provides a starting point for implementing a submission to the [SAFE: Image Edit Detection and Localization Challenge 2025](https://app.dyff.io/challenges/dc509a8c771b492b90c43012fde9a04f). You do not need to use this code to participate in the challenge.
+
+# How to participate
+
+Visit the [challenge home page](https://app.dyff.io/challenges/dc509a8c771b492b90c43012fde9a04f) and sign up using the linked registration form.
+
+# How to make a submission
+
+The infrastructure for the challenge runs on [DSRI's Dyff platform](https://app.dyff.io). Submissions to the challenge must be in the form of a containerized web service that serves a simple JSON HTTP API. If you're comfortable building a Docker image yourself, you can create a submission from a built image using the [Dyff client](https://docs.dyff.io/python-api/dyff.client/). Or, you can create a Docker [HuggingFace Space](https://huggingface.co/new-space?sdk=docker) and create submissions from the space using a [webform](https://challenge.dyff.io/submit).
+
+## General considerations
+
+* Your submission will run **without Internet access** during evaluation. All of the files required to run your submission must be packaged along with it. You can either include files in the Docker image, or upload the files as a separate package and mount them in your application container during execution.
+
+## Submitting a Docker HuggingFace Space
+
+These are the steps to prepare a HF Space for making submissions to the challenge:
+
+1. Create a new HuggingFace [**Organization**](https://huggingface.co/organizations/new) (**not a user account**) for your challenge team. **The length of your combined Organization name + Space name must be less than 47 characters** due to a limitation of the HuggingFace API.
+2. Add the [official SAFE Challenge user account](https://huggingface.co/safe-challenge-2025-submissions) as a Member of your organization with `read` permissions. **Make sure you are adding the correct user account;** the account name is `safe-challenge-2025-submissions`. This allows our infrastructure to pull the Docker image built by your Space.
+3. Create a new `Space` within your `Organization`. The Space must use the [Docker SDK](https://huggingface.co/new-space?sdk=docker). **Private Spaces are OK and they will work with the submission process** because you granted `read` access in the previous step. **The length of your combined Organization name + Space name must be less than 47 characters** due to a limitation of the HuggingFace API.
+4. Create a file called `DYFF_TEAM` in the root directory of your HF Space. The contents of the file should be your Team ID (not your Account ID); your Team ID is a 32-character hexidecimal string. This file allows our infrastructure to verify that your Team controls this HF Space.
+5. Create a `Dockerfile` in your Space that builds your challenge submission image.
+6. Run the Space; this will build the Docker image.
+7. When you're ready to submit, use the [submission webform](https://challenge.dyff.io/submit) and enter the URL of your Space and the branch that you want to submit.
+
+### Handling large models
+
+There is a size limitation on Space repositories. If your submission contains large files (such as neural network weights), it may be too large to store in the space. In this case, you need to fetch your files from somewhere else **during the Docker build process**.
+
+This means that your Dockerfile should contain something like this:
+
+```
+COPY download-my-model.sh ./ 
+RUN ./download-my-model.sh
+```
+
+### Handling private models
+
+If access credentials are required to download your model files, you should provide them using the [Secrets feature](https://huggingface.co/docs/hub/spaces-overview#managing-secrets) of HuggingFace Spaces. **Do not hard-code credentials in your Dockerfile or anywhere else in your Space or Organization!**
+
+Access the secrets as described in the [Secrets > Buildtime section](https://huggingface.co/docs/hub/spaces-sdks-docker#secrets). Remember that you can't download files at run-time because your system will not have access to the Internet.
+
+## Submitting using the Dyff API
+
+If you need more flexibility than HuggingFace Spaces allow, or if you just prefer to work with CLI tools and scripts, you can create submissions using the Dyff API.
+
+In the [terminology of Dyff](https://docs.dyff.io/tutorial/), the thing that you're submitting is an `InferenceService`. You can think of an `InferenceService` as a recipe for spinning up a Docker container that runs an HTTP server that serves an inference API. To create a new submission, you need to upload the Docker image that the service should run, and, optionally, a volume of files such as neural network weights that will be mounted in the container.
+
+### Install the Dyff SDK
+
+You need Python 3.10+ (3.12 recommended). We recommend you install into a virtual environment:
+
+```
+python3 -m venv venv
+source venv/bin/activate
+```
+
+Then, install the Dyff SDK:
+
+```
+python3 -m pip install --upgrade dyff
+```
+
+### Prepare the submission data
+
+Before creating a submission, you need to build the Docker image you want to submit locally. For example, running the `make docker-build` command in this repository will build a Docker image in your local Docker daemon with the name `safe-challenge-2025/example-submission:latest`. You can check that the image exists using the `docker images` command:
+
+```
+$ docker images
+REPOSITORY                                                                  TAG                                                                IMAGE ID       CREATED        SIZE
+safe-challenge-2025/example-submission                                      latest                                                             b86a46d856f0   3 hours ago    1.86GB
+...
+```
+
+If your submission includes large data files such as neural network weights, we recommend that you upload these separately from the Docker image and then arrange for them to be mounted in the running container at run-time. You can upload a local directory recursively to the Dyff platform. Once uploaded, you will get the ID of a Dyff `Model` resources that you can reference 
+
+### Use the `challenge-cli` tool
+
+This repository contains a CLI script that simplifies the submission process. Usage is like this:
+
+```
+$ python3 challenge-cli.py
+Usage: challenge-cli.py [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  submit
+  upload-submission
+```
+
+You create a new submission in two steps. First, upload the submission files:
+
+```
+DYFF_API_TOKEN=<your token> python3 challenge-cli.py upload-submission [OPTIONS]
+```
+
+Notice that we're providing an access token via an environment variable.
+
+This command creates a Dyff `Artifact` resource corresponding to your Docker image, an optional Dyff `Model` resource containing your uploaded model files, and a Dyff `InferenceService` resource that references the `Artifact` and the `Model`.
+
+You need to specify your Account ID and the names and paths of the resources you want to upload. You can also use the `--artifact` and `--model` flags to provide the IDs of an `Artifact` or `Model` that already exists instead of creating a new one. For example, if you always use the same Docker image but you mount different model weights in it for different submissions, you can create the Docker image `Artifact`  once, and then reference its ID in `--artifact`.
+
+After uploading the submission files, you create the actual `Submission` resource with:
+
+```
+DYFF_API_TOKEN=<your token> python3 challenge-cli.py submit [OPTIONS]
+```
+
+When submitting, you reference the ID of the `InferenceService` you created in the previous step. You also provide your Account ID, your Team ID for the challenge, and the Task ID that you're submitting to.
+
+
+# How to implement a detector
+
+To implement a new detector that you can submit to the challenge, you need to implement an HTTP server that serves the required JSON API for inference requests. This repository contains a template that you can use as a starting point for implementing a detector in Python. You should be able to adapt this template easily to support common model formats such as neural networks built with PyTorch.
+
+You are also free to build detectors with any other technologies and software stacks that you want, but you may have to figure out packaging on your own.
 
 ## Quick Start
 
@@ -70,34 +188,36 @@ Example response:
 
 ```
 example-submission/
-├── main.py                      # Entry point
+├── main.py                        # Entry point
 ├── app/
 │   ├── core/
-│   │   ├── app.py               # <= INSTANTIATE YOUR DETECTOR HERE
-│   │   └── logging.py           # Logging setup
+│   │   ├── app.py                 # <= INSTANTIATE YOUR DETECTOR HERE
+│   │   └── logging.py           
 │   ├── api/
-│   │   ├── models.py            # Request/response schemas
-│   │   ├── controllers.py       # Business logic
+│   │   ├── models.py              # Request/response schemas
+│   │   ├── controllers.py         # Business logic
 │   │   └── routes/
-│   │       └── prediction.py    # POST /predict
+│   │       └── prediction.py      # POST /predict
 │   └── services/
-│       ├── base.py              # <= YOUR DETECTOR IMPLEMENTS THIS INTERFACE
-│       └── inference.py         # Example service based on ResNet-18
+│       ├── base.py                # <= YOUR DETECTOR IMPLEMENTS THIS INTERFACE
+│       └── inference.py           # Example service based on ResNet-18
 ├── models/
 │   └── microsoft/
-│       └── resnet-18/           # Model weights and config
+│       └── resnet-18/             # Model weights and config
 ├── scripts/
-│   ├── model_download.bash
-│   ├── generate_test_datasets.py
-│   └── test_datasets.py
+│   ├── model_download.bash        # Downloads resnet-18
+│   ├── generate_test_datasets.py  # Creates test datasets
+│   └── test_datasets.py           # Runs inference on test datasets
 ├── Dockerfile
-├── .env.example                 # Environment config template
-├── cat.json                     # An example /predict request object
+├── .env.example                   # Environment config template
+├── cat.json                       # An example /predict request object
 ├── makefile
-├── prompt.sh                    # Script that makes a /predict request
-├── requirements.in              
-├── requirements.txt
-├── response.json                # An example /predict response object
+├── prompt.sh                      # Script that makes a /predict request
+├── requirements.cpu.in              
+├── requirements.cpu.txt
+├── requirements.torch.cpu.in    
+├── requirements.torch.cpu.in    
+├── response.json                  # An example /predict response object
 └──
 ```
 
@@ -169,6 +289,16 @@ models/
         └── (other files)
 ```
 
+## GPU inference
+
+The default configuration in this repo runs the model on CPU and does not contain the necessary dependencies for using GPUs.
+
+To enable GPU inference, you need to:
+
+1. Base your Docker image on an image that contains the CUDA system packages
+2. Install the GPU version of PyTorch and its dependencies
+3. Use the `.to()` function as necessary in the `load_model()` and `predict()` functions to move model weights and input data to and from the CUDA device
+
 ## Configuration
 
 Settings are managed via environment variables or a `.env` file. See `.env.example` for all available options.
@@ -196,26 +326,6 @@ export MODEL_NAME="google/vit-base-patch16-224"
 uvicorn main:app --reload
 ```
 
-## Deployment
-
-**Development:**
-```bash
-uvicorn main:app --reload
-```
-
-**Production:**
-```bash
-gunicorn main:app -w 4 -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000
-```
-
-The service runs on CPU by default. For GPU inference, install CUDA-enabled PyTorch and modify your service to move tensors to the GPU device.
-
-**Docker:**
-- Multi-stage build keeps the image small
-- Runs as non-root user (`appuser`)
-- Python dependencies installed in user site-packages
-- Model files baked into the image
-
 ## What Happens When You Start the Server
 
 ```

challenge-cli.py

@@ -0,0 +1,293 @@
+# SPDX-FileCopyrightText: 2025 UL Research Institutes
+# SPDX-License-Identifier: Apache-2.0
+
+import functools
+import time
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+
+import click
+
+from dyff.client import Client
+from dyff.schema.platform import *
+from dyff.schema.requests import *
+
+from app.api.models import PredictionResponse
+
+# ----------------------------------------------------------------------------
+
+
+def _wait_for_status(
+    get_entity_fn, target_status: str | list[str], *, timeout: timedelta
+) -> str:
+    if isinstance(target_status, str):
+        target_status = [target_status]
+    then = datetime.now(timezone.utc)
+    while True:
+        try:
+            status = get_entity_fn().status
+            if status in target_status:
+                return status
+        except Exception as ex:
+            if hasattr(ex, "status"):
+                if ex.status != 404:  # pyright: ignore
+                    raise
+            elif hasattr(ex, "status_code"):
+                if ex.status_code != 404:  # pyright: ignore
+                    raise
+            else:
+                raise
+        if (datetime.now(timezone.utc) - then) >= timeout:
+            break
+        time.sleep(5)
+    raise AssertionError("timeout")
+
+
+def _common_options(f):
+    @click.option(
+        "--account",
+        type=str,
+        required=True,
+        help="Your account ID",
+        metavar="ID",
+    )
+    @functools.wraps(f)
+    def wrapper(*args, **kwargs):
+        return f(*args, **kwargs)
+    return wrapper
+
+
+@click.group()
+def cli():
+    pass
+
+
+@cli.command()
+@_common_options
+@click.option(
+    "--name",
+    type=str,
+    required=True,
+    help="The name of your detector model. For display and querying purposes only.",
+)
+@click.option(
+    "--image",
+    type=str,
+    default=None,
+    help="The Docker image to upload (e.g., 'some/image:latest')."
+    " Must exist in your local Docker deamon."
+    " Required if --artifact is not specified.",
+)
+@click.option(
+    "--endpoint",
+    type=str,
+    default="predict",
+    help="The endpoint to call on your service to make a prediction.",
+)
+@click.option(
+    "--volume",
+    type=click.Path(exists=True, file_okay=False, readable=True, resolve_path=True, path_type=Path),
+    default=None,
+    help="A local directory path containing files to upload and mount in the running Docker container."
+    " You should use this if your submission includes large files like neural network weights."
+)
+@click.option(
+    "--volume-mount",
+    type=click.Path(exists=False, path_type=Path),
+    default=None,
+    help="The path to mount your uploaded directory in the running Docker container."
+    " Must be an absolute path."
+    " Required if --volume is specified.")
+@click.option(
+    "--artifact",
+    "artifact_id",
+    type=str,
+    default=None,
+    help="The ID of the Artifact (i.e., Docker image) to use in the submission, if it already exists."
+    " You can pass the artifact.id from a previous invocation.",
+    metavar="ID",
+)
+@click.option(
+    "--model",
+    "model_id",
+    type=str,
+    default=None,
+    help="The ID of the Model (i.e., neural network weights) to use in the submission, if it already exists."
+    " You can pass the model.id from a previous invocation.",
+    metavar="ID",
+)
+def upload_submission(
+    account: str,
+    name: str,
+    image: str | None,
+    endpoint: str,
+    volume: Path | None,
+    volume_mount: Path | None,
+    artifact_id: str | None,
+    model_id: str | None,
+) -> None:
+    dyffapi = Client()
+
+    # Upload the image
+    if artifact_id is None:
+        # Create an Artifact resource
+        click.echo("creating Artifact ...")
+        artifact = dyffapi.artifacts.create(ArtifactCreateRequest(account=account))
+        click.echo(f"artifact.id: \"{artifact.id}\"")
+        _wait_for_status(
+            lambda: dyffapi.artifacts.get(artifact.id),
+            "WaitingForUpload",
+            timeout=timedelta(seconds=30),
+        )
+
+        # Push the image from the local Docker daemon
+        click.echo("pushing Artifact ...")
+        dyffapi.artifacts.push(artifact, source=f"docker-daemon:{image}")
+        time.sleep(5)
+
+        # Indicate that we're done pushing
+        dyffapi.artifacts.finalize(artifact.id)
+        _wait_for_status(
+            lambda: dyffapi.artifacts.get(artifact.id),
+            "Ready",
+            timeout=timedelta(seconds=30),
+        )
+
+        click.echo("... done")
+    else:
+        artifact = dyffapi.artifacts.get(artifact_id)
+        assert artifact is not None
+
+    model: Model | None = None
+    if model_id is None:
+        if volume is not None:
+            if volume_mount is None:
+                raise click.UsageError("--volume-mount is required when --volume is used")
+            
+            click.echo("creating Model from local directory ...")
+
+            model = dyffapi.models.create_from_volume(
+                volume, name="model_volume", account=account, resources=ModelResources()
+            )
+            click.echo(f"model.id: \"{model.id}\"")
+            _wait_for_status(
+                lambda: dyffapi.models.get(model.id),
+                "WaitingForUpload",
+                timeout=timedelta(seconds=30),
+            )
+
+            click.echo("uploading Model ...")
+            dyffapi.models.upload_volume(model, volume)
+            _wait_for_status(
+                lambda: dyffapi.models.get(model.id),
+                "Ready",
+                timeout=timedelta(seconds=30),
+            )
+
+            click.echo("... done")
+        else:
+            model = None
+    else:
+        model = dyffapi.models.get(model_id)
+        assert model is not None
+
+    # Create a runnable InferenceService
+    if volume_mount is not None:
+        if model is None:
+            raise click.UsageError("--volume-mount requires --volume or --model")
+        if not volume_mount.is_absolute():
+            raise click.UsageError("--volume-mount must be an absolute path")
+        volumeMounts=[
+            VolumeMount(
+                kind=VolumeMountKind.data,
+                name="model",
+                mountPath=volume_mount,
+                data=VolumeMountData(
+                    source=EntityIdentifier.of(model),
+                ),
+            ),
+        ]
+    else:
+        volumeMounts = None
+
+    # Don't change this
+    service_request = InferenceServiceCreateRequest(
+        account=account,
+        name=name,
+        model=None,
+        runner=InferenceServiceRunner(
+            kind=InferenceServiceRunnerKind.CONTAINER,
+            imageRef=EntityIdentifier.of(artifact),
+            resources=ModelResources(),
+            volumeMounts=volumeMounts,
+        ),
+        interface=InferenceInterface(
+            endpoint=endpoint,
+            outputSchema=DataSchema.make_output_schema(PredictionResponse),
+        ),
+    )
+    click.echo("creating InferenceService ...")
+    service = dyffapi.inferenceservices.create(service_request)
+    click.echo(f"service.id: \"{service.id}\"")
+    click.echo("... done")
+
+
+@cli.command()
+@_common_options
+@click.option(
+    "--task",
+    "task_id",
+    type=str,
+    required=True,
+    help="The Task ID to submit to.",
+    metavar="ID",
+)
+@click.option(
+    "--team",
+    "team_id",
+    type=str,
+    required=True,
+    help="The Team ID making the submission.",
+    metavar="ID",
+)
+@click.option(
+    "--service",
+    "service_id",
+    type=str,
+    required=True,
+    help="The InferenceService ID to submit.",
+    metavar="ID",
+)
+@click.option(
+    "--challenge",
+    "challenge_id",
+    type=str,
+    default="dc509a8c771b492b90c43012fde9a04f",
+    help="The Challenge ID to submit to.",
+    metavar="ID",
+)
+def submit(account: str, task_id: str, team_id: str, service_id: str, challenge_id: str) -> None:
+    dyffapi = Client()
+
+    challenge = dyffapi.challenges.get(challenge_id)
+    challengetask = challenge.tasks[task_id]
+
+    team = dyffapi.teams.get(team_id)
+
+    service = dyffapi.inferenceservices.get(service_id)
+
+    submission = dyffapi.challenges.submit(
+        challenge.id,
+        challengetask.id,
+        SubmissionCreateRequest(
+            account=account,
+            team=team.id,
+            submission=EntityIdentifier(kind="InferenceService", id=service.id),
+        ),
+    )
+    click.echo(submission.model_dump_json(indent=2))
+    click.echo(f"submission.id: \"{submission.id}\"")
+
+
+if __name__ == "__main__":
+    cli(show_default=True)

upload_model.py

@@ -1,93 +0,0 @@
-# SPDX-FileCopyrightText: 2025 UL Research Institutes
-# SPDX-License-Identifier: Apache-2.0
-
-import sys
-import time
-from pathlib import Path
-
-import click
-
-from dyff.client import Client
-from dyff.schema.platform import *
-from dyff.schema.requests import *
-
-from app.api.models import PredictionResponse
-
-# ----------------------------------------------------------------------------
-
-WORKDIR = Path(__file__).resolve().parent
-
-
-@click.command()
-@click.option(
-    "--account",
-    type=str,
-    required=True,
-    help="Your account ID",
-)
-@click.option(
-    "--name",
-    type=str,
-    required=True,
-    help="The name of your detector model. For display and querying purposes only.",
-)
-@click.option(
-    "--image",
-    type=str,
-    required=True,
-    help="The Docker image to upload. Must exist in your local Docker deamon.",
-)
-@click.option(
-    "--endpoint",
-    type=str,
-    default="predict",
-    help="The endpoint to call on your model to make a prediction.",
-)
-def main(account: str, name: str, image: str, endpoint: str) -> None:
-    dyffapi = Client()
-
-    # You can set these to a known ID to skip that step
-    artifact_id = None
-    service_id = None
-
-    # Upload the image
-    if artifact_id is None:
-        # Create an Artifact resource
-        artifact = dyffapi.artifacts.create(ArtifactCreateRequest(account=account))
-        click.echo(f"artifact_id = \"{artifact.id}\"")
-        time.sleep(5)
-        # Push the image from the local Docker daemon
-        dyffapi.artifacts.push(artifact, source=f"docker-daemon:{image}")
-        time.sleep(5)
-        # Indicate that we're done pushing
-        dyffapi.artifacts.finalize(artifact.id)
-    else:
-        artifact = dyffapi.artifacts.get(artifact_id)
-        assert artifact is not None
-
-    # Create a runnable InferenceService
-    if service_id is None:
-        # Don't change this
-        service_request = InferenceServiceCreateRequest(
-            account=account,
-            name=name,
-            model=None,
-            runner=InferenceServiceRunner(
-                kind=InferenceServiceRunnerKind.CONTAINER,
-                imageRef=EntityIdentifier.of(artifact),
-                resources=ModelResources(),
-            ),
-            interface=InferenceInterface(
-                endpoint=endpoint,
-                outputSchema=DataSchema.make_output_schema(PredictionResponse),
-            ),
-        )
-        service = dyffapi.inferenceservices.create(service_request)
-        click.echo(f"service_id = \"{service.id}\"")
-    else:
-        service = dyffapi.inferenceservices.get(service_id)
-        assert service is not None
-
-
-if __name__ == "__main__":
-    main()