everbench / API

Upload, remove, and backtest models.

All requests need X-API-Key. Model uploads also need an HMAC signature in X-Everbench-Artifact-Signature. Ask maxhalford25@gmail.com for both the API key and the signing key.

Upload a model

POST /api/tasks/<task>/models accepts multipart fields model, model_id, owner, and class_definition. Models are serialized with cloudpickle.

Save this standard-library script as sign_model.py in your model project:

import hashlib, hmac, os, sys
from pathlib import Path

payload = Path(sys.argv[1]).read_bytes()
digest = hashlib.sha256(payload).hexdigest().encode()
print(hmac.new(os.environ["EVERBENCH_MODEL_SIGNING_KEY"].encode(), digest, hashlib.sha256).hexdigest())

signature="$(python sign_model.py model.pkl)"
curl -X POST http://127.0.0.1:8000/api/tasks/dummy/models \
  -H 'X-API-Key: …' \
  -H "X-Everbench-Artifact-Signature: $signature" \
  -F 'model=@model.pkl' \
  -F 'model_id=my-model' \
  -F 'owner=your-name' \
  -F 'class_definition=<your_model.py'

Model protocol

An upload is a cloudpickled Python object. Implement the prediction interface for the task type:

regression — predict_one(event_id: str, event: dict[str, Any]) -> float.
binary classification — predict_proba_one(event_id: str, event: dict[str, Any]) -> dict[bool, float]; predict_one(event_id: str, event: dict[str, Any]) -> bool is also accepted.
multiclass classification — predict_proba_one(event_id: str, event: dict[str, Any]) -> dict[str, float]; predict_one(event_id: str, event: dict[str, Any]) -> str is also accepted.
clustering — predict_one(event_id: str, event: dict[str, Any]) -> object returns a cluster identifier.
anomaly detection — score_one(event_id: str, event: dict[str, Any]) -> float; predict_one(event_id: str, event: dict[str, Any]) -> float is also accepted.

learn_one(event_id: str, event: dict[str, Any], label: object) -> None is optional and receives each label when it becomes available.

Everbench always passes the raw accepted event and its ID. Models own feature extraction and can ignore either input when it is not useful.

Upload validation deep-copies the model, predicts recent labelled examples, and then learns when supported. This checks that the uploaded model works as expected without changing the uploaded object.

Delete a model

DELETE /api/tasks/<task>/models/<model> removes a model and its metrics. Uploading afterward creates a fresh registration, with independent metrics.

Backtest a model

POST /api/tasks/<task>/backtest accepts a signed model pickle and archive_sha256. It does not register or change the model. The response contains final metrics and prediction/learning durations.

curl -X POST http://127.0.0.1:8000/api/tasks/dummy/backtest \
  -H 'X-API-Key: …' \
  -H "X-Everbench-Artifact-Signature: $signature" \
  -F 'model=@model.pkl' \
  -F 'archive_sha256=the-SHA-256-in-the-archive-link'

Python environment

GET /api/environment returns the server Python and package versions.