Task Template API Reference¶

Tracked templates demonstrate the task-project layout. They are not a production task catalog or complete worked examples. Use Examples And Templates to choose the right asset, and inspect Rocket Booster Recovery for a complete runnable integration.

Template Runner¶

runner ¶

Offline fixture smoke runner for the template task project.

create_task_runner ¶

create_task_runner(
    task_project: TaskProject,
) -> FakeWorkflowFixtureTaskRunner

Task project entrypoint that constructs the template fixture runner.

Toy Math Runner¶

runner ¶

Offline fixture smoke runner for the toy math task project.

create_task_runner ¶

create_task_runner(
    task_project: TaskProject,
) -> FakeWorkflowFixtureTaskRunner

Task project entrypoint that constructs the template fixture runner.

SAM Optimizer Reference Evaluation¶

evaluator ¶

SAM tiered Pareto evaluation contract for the task project.

SamParetoTieredEvaluation ¶

Task-local promotion/ranking contract for tiered Pareto metrics.

create_evaluation ¶

create_evaluation() -> SamParetoTieredEvaluation

Factory entrypoint that constructs the task-local evaluation contract.

SAM Optimizer Reference Evaluation Runner¶

run ¶

Run the SAM task's canonical tiered evaluation protocol.

The Praxist agent still owns the research idea and optimizer implementation. This is the only public evaluation entrypoint that peers should call. The script owns the repetitive mechanics: run T1/T2/T3 in order, apply task-local promotion gates, preserve raw benchmark JSON/logs, and print a compact summary that the agent can use for its notebook and finding. The benchmark runner under assets/harness/benchmark is an internal implementation detail.

task_root_from_script ¶

task_root_from_script() -> Path

Return the task project root for this script.

default_max_tier ¶

default_max_tier() -> str

Return the default maximum tier, honoring smoke-run env overrides.

parse_args ¶

parse_args(
    argv: list[str] | None = None,
) -> argparse.Namespace

Parse command-line arguments.

variant_name_from_path ¶

variant_name_from_path(path: str) -> str

Return the benchmark's custom variant stem for a variant path.

tier_range ¶

tier_range(min_tier: str, max_tier: str) -> list[str]

Return the inclusive tier sequence from min_tier to max_tier.

load_baselines ¶

load_baselines(
    path: Path,
) -> dict[str, dict[str, dict[str, Any]]]

Load curated baseline rows keyed by optimizer and dataset.

metric_mean ¶

metric_mean(
    payload: dict[str, Any], metric_name: str
) -> float | None

Extract a numeric mean from a metric object.

marker_is_true ¶

marker_is_true(value: Any) -> bool

Interpret explicit evaluator control markers without guessing.

dataset_metric ¶

dataset_metric(
    result: dict[str, Any], dataset: str, metric_name: str
) -> float | None

Extract a dataset-level metric mean from a benchmark result.

dataset_metric_stat ¶

dataset_metric_stat(
    result: dict[str, Any],
    dataset: str,
    metric_name: str,
    stat: str,
) -> float | None

Extract a dataset-level metric statistic such as mean or max.

tier_has_failure ¶

tier_has_failure(result: dict[str, Any]) -> str | None

Return a failure reason if any dataset visibly failed.

gate_decision ¶

gate_decision(
    tier: str,
    result: dict[str, Any],
    baselines: dict[str, dict[str, dict[str, Any]]],
) -> dict[str, Any]

Apply the task-local T1/T2/T3 gate for one benchmark result.

result_path_for ¶

result_path_for(
    output_dir: Path, variant_name: str, tier: str
) -> Path

Return the run_benchmark multi-result path for a custom variant.

summary_path_for ¶

summary_path_for(
    output_dir: Path, variant_name: str
) -> Path

Return the compact summary path for this tool invocation.

run_benchmark_tier ¶

run_benchmark_tier(
    args: Namespace, tier: str, variant_name: str
) -> dict[str, Any]

Run one benchmark tier or reuse an existing tier JSON.

observed_train_seconds ¶

observed_train_seconds(
    result: dict[str, Any],
) -> float | None

Return total observed train time across completed evaluation cells.

summarize_metrics ¶

summarize_metrics(
    result: dict[str, Any] | None,
) -> dict[str, Any]

Return the small metric subset agents normally need.

attach_maturity_ratios ¶

attach_maturity_ratios(
    metrics_summary: dict[str, Any],
    *,
    tier: str,
    min_tier: str = REFERENCE_MIN_TIER,
    max_tier: str = REFERENCE_MAX_TIER,
) -> None

Attach generic Praxist maturity ratios using this task's tier/cell semantics.

sync_maturity_metadata_to_benchmark_result ¶

sync_maturity_metadata_to_benchmark_result(
    result: dict[str, Any],
    metrics_summary: dict[str, Any],
    *,
    tier: str,
    result_path: str,
) -> None

Mirror peer-facing maturity metadata into the raw benchmark JSON.

run_tiered_evaluation ¶

run_tiered_evaluation(
    args: Namespace,
) -> tuple[int, dict[str, Any]]

Run the configured tier sequence and return process exit code plus summary.

compact_stdout_summary ¶

compact_stdout_summary(
    summary: dict[str, Any],
) -> dict[str, Any]

Return the concise JSON printed to the agent context.

main ¶

main(argv: list[str] | None = None) -> int

CLI entrypoint.

Template Evaluation¶

evaluator ¶

Placeholder task-template evaluation contract.

TemplateParetoTieredEvaluation ¶

Template task-local evaluation contract for task authors.

create_evaluation ¶

create_evaluation() -> TemplateParetoTieredEvaluation

Factory entrypoint that constructs a task-local evaluation contract.

Template Evaluation Runner¶

run ¶

Placeholder canonical evaluation entrypoint for task templates.

main ¶

main() -> None

Explain that real tasks must replace the placeholder evaluator.

Toy Math Evaluation¶

evaluator ¶

Placeholder toy-math evaluation contract.

ToyMathParetoTieredEvaluation ¶

Template task-local evaluation contract for toy-math findings.

create_evaluation ¶

create_evaluation() -> ToyMathParetoTieredEvaluation

Factory entrypoint that constructs a task-local evaluation contract.

Toy Math Evaluation Runner¶

run ¶

Placeholder canonical evaluation entrypoint for the toy task template.

main ¶

main() -> None

Explain that real task evaluators replace this placeholder.

Configuration Source¶

The executable API objects above are the purpose of this reference page. Configuration, evidence maturity, Deep Innovation Gate (DIG), Quality-Diversity (QD), Gems, lane routing, artifact ownership, and scheduler requirements for templates are defined once in Task Projects. The tracked task descriptors under templates/tasks/ are validated fixtures of that contract, not a second schema definition.