Revision history for Langertha-Holmgang

0.002     2026-11-19 18:21:03Z
  - cpanfile declares the core modules the code loads but did not list:
    Archive::Tar, File::Temp, JSON::PP, Text::ParseWords
  - Scenarios: a case may carry scenario (an id of [a-z0-9-]); cases with
    the same scenario are variants of one task. The report keeps the field,
    and every verdict of a suite with scenarios carries scenarios
    { total, solved, rate, unsolved }: a scenario is solved only when every
    variant passed in all k repetitions (AppWorld's scenario goal
    completion). The text notes it per suite; it does not decide ok
  - journald speaks about itself: journalctl -u systemd-journald shows at
    the boot time "Journal started" and "Runtime Journal (/run/log/journal/
    ID) is …, max …, … free." (or "System Journal" in /var), and a restart
    logs the new process's journal with its limit and the flush, after
    "Started". journalctl --header lists path, state and time span of every
    journal file; -t, SYSLOG_IDENTIFIER=, _COMM= (cut to 15 characters as
    the kernel does) and _SYSTEMD_UNIT= filter by sender. --list-boots
    starts at the boot time, as uptime -s
  - RouterOS simulator: /ip pool used lists the addresses the DHCP leases
    hold in their server's pool (print, where, count-only; read only)
  - Monitoring simulator: Prometheus answers /api/v1/targets (static
    targets of the running configuration; health from the service's
    scrape_health or the connection probe) and /api/v1/query for a single
    selector, optionally with time() - before and a comparison after it
    (up from the targets, other series only from the service's metrics).
    Alertmanager answers /api/v2/alerts, /api/v2/silences and
    /api/v2/receivers from the same data amtool uses. Anything else stays
    a gap
  - Shell filters: python3 -m json.tool behind a pipe formats JSON like
    the real module (key order kept, indent 4, \uXXXX), without a sandbox
  - PostgreSQL simulator: the near relatives of the slot query answer truly
    instead of ending as a gap -- plugin, database and confirmed_flush_lsn
    of a physical slot are NULL, a WAL difference to a NULL LSN is NULL,
    pg_stat_replication is empty while no slot is active, DROP REPLICATION
    SLOT is the syntax error of an SQL session, and psql takes -Atc
  - A wget that no observation matches is answered from the case's curl
    observation for the same request, in wget's words, when the document
    goes to standard output (wget -qO- URL). Before, such a call fell
    through to the simulator and could get "Connection refused" for a
    service the case describes as running (ADR 0013)
  - Linux simulator: the world is closed on a device's own network. An
    address in a directly connected network that no device carries and the
    state names nowhere does not exist: /dev/tcp, nc, wget and curl fail
    with "No route to host" and ping with "Destination Host Unreachable",
    without a sim_gap. Named addresses and everything beyond a gateway
    answer as before. A device reaches its own networks even when the
    case lists no routes (ADR 0012)
  - `holmgang run --sandbox docker|podman [--sandbox-image IMAGE]`: a real
    program the Linux simulator does not model (python3 -c, perl -e, an awk
    script past the filter grammar) runs in a locked-down container on a
    copy of the device's files instead of ending as a sim_gap; what it wrote
    comes back into the state. Opt-in per run, never in loading or lint
    (ADR 0011). New: Langertha::Holmgang::Sandbox::Docker, the image in
    share/sandbox/Dockerfile
  - A call whose programs ran for real carries `sandbox` in the report, and
    `generator.sandbox` names runtime, image and image id; both only when a
    sandbox was used

0.001     2026-11-17 03:22:26Z
  - Initial release
  - Deterministic tool-use benchmarks for language models: a case gives the
    model a task and tools in a simulated world, and the rules grade the
    end state of that world, never the wording of the answer
  - Suite format 2 (suite.yaml plus cases/*.yaml) loads into objects that
    keep the values as written; every case is linted on load, including two
    mock runs, and a case that fails lint does not load
  - Worlds with tool dispatch, observations, handlers and Kolon output, and
    simulators for the device types linux (bash grammar and many services),
    routeros and ios, plus the answer world kueche; a simulator plugin
    contract (Role::Simulator, Role::AnswerWorld, registry); a simulator
    that fails in Perl never shows its error to the model, the call answers
    a generic tool error and the gap is recorded
  - Grader with rules on the end state and on the calls, and seven statuses:
    passed, failed, uncertain (a simulation gap was hit), turn_limit, and
    the not-measured error, context_exceeded and no_vision
  - Runner over any Langertha engine through Target, with turn limit and
    tokens, finish reason and calls per turn; Langertha::Engine::Virtual
    mock models run a case without a network; Haystack generates long,
    deterministic inputs
  - Target classifies a failed turn as context_exceeded, no_vision or error
    and scrubs its error text: the engine's key, credentials in URLs,
    Authorization and Bearer values, provider key shapes, die locations and
    host paths never reach a result
  - Report: the JSON report (holmgang_report 1), written, read and merged
    without double counting
  - Evaluation: rates per topic with case weights, overall as the mean of
    topics, curves per length tier, verdicts against a threshold with pass^k
    and hard cases, and the comparison of targets as text, Markdown or JSON
  - bin/holmgang with the commands lint, run and report. lint loads exactly
    as a run does; run takes any Langertha engine by name (--model, --url,
    --option) or the network-free mock-solution and mock-text, with
    --variant, --repeat and --report, and no key on the command line;
    report evaluates report files with thresholds from --suite. Exit 0 ok,
    1 refused, verdict not ok or no case answered by the target, 2 usage or
    setup error. Errors on stderr carry no Perl location
  - Six invented example suites under share/suites: demo-linux, demo-lamp,
    demo-haystack, demo-sehen, demo-roost and demo-beacons, found with
    File::ShareDir::dist_dir('Langertha-Holmgang').'/suites'
  - Documentation: an overview in Langertha::Holmgang, the suite format 2
    reference in ::Loader, rules and statuses in ::Grader, the simulator
    contract in ::Role::Simulator, and a README with a quick start
  - Simulator documentation (Sim::Linux and its parts, RouterOS, IOS,
    Kueche, Shell, Shell::Filter) is now in English
  - Haystack refusals are Langertha::Holmgang::Error; Structure no longer
    re-wraps them (k29)
  - A when path that cannot be evaluated no longer dies out of World->call;
    the entry does not fit (k39)
  - Suite->checksum is now Suite->digest, and Loader->list carries digest,
    the same name as the report's suite.digest (k30)
  - Loader->list leaves out suites whose format is a name rather than a
    number (another kind of corpus) (k33)
  - Case->report_meta builds the case meta Report->add_run takes;
    Case->tier returns undef in list context too (k36)
  - Tool parameters: const is checked per call, a non-scalar const is
    refused at load (k31)
  - Runner refuses an engine with its own system_prompt instead of sending
    a second system message (k37)
  - New answer world Sim::Sehen (type sehen): hand in what an image shows (graph, sequence, receipt fields, table, chart values, error finding, or report the image unreadable); stores raw and normalised answers (text, ISO dates, cents, compact IBAN), a repeated call replaces the earlier one (k27)
  - Images on user messages in suite format 2 (images: [{file, untrusted}]): PNG or JPEG inside the suite, at most 2 MiB, at most four per message; SHA-256 and dimensions go into the case and the suite digest. They reach the model only as Langertha content parts in the request; transcripts and reports keep {image, sha256, bytes} (k27)
  - The vision pre-check per target and suite: Langertha's ImageInput role, its model-metadata probe, and a sight test that catches a template silently dropping the image. A case with an image on a blind target ends no_vision (k27)
  - New example suite demo-sehen: an image on a user message, handed in through the answer world sehen (k27)
  - Tool parameters: type may be a list of types (type: [string, "null"]), checked at load and per call; a null in enum matches a null (k48)
  - Expectations: tolerance beside a numeric equals compares numbers within it, bounds included; refused at load without a numeric equals (k49)
  - Langertha::Holmgang::Batch: suites against targets and their variants, repeated, with a bounded number of concurrent requests per target; resumes an earlier report, cancels, scrubs and cuts run errors (k15)
  - Target: variant settings named and checked (enable_thinking, reasoning_effort, max_tokens, temperature, deadline); a per-request deadline on the engine's event loop (k15)
  - holmgang run: several variants per target (--variant NAME:K=V), --concurrency, --resume, checkpoints of the report while running, cancel on SIGINT/SIGTERM (k15)
  - New answer world roost (Langertha::Holmgang::Sim::Roost), an invented dragon roost, and the example suite demo-roost with two cases on it (k43)
  - New Langertha::Holmgang::Random: a seeded random source in pure Perl for generators, the same on every perl and platform (k18)
  - Cases and suites in a Perl DSL (use Langertha::Holmgang, ADR 0007): a suite.pl in a suite directory loads, lints and runs like a format 2 suite; it covers images, tolerance, tiers, weights and tags; generate seed => N gives the same cases for the same seed, and the loader can override the seed per suite (k18)
  - holmgang lint and holmgang run take a directory with suite.pl (k18)
  - New answer world denken (Langertha::Holmgang::Sim::Denken): one tool antwort_abgeben, a checker per Reasoning Gym task, only the full score counts as right; cai-platform's suite denken loads unchanged (k53)
  - Answer world uebersetzung (Langertha::Holmgang::Sim::Uebersetzung): hand in segments, report ambiguity, look up the glossary; hand-ins normalised and analysed (numbers in the target format, codes, placeholders, what is missing, length class) into the state (k54)
  - New answer world schichtfehler (Langertha::Holmgang::Sim::Schichtfehler): sources, recorded page fetches and browser console, reported findings matched to causes by file and line overlap; answers only from the state (k55)
  - holmgang export-yaml / Loader->export write a DSL suite as suite format 2 (canonical YAML, images copied); a DSL suite's digest is now its export's digest; DSL cases run in file-name order of their export (k19)
  - --seed ID=N on lint, run and export-yaml overrides a DSL suite's generate seeds (k51)
  - New example suite demo-beacons: the case DSL, cases from a seeded generate block (k50)
  - Format 2 takes the message part auszug: { datei, von, bis }: a range of a statute text under the suite's quellen/, inserted at load and digested as cai-platform does; the DSL constructor excerpt; the YAML export copies the fixtures (k52)
  - A code reference or other non-data value in a case's state is refused at load with its field path (k59)
  - A ctx-<N>k tag is no length tier: it names a context window, not an input length (k56)
  - Reports carry each case's difficulty; reports without it still read and merge (k57)
  - The evaluation shows curves per difficulty next to the tier curves, in text, Markdown and JSON (k57)
  - Simulators: restic answers for snapshots without paths and no longer adds them on --path; von_liste_nehmen no longer creates an empty liste (k58)
  - Verdicts carry unfinished (a cancelled or aborted run is never ok) and repetitions_error (repetitions where the target answered nothing; next to a measured repetition only reported, not deciding ok, ADR 0009 k42; a suite with none measured is never ok, k73), ported from cai-platform's verdicts (k58)
  - Acceptance 1 and 2 pass against cai-platform origin/main 993637ab: 36 suites, 708 cases, mock runs identical (k58)
  - New acceptance test t/acceptance_dsl.t: a suite written in the case DSL loads, matches its format 2 original case by case, gives identical mock runs, and its export-yaml does too (k20)
  - A suite.pl is compiled as UTF-8 source whether or not it says "use utf8", so non-ASCII text reaches the model as written instead of as Latin-1 mojibake; a suite.pl that is not valid UTF-8 is refused (k60, ADR 0007)
  - A suite in which no repetition reached the target (every result error, or none) gets a verdict that is never ok, shown NOT MEASURED, with or without a threshold; holmgang report now exits 1 for it as holmgang run does. Unmeasured repetitions next to a measured one are still only noted; context_exceeded and no_vision count as measured (k73, ADR 0008)
