Lingee BenchLingee Bench
An agent evaluation benchmark oriented to enterprise business tasks and processes, answering one core question: given a well-defined business context and resource constraints, can an agent deliver verifiable business outcomes in business systems. It organizes evaluation capability in a three-tier architecture: the foundational tier (general domain, 500 tasks), the business-process tier (enterprise management domain, 900 tasks — finance / supply chain / manufacturing, 300 each), and the organizational-collaboration tier (future extension). The benchmark consists of identically distributed groups — a 65-task public group (first batch open-sourced, reproducible offline) and the held-out group of 1,335 tasks — measuring contamination through controlled evaluation; tasks run offline in fully self-contained Docker environments and are scored through the deterministic-rule and LLM Judge dual channel.
Four Evaluation Suites
Click a card to expand task composition and difficulty distribution. Card data reflects the public group; see the whitepaper for the public/held-out contamination-control mechanism.
Measures foundational capability through verifiable professional work packages — multi-table reconciliation, supplier comparison, knowledge-base retrieval, document checking, schedule handling — across ten categories; the scoring target is on-disk artifacts, not the execution process.
Ten categories
Data analysisDocument review & verificationData filtering & processingFault diagnosisKnowledge-base retrievalSchedule managementReport reading QACompliance & policy verificationContent generationImage processingDifficulty tiers
Two task forms, single-turn and multi-turn; mixes are controlled by scenario category, operation type, and difficulty tier; multi-turn tasks persist artifacts turn by turn, and each later turn must build on the previous turn’s conclusions.
Scoring
Rule validation covers artifact contracts (files and required result.json fields), numeric-chain reconciliation, format conventions, and safety compliance; semantic quality is rated by the LLM Judge on graded rubrics (citation grounding, no fabrication, etc.). Dual-channel weights are declared per task and frozen with the task; the Task Score and domain macro-average aggregate at two levels.
Scenario focus: vouchers, expense reimbursement, reconciliation, report analysis, tax. Strongest data rigor and compliance constraints — voucher debit-credit balance, accounting periods, and document states are all validated by business rules, with expected outputs confirmed by business experts.
Task types
Voucher entryFinancial statement preparationBusiness-finance reconciliationReceivable/payable write-offBudget monitoringAssets & depreciationCashier & billsTax complianceCosting & period closeDifficulty tiers
Graded on four levels, L1–L4: L1 single-step queries and entry, L2 multi-step voucher and reporting flows, and L3 cross-module reconciliation and anomaly attribution form the core, with L4 deep reasoning as a supplement; mostly single-turn tasks with a few multi-turn chains. Every business module has at least one task for each of the four high-value exception types (business-rule violation / empty result / abnormal state / input error).
Scoring
Rule validation covers amounts, accounting periods, debit-credit balance, and voucher states; semantic quality (anomaly-analysis evidence, advice reasonableness) is rated by the Judge on graded levels. Fatal items veto: a voucher with unbalanced debit-credit saved as a valid result, or unauthorized modification of business data, zeroes the run.
Scenario focus: orders, inventory, procurement, logistics collaboration. High planning and coordination complexity — order-booking conventions, delivery overdue, and collection risk involve cross-document, cross-period data associations, testing whether end-to-end tasks close the loop.
Task types
Sales ordersCredit managementProcurement collaborationInventory queriesAuto replenishmentExpiry alertsSupply-chain risk controlDemand forecastingTrade complianceDifficulty tiers
Graded on three levels, L1–L3: L1 queries and basic document operations, L2 order creation and cross-table checks, L3 credit risk control and cross-document association judgments; industry enters as a task-tag dimension supporting sliced statistics.
Scoring
Rule checks verify order and goods-receipt/issue document associations, quantity consistency, and inventory changes; numeric verification enforces the result-table contract — reading only designated tables, missing values not counted as zero, one row per business key, consistent across artifacts. Semantic quality (the data basis of delivery and collection-risk analysis, advice specificity) is rated by the Judge; empty results must be properly reported, not errored.
Scenario focus: BOM, production planning, capacity, quality. The longest industrial chains — statistical conventions for process inspection, first/patrol inspection split books, defect top-5 comparison and similar scenarios test cross-link data integration and business reasoning.
Task types
MPS schedulingMulti-plant demand allocationMaterial master data management8D quality analysisOperation-level exception handlingMaterial shortage responseProduction progress reportingDifficulty tiers
Graded on three levels, L1–L3: L1 basic data queries, L2 multi-step scheduling and master-data management, L3 cross-stage exception handling and in-depth quality analysis; the highest share of multi-turn tasks, which persist artifacts turn by turn, with each later turn building on the previous turn’s conclusions.
Scoring
Rule checks verify BOM relations, production-plan constraints, work-order states, and inspection counts (ratios compared to one decimal place, no rounding tolerance); content quality is rated on 0–4 graded criteria.
Task Design
Lingee Bench is organized along the line of “from task completion to business loop closure”: it measures not only isolated operations, but whether an agent can drive a task chain in business systems to a verifiable correct end state. Three keywords summarize the task design:
①Real business flows
The foundational tier uses verifiable professional work packages as its unit; the business-process tier uses end-to-end business task chains — from voucher entry through posting and closing, from orders through goods receipt and issue — checking business artifacts, business states, and necessary controls. Tasks are reconstructed and synthesized from scenarios and public materials, so statements cannot be hit by search; all data is synthesized from public materials.
②Auditable end to end
End-state scoring: scores count only the environment’s true end state and artifact logs, rejecting agent self-reported success, with full trajectories auditable after the fact. Every score is attributable — to the “Agent × Harness × environment × evaluator” combination; rule and Judge scores are kept in separate channels; all expected outputs in the enterprise management domain are confirmed by business experts.
③Fully self-contained Docker, reproducible
All task data, tools, and dependencies are baked into Docker images, external MCP and API calls stripped away; the agent-under-test connects via a CLI harness and runs fully offline in containers — runnable offline, reproducible by third parties, no approval gate. Evaluation configurations are fully public under the seven-layer disclosure checklist (HarnessCard). See “Whitepaper · Evaluation Protocol and Metrics” for protocol details.
Anatomy of a Task
A typical task is a self-contained work package: the task directory defines the task description, Docker execution environment, workspace data assets, and scoring assets, which the evaluation pipeline brings in post-hoc after the run ends. Take the public-group finance task finance-fa-20 “December Business Analysis Report” as an example:
Layered Verification
Rule validation builds on business expected results, the core delivery threshold, and check items; the LLM Judge rates semantic quality on graded levels. The two layers each do their own job, with the dual channel kept separate for diagnosis.
Deterministic layer (rules)
Parser scripts extract facts from artifact logs and the final environment state, verifying item by item:
- Core delivery threshold: files parseable, states verifiable, answers on a carrier
- Fatal items: unbalanced vouchers, unauthorized operations, and the like zero the run
- Ordinary items scored with equal weight: numbers and formulas, business states, cross-document relations
Judge layer (LLM rubric)
Each item defines exactly one quality condition, rated on graded levels (pass/fail, or more):
- Receives only the designated fixed evidence; never accesses the running workspace
- Verified by five experiments: position bias, length bias, self-preference, and more
- Scoring model separated from execution model (first round: GLM-5.3 / GLM-5.3-Flash)
S = w·R + (1−w)·J (w declared per task, frozen with the task freeze, no unified range), two-level aggregation of within-task averaging and domain macro-averaging, fatal items vetoing. See “Whitepaper · Scoring Mechanism”.Reading the Score
A score answers exactly one question: whether the agent completed the business work in a verifiable way. Note three boundaries when reading scores:
AScores reflect delivery completeness
The Task Score ranges over [0,1] and awards partial credit for partial completion; beyond the macro-average, four business-perspective metrics are also reported — risk-weighted score, end-to-end success rate on key flows, fatal-error rate, and function-boundary correctness.
BBound to environment version
Scores are bound to the “Agent × Harness × environment × evaluator” combination, the task version, and the time window, and do not evaluate any other model or Harness; cross-version results are not directly comparable — re-run on the same version.
CNot a production SLA
Sandbox results are not extrapolated to online production scenarios, nor converted into business coverage, job replacement rates, or cost savings; small gaps within the repeated-run noise band are not evidence of real differences.