Zaira Standard

1. Evaluation Logic

The design principles behind the Zaira Standard. These explain why the standard is structured the way it is and guide interpretation of the criteria.

The standard scales with tool complexity

The simplest tools face the simplest evaluation. A library installed via npm install and used locally is evaluated against the Base Standard alone: documentation, lifecycle health, supply chain integrity, installation simplicity. No authentication criteria. No API surface area management. No rate limit communication. Those criteria don't apply because the tool doesn't have those concerns.

As a tool's complexity increases (it exposes an API, runs as a hosted service, handles destructive operations, requires credentials) complexity modules activate and add criteria proportional to that complexity. A cloud database with auth, writes, and a REST API is evaluated against the Base Standard plus four complexity modules. Evaluation scope matches the tool's surface area.

Every criterion in a tool's evaluation applies to that tool. There are no N/A markings, no skipped criteria, no denominator adjustments. If a criterion is activated in a tool's evaluation, it is relevant to that evaluation.

Health over activity

The standard measures whether a tool functions, not whether it is being actively developed. A stable, feature-complete library with zero open CVEs, passing continuous integration, and accurate documentation satisfies health criteria even without recent commits; a project with weekly commits and unaddressed issues does not. Criteria that might otherwise penalize inactivity (B4, B11) are defined in terms of empirical health signals (documentation-behavior correspondence, vulnerability patching cadence, installation success on current runtimes) rather than recency signals. Completion is a state demonstrated through continued health, not declared through version increments or announcements.

Discoverability is outside scope

The Zaira Standard evaluates whether a tool is ready for agents to use, not whether agents can find it. Discoverability (registry presence, search engine indexing, structured metadata) is a separate concern. Conflating discoverability with agent-readiness would penalize tools whose distribution is weak and reward tools whose distribution is strong, independent of agent-readiness.

Agent capability floor

The Zaira Standard defines a minimum agent capability threshold: the capability floor. Agents scoring below the floor are not target consumers of Zaira Standard evaluation results.

The capability floor for Zaira Standard v0.9 is 35% on SWE-Bench Pro V9, administered by SEAL. This benchmark evaluates base model and sub-agent performance on software engineering tasks with tool usage. It strips wrapper scaffolding and vendor-specific enhancements.

The floor is defined by benchmark score, not by model name. The capability floor is a normative parameter of each standard version, reviewed with each minor revision (see §5) following the standard change process.

2. How the Standard Works

Module Activation

Every tool starts with the Base Standard: 15 criteria that apply universally. Then, based on what the tool does and how it's accessed, complexity modules activate:

After complexity modules, one or more domain modules may apply based on the tool's functional domains (Payments, Databases, Communications, and others defined in §9).

Each trigger question is a simple yes/no. A tool may trigger zero, one, or many complexity modules. The triggers are independent: a tool can be a Network Service without having a CLI, or have Write Operations without requiring Authentication.

3. How to Read This Document

Criterion Structure

Each criterion includes:

Criterion IDs use module-based prefixes: B (Base Standard), PI (Programmatic Interface), NS (Network Service), WO (Write Operations), AU (Authentication), CLI (CLI), PM (Payments), CM (Communications), DB (Databases), HI (Hosting), AP (Auth Providers), FL (Frameworks). IDs are sequential within each module: a criterion's ID indicates which module it belongs to.

Requirement Levels (RFC 2119)

• MUST: Binary gate. Score ≥1 required for Agent Ready or Agent Native. A tool that scores 0 on any MUST criterion in its activated modules cannot achieve either designation. MUST gates only appear in complexity modules: the Base Standard has no MUST gates.
• SHOULD: Expected for meaningful agent readiness. Scored and weighted. Low scores reduce tier eligibility.
• MAY: Aspirational. Demonstrates excellence. Primarily differentiates the highest tier.

Weight Categories

• Critical (×2): Criteria with the strongest documented impact on agent success rates, including error handling, description quality, and authentication.
• Standard (×1): All other criteria. Important but with less dramatic measured impact or less universal applicability.

Open-Source vs. Commercial Tool Splits

Where a criterion measures fundamentally different things depending on whether the tool is open-source or commercial, two scoring gradients are provided. An open-source library's sustainability risk is contributor concentration; a commercial SaaS tool's sustainability risk is corporate strategy and sunset policy. Both matter, but they require different evidence.

• Open-source tools: Use the open-source rubric (listed first).
• Commercial/closed-source tools: Use the commercial rubric instead.
• Hybrid tools (open-source core with commercial hosted offering): Evaluate against the open-source rubric for the open-source artifact. If the primary product being evaluated is the hosted service, use the commercial rubric.

Scores are directly comparable: a score of 2 on either rubric indicates the same qualitative level (Good).

4. Scoring System

How Criteria Map to Points

Each criterion is scored on a 0-3 scale:

Point Calculation

Criterion Score = Raw Score (0-3) × Weight (1 or 2)

Module Score = Sum of all criterion scores in module
Total Score = Sum of all module scores

Percentage = Total Score / Maximum Possible Score for activated modules

Because modules only activate when relevant, there are no N/A adjustments. The denominator is always the maximum possible score for the specific modules a tool activates.

What is a good score?

A Zaira Score indicates a tool's agent-readiness. Scores fall into three descriptive bands, each of which characterizes a range of agent-usability. These bands describe how a score is interpreted; they are not certification outcomes.

Agent-Ready Base

Describes the score of a tool whose entire surface is covered by the Base Standard: a utility library, a pure-function module, or any tool without an API, network calls, write operations, or credentials. A score in this band indicates that the fundamentals (documentation, lifecycle health, supply chain integrity, installation) meet the standard. No additional evaluation applies because the tool does not expose further surface area.

Agent Ready

Describes the score of a tool with substantial surface area (APIs, hosted services, write operations, authentication) that is substantively usable by agents for standard workflows across that surface. MUST gates pass, every applicable module is covered, and any remaining gaps are documented in the scorecard.

Agent Native

The highest band. Describes the score of a tool that treats agents as first-class consumers across its full surface: no criterion at zero on any MUST or SHOULD, every MUST scoring at least 2, and an overall percentage of 80% or higher. A score in this band satisfies every Agent Ready indicator plus additional thresholds.

Anti-Gaming Mechanisms

1. MUST gates. MUST criteria in activated modules are binary gates: score 0 on any and the tool cannot achieve Agent Ready or Agent Native regardless of total score.
2. Critical weighting. Criteria with the strongest empirical impact on agent success count double, preventing optimization toward lower-weighted criteria.
3. Zero-score limits. Agent Ready allows no more than 5 SHOULD/MUST criteria at 0; Agent Native allows none. This prevents broad neglect across any part of the evaluation.
4. Evidence requirements. Every score requires documented evidence (automated test output, URL, or evaluator rationale). No score without evidence.

Together, these mechanisms ensure that band placement reflects agent-readiness across a tool's full surface area, not selective optimization of lower-weighted criteria.

Scoring Examples

Example 1: Simple library (lodash)

Modules: Base only (15 criteria)

• 2 Critical × 3 × 2 = 12
• 13 Standard × 3 × 1 = 39
• Max: 51

No complexity modules triggered. Scores produced under this profile fall in the Agent-Ready Base band: the Base Standard covers the tool's entire surface, so the fundamentals are the complete evaluation.

Example 2: CLI tool with write operations (Terraform CLI)

Modules: Base (15) + Write Operations (4) + Authentication (4) + CLI (4) = 27 criteria

• Critical criteria: B1, B10, WO1, AU1 = 4 Critical
• 4 Critical × 3 × 2 = 24
• 23 Standard × 3 × 1 = 69
• Max: 93

MUST gates: AU1 Eligible for Agent Ready or Agent Native band.

Example 3: Full SaaS payment platform (Stripe)

Modules: Base (15) + Programmatic Interface (16) + Network Service (8) + Write Operations (4) + Authentication (4) + Payments (4) = 51 criteria

• Critical criteria: B1, B10, PI1, PI2, PI3, PI4, PI10, NS1, NS2, WO1, AU1 = 11 Critical
• 11 Critical × 3 × 2 = 66
• 40 Standard × 3 × 1 = 120
• Max: 186

MUST gates: PI1, PI2, PI15, NS1, AU1 Eligible for Agent Ready or Agent Native band.

Example scores:

• Base: 34/51 (67%)
• Programmatic Interface: 42/63 (67%)
• Network Service: 20/30 (67%)
• Write Operations: 11/15 (73%)
• Authentication: 11/15 (73%)
• Payments: 8/12 (67%)

Total: 126/186 = 68%

Band check:

• ≥60%? Yes → Agent Ready candidate
• All MUSTs ≥1? (PI1 ✓, PI2 ✓, PI15 ✓, NS1 ✓, AU1 ✓) Yes ✓
• ≤5 SHOULD/MUST criteria score 0? Yes (2 zeros) ✓
• Result: Agent Ready

5. Versioning & Governance

The Zaira Standard is public and stable. Revisions are made only when durable shifts in agent-tool interaction warrant them, not in response to short-term trends. Version changes are announced at least 30 days before they take effect, giving implementers and downstream evaluators time to adapt.

Version Scheme

• v0.9: Current version. Scoring thresholds and weights may be refined based on evaluation data before v1.0.
• v1.0: Stable release with finalized thresholds.
• v1.x (Minor): Additive criteria, refined scoring, weight adjustments. Backward compatible.
• v2.0 (Major): Structural changes (module additions/removals, score band or threshold revisions).

6. Scope & Limitations

The Zaira Standard evaluates agent usability. It does not evaluate:

• Tool quality or fitness for purpose. Whether the tool is good at what it does.
• Security guarantees. A tool can meet all Safety criteria and still have undiscovered vulnerabilities. The standard evaluates evidence and practices, not absence of risk.
• Performance benchmarks. Response time, throughput, and uptime are not evaluated (though health endpoints are).
• Pricing fairness. Whether pricing is transparent and machine-readable is evaluated, not whether it's competitive.
• Training data representation. How well current models know a tool is not under the tool's control.
• Compliance certifications. SOC 2, ISO 27001, HIPAA, etc. are noted when present but not replicated.
• Runtime enforcement. The standard publishes evaluation data; enforcement is the responsibility of agent runtimes and policy engines.
• Human developer experience. Criteria are evaluated from the agent's perspective.
• Discoverability. Whether agents can find a tool is outside the standard's scope.

7. Base Standard

15 criteria that apply to every tool, regardless of type.

The Base Standard evaluates the fundamentals: whether an agent can learn to use a tool, whether it is safe to depend on, and whether it will remain available in the future. Every tool (from a 50-line utility library to a cloud platform) is evaluated against these criteria.

The Base Standard has no MUST gates. Simple tools meet the Base Standard or do not, based on overall score. MUST gates appear in complexity modules, where failure has greater operational impact.

Documentation & Usability

Documentation and installation quality.

Documentation and installation are the entry point. An agent encountering a tool for the first time needs machine-readable, structured, self-contained information to understand capabilities and usage, and a friction-free path to get it installed and working. More documentation is not better. Irrelevant docs actively harm agent performance. Quality, structure, and machine-readability matter more than volume.

B1Machine-Readable Documentation Formats

Whether documentation is available in formats agents can directly consume, not just human-rendered HTML requiring JavaScript.

B2Code Example Coverage & Quality

The density, quality, realism, and progressive complexity of code examples.

B3Documentation Structure & Self-Containment

Whether documentation sections are self-contained (extractable independently), structured for agent consumption, and appropriately chunked.

B4Documentation Accuracy & Synchronization

Whether documentation accurately reflects actual tool behavior. Accuracy takes precedence over recency. Documentation that has been unchanged for an extended period but accurately describes current tool behavior scores higher than recently updated documentation that contains errors.

B5Getting Started Completeness

Whether an agent can go from zero to working usage using only the documentation.

B6Changelog & Migration Guidance

Whether changes are communicated in structured, parseable formats.

B7Installation & Configuration Simplicity

How easily an agent can install, set up, and start using a tool.

Safety Fundamentals

Supply chain integrity and vulnerability disclosure.

Two safety criteria apply to every tool regardless of type: supply chain integrity and vulnerability disclosure. Together they establish the minimum baseline: provenance verifiability and a defined path for reporting security issues.

B8Supply Chain Integrity

Whether publisher identity is verified, releases are signed, and the supply chain is tamper-evident.

Open-source tools: evaluate this section:

Trust signal hierarchy: Reproducible builds > SLSA attestation > Code signing > Verified domain > GitHub org verification > npm/PyPI verified publisher > SBOM > security.txt

Commercial/closed-source tools: evaluate this section instead:

B9Vulnerability Disclosure & Security Contact

Whether a clear, machine-readable path exists for reporting security vulnerabilities.

Lifecycle Health

Long-term reliability and continuity.

These criteria evaluate long-term viability: whether a tool will continue to work, be maintained, and remain safe to depend on. A tool that scores well at evaluation but is abandoned soon after offers limited long-term reliability for agent dependency.

Open-source and commercial tools have fundamentally different risk profiles. An open-source tool's risk is contributor abandonment; a commercial tool's risk is corporate sunset or acquisition. Where this distinction matters, criteria provide separate rubrics. For open-source tools, most criteria are fully automatable from public data (GitHub API, package registries, OpenSSF Scorecard).

B10Project Sustainability

The likelihood that this tool will continue to be maintained and supported over time.

Open-source tools: evaluate this section:

Commercial/closed-source tools: evaluate this section instead:

B11Maintenance Health

Whether the tool demonstrates operational health. Active health is not synonymous with recent commit activity: it is evidence that the tool continues to function and that unresolved issues receive maintainer response. A project with no open issues and no recent commits satisfies this criterion; a project with numerous unanswered issues and no recent commits does not. The distinction is measurable.

B12Semver Adherence & Version Stability

Whether breaking changes are confined to major versions and the tool follows predictable versioning.

B13Governance & Continuity

Whether the tool has governance structures that reduce single-entity risk and provide continuity assurance.

Open-source tools: evaluate this section:

Commercial/closed-source tools: evaluate this section instead:

B14Security Track Record

Vulnerability response speed and proactive security practices.

Open-source tools: evaluate this section:

Commercial/closed-source tools: evaluate this section instead:

B15Terms & Licensing Stability

Whether the terms under which the tool is available are stable and free from change risk signals.

Open-source tools: evaluate this section:

Commercial/closed-source tools: evaluate this section instead:

8. Complexity Modules

Complexity modules add criteria based on how the tool is accessed and what it does. Each module is activated by a yes/no trigger question. A tool may activate zero, one, or many complexity modules: the triggers are independent.

Activating at least one complexity module places a tool's score in the Agent Ready or Agent Native band (see §4). Tools evaluated on the Base Standard alone land in the Agent-Ready Base band.

9. Domain Modules

Domain modules add criteria based on a tool's functional domains. A tool may trigger one or more domain modules: for example, Supabase (Databases + Auth Providers + Hosting) or Firebase (Databases + Auth Providers + Communications). Each activated domain module adds its criteria to the tool's evaluation, expanding both the numerator and denominator like complexity modules.