Lesson 11

Property-Based Testing

The Problem with Traditional Unit Tests

A traditional unit test is example-based: you pick specific inputs, define expected outputs, and verify they match.

// Example-based test
it('should add item to cart', () => {
  const cart = new Cart();
  cart.add(item1);
  expect(cart.items.length).toBe(1);
});

The weakness: you're limited to the scenarios you think of. You test the happy path, maybe a few edge cases, and call it done. But the bugs that reach production are almost always in cases you didn't think to test.

This is even worse when AI writes both the code and the tests - the same blind spots exist in both.

What Property-Based Testing Is

Note: PBT is an independent testing technique (originating in Haskell's QuickCheck, 1999) - not something Kiro invented. You can use fast-check directly in any project without Kiro. It's covered here because Kiro has built PBT into its spec workflow as the primary verification mechanism for generated code. Understanding PBT helps you understand what Kiro is doing during the "verify" step, and gives you a technique you can adopt independently of Kiro.

Instead of testing specific examples, you define properties - universal statements about how your system should behave for any valid input.

What is a "property"?

A property is a rule of the form: "for any valid input meeting certain preconditions, some expected behaviour must hold." Examples:

• "For any two numbers, add(a, b) equals add(b, a)" (commutativity)
• "For any user and any active listing, adding it to favourites means it appears in their list" (invariant)
• "For any order, the total is never negative" (contract)
• "For any valid email string, isValidEmail() returns true" (correctness)
• "For any item added then removed from a cart, the cart returns to its previous state" (round-trip)

These are fundamentally different from "when I pass 'test@example.com', it returns true." Properties describe the general rule, not a specific instance.

How fast-check executes properties

import * as fc from 'fast-check';

// 1. Define an ARBITRARY - a generator for random valid inputs
const arbitraryEmail = fc.emailAddress();
const arbitraryUser = fc.record({
  id: fc.uuid(),
  name: fc.string({ minLength: 1, maxLength: 100 }),
  email: arbitraryEmail,
});

// 2. Define the PROPERTY - what must always be true
fc.assert(
  fc.property(arbitraryUser, (user) => {
    // This runs HUNDREDS of times with different random users
    const created = createUser(user);
    const fetched = getUser(created.id);

    // The property: creating then fetching should return the same data
    expect(fetched.name).toBe(user.name);
    expect(fetched.email).toBe(user.email);
  })
);

// 3. If any generated user causes a failure, fast-check SHRINKS:
//    "Failed with user { name: '🎭', email: 'a+b@c.com' }"
//    → shrinks to minimal failing case
//    "Minimal failure: user { name: '\\', email: 'a@b.c' }"

The framework does three things:

1. Generates - creates hundreds of random valid inputs based on your arbitrary definitions
2. Checks - runs your property assertion against each one
3. Shrinks - when a failure is found, reduces the input to the smallest case that still fails (making the bug obvious)

How Kiro Uses This - The Full Picture

PBT isn't just a testing pattern you could use yourself - it's built into Kiro's spec workflow as the primary mechanism for verifying that generated code matches your intent. Here's why this matters:

The AI-specific problem

When AI generates code, who verifies it's correct? Traditionally:

• The AI writes code → the AI writes tests → same blind spots in both
• You review manually → but for complex logic, you'd miss the same edge cases the AI did

Kiro's solution: separate the concerns. The spec (your requirements) is the source of truth. Code and tests are derived independently from it:

If tests were derived from the code, they'd just validate internal consistency - the code does what the code does. Circular. By deriving tests from the spec instead, you get genuine verification: does the code do what you intended?

The concrete flow in a Kiro spec session

When you run a spec, this happens behind the scenes:

Notice: the property test found a rounding bug with a specific combination (£25.01 + £25.00 = £50.01 - just over the threshold). A hand-written test would likely have used round numbers and missed this.

Where you see this in the IDE

In a spec session:

• When Kiro executes tasks, look for a "Verify correctness" or "Run property tests" step
• If verification fails, Kiro automatically fixes the code and re-runs - you see the feedback loop in the chat
• The generated test file (usually *.property.spec.ts or similar) is committed alongside your code
• These tests become part of your test suite - they run in CI like any other test

What makes this Kiro-specific (not just "use fast-check yourself")

The key insight: PBT in Kiro isn't a tool you use - it's part of the code generation feedback loop. Kiro generates code, then immediately tries to break it with property tests, then fixes what breaks. You get more correct code on the first pass.

When does Kiro NOT use PBT?

• Vibe mode - PBT is tied to specs. In vibe mode, there are no formal requirements to derive properties from.
• Pure UI work - template changes, styling, layout. PBT tests logic, not appearance.
• Simple file operations - renaming, moving, configuration changes.
• When requirements are vague - "make it look better" can't be translated into testable properties. Clear, specific requirements produce better PBT.

How to make PBT work harder for you

1. Use spec mode for logic-heavy features - PBT only triggers in spec mode, not vibe
2. Write precise requirements - "SHALL apply 10% discount" is testable; "should handle discounts" isn't
3. Include boundary conditions in requirements - "WHEN total is exactly £50" forces Kiro to test the boundary
4. Keep the generated property tests - they're valuable regression protection even after the initial generation
5. Ask explicitly - "Verify this implementation against the spec using property-based tests" if Kiro doesn't do it automatically

Concrete Example

Your spec requirement:

WHEN a user adds a listing to favourites,
THE System SHALL display it in their favourites list.

Traditional test (example-based):

it('user A adds listing #5 to favourites', () => {
  addToFavourites(userA, listing5);
  expect(getFavourites(userA)).toContain(listing5);
});

Property-based test (what Kiro generates):

it('any user adding any active listing to favourites should see it in their list', () => {
  fc.assert(
    fc.property(
      arbitraryUser(),
      arbitraryActiveListing(),
      (user, listing) => {
        addToFavourites(user, listing);
        expect(getFavourites(user)).toContain(listing);
      }
    )
  );
});

The property test checks this with: users with special characters in names, listings with extreme prices, users with empty favourites lists, users with 1000 existing favourites, and hundreds more combinations - catching edge cases you'd never think to write manually.

What This Catches That Unit Tests Miss

• Boundary conditions - empty strings, zero values, maximum lengths
• Encoding issues - special characters, unicode, emoji in inputs
• Ordering bugs - operations that should be commutative but aren't
• State-dependent failures - bugs that only appear with specific combinations
• Off-by-one errors - tested with ranges that hit exact boundaries

How to Trigger PBT in Kiro

Property-based testing is part of the spec workflow. When you use spec mode:

1. Create a spec with clear requirements (EARS-format helps)
2. Kiro generates the implementation tasks
3. When tasks execute, Kiro generates property-based tests alongside the code
4. Look for a "Verify" or "Correctness" step - that's where PBT runs

You can also ask explicitly: "Generate property-based tests for this service" or "Verify this code against the spec using PBT."

The Library: fast-check

Kiro uses fast-check for TypeScript/JavaScript PBT. It provides:

• Arbitraries - generators for random data (strings, numbers, arrays, records, custom types)
• Properties - assertions that should hold for all generated inputs
• Shrinking - when a test fails, it automatically reduces the input to the minimal failing case

Where AI stops and fast-check starts

fast-check has no AI in it. It's purely algorithmic - deterministic pseudo-random generation based on type constraints. It predates the AI era (2017). The division of labour:

The AI's job is writing the property tests. fast-check's job is running them exhaustively. Two different kinds of "smart" working together.

fast-check integrates with Jest, Vitest, and other test runners you're already using. It's not a separate framework - it plugs into your existing test setup.

PBT vs Unit Tests: Complementary, Not Replacement

You keep both. Unit tests for clarity and regression coverage. Property tests for finding bugs you didn't think to look for.

Keep Your Specs - They're Not Disposable

Since specs are the source from which both code and tests are derived, they're living documentation - not scaffolding you discard once the feature ships. Reasons to commit them to git alongside the code:

• Re-verification - change the code later, re-run property tests to confirm it still satisfies the original intent
• Onboarding - new team members read the spec to understand what the feature should do, not just how it's implemented
• Spec evolution - when requirements change, update the spec and Kiro re-derives tests for the new behaviour
• Traceability - every test traces back to a specific requirement. When a test fails, you know which requirement is violated.

Kiro stores specs in .kiro/specs/ and they're designed to be committed. They're the "why" that outlives any particular implementation.

When Kiro Does This Automatically

In spec mode, Kiro will often generate PBT without you asking - it's part of the verification step. You'll see it when:

• A spec has clear, testable requirements
• The code involves data transformations, state changes, or business logic
• Kiro detects invariants in your requirements (anything with "shall always", "for any", "regardless of")

For purely UI/template work, PBT is less relevant - it shines with logic, not layout.

Your Tangible Win

You now understand what property-based testing is, how it differs from example-based testing, why it catches bugs that unit tests miss, and how Kiro uses it as part of the spec-to-code verification loop. You know it uses fast-check, integrates with your existing test runner, and triggers automatically in spec mode for logic-heavy requirements.

Quiz