86: Add AssetAZ Journal Service with durable PDE event journal and replayable consumers
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-09-09
|
||||
@@ -0,0 +1,32 @@
|
||||
## Context
|
||||
|
||||
See proposal.md. The authoritative baseline is Gitea 0.1-dev at f1dd5ed. Nenjim separates public component APIs from impl.ref processes; the Hub owns explicit construction and registry registration. The artifact JournalService is unrelated.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
Implement the two capability contracts. Keep primitive PDE parsing below event codecs and record integrity. Do not extend the v1 scope in the proposal.
|
||||
|
||||
## Decisions
|
||||
|
||||
- Place events in com.r35157.assetaz.events, service API in services.journal, codecs under services.journal.codec and implementations under impl.ref. PDE belongs to com.r35157.libs.pde. Codec components expose read identity, domain type and whether they encode current writes; historical read-only codecs can coexist without implicit highest-version selection.
|
||||
- The Hub registers explicit codec instances and offers assetaz.journal.service through its process constructor catalogue. The journal resolves the registry catalogue at startup, validates duplicate identities/current encoders, and uses a single lifetime file channel and exclusive file lock. No service restart API is added.
|
||||
- One journal monitor serializes durable append, registration and queue publication. Each subscription has its own monitor, FIFO and worker. Every publication carries a sequence, including irrelevant records, so workers advance only after examining the ordered stream. Consumer code never runs under the journal monitor. An empty queue observed under the subscription monitor selects the ONLINE cutover; subsequent publications queue behind it while the same worker calls ONLINE.
|
||||
- Historical readers scan from the beginning to validate chain state but invoke callbacks only from replayFromSequence through captured registrationHead. This avoids adding indexes or snapshots. The reader closes before pending live records drain.
|
||||
- PDE uses the published little-endian integer/length type table; UUID conversion alone is big endian at codec level. Record hashes operate on original bounded ranges, not canonicalized representations.
|
||||
- JournalFormatVersionChanged has common event fields followed by a target version. Only format 1 is implemented; unsupported targets fail before append and during loading. Future parsers can switch after the ordinary event without a new physical record type.
|
||||
- Barriers are interruptible (InterruptedException alongside DegradedException); no timeout or recovery API is introduced.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- Slow consumers can accumulate FIFO entries → intentional independent v1 workers; no unrequested tuning or overflow policy.
|
||||
- Callbacks cannot be rolled back by the journal → document consumer atomic update responsibility and preserve safe cursor on failure.
|
||||
- Uncertain force/write can leave a complete or partial record → permanently fail closed and validate the authoritative file next JVM; no automatic repair.
|
||||
- Hash chains do not prevent a writer recomputing history → external anchors remain out of scope.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
New optional process, using data/assetaz/journal/events.pde. No migration of the Nenjim artifact journal or existing service histories. Add the process ID to the example configuration; existing user configurations remain opt-in. No commits or publication in this task.
|
||||
|
||||
## ValueTags
|
||||
|
||||
The owner name uses the existing Name tag (String). No shared Detag definitions are changed. Proposed future tags, pending separate approval, are AssetAZJournalSequence (parent long, Java long) for authoritative cursors, AssetAZEventType (parent String, Java String) for persistent short identifiers, AssetAZEventVersion and AssetAZJournalFormatVersion (parent int, Java int) to distinguish schema and envelope versions, and AssetOwnerId and AssetAZCorrelationId (parent UUID, Java UUID) to distinguish owner identity from workflow grouping. This implementation keeps the issue's primitive/UUID signatures where these tags do not yet exist.
|
||||
@@ -0,0 +1,28 @@
|
||||
## Why
|
||||
|
||||
AssetAZ needs an authoritative durable event history from which services can reconstruct their models and establish causal freshness. Gitea issue #86 defines a deliberately strict first version.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add an append-only PDE journal with SHA-256 integrity, durable sequence commitment and fail-closed writes.
|
||||
- Add domain events and registry-provided positional event codecs, independent of Java class naming.
|
||||
- Add independent virtual-thread subscriptions, replay, exact filtering, progress barriers and isolated degradation.
|
||||
- Add reusable bounded PDE primitive encoding and decoding.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `assetaz-journal`: Persistence, event schemas, replay and consumer consistency contracts.
|
||||
- `pde-encoding`: Minimal bounded PDE primitives and object framing.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
None.
|
||||
|
||||
## Impact
|
||||
|
||||
New AssetAZ API and reference implementation packages, a reusable PDE library, and explicit codec/process entries in the existing Hub composition root. No new dependencies. The Nenjim artifact journal is separate and unchanged.
|
||||
|
||||
## Non-goals
|
||||
|
||||
No snapshots, repair, truncation, retries, consumer recovery, unregister, restart support, adaptive limits, pools, generalized serialization, reflection discovery, migration, PDL, signatures, causation IDs or unrelated thread changes.
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
## Purpose
|
||||
|
||||
Define the authoritative AssetAZ fact journal and independent consumer reconstruction and causal freshness contracts.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Durable authoritative ordering
|
||||
The journal SHALL be append-only and serialize appends, assigning contiguous sequences from 1 independently of occurredAt. Fields 2..6 SHALL be encoded once, hashed, and copied into the final record. Only successful FileChannel.force(true) SHALL commit the candidate head, publish and return its sequence. Missing append codecs and oversized records MUST fail before any append. An uncertain write SHALL permanently fail closed in that JVM without retries, repair or truncation.
|
||||
|
||||
#### Scenario: Failed force
|
||||
- **WHEN** persistence may have left an uncertain tail
|
||||
- **THEN** the candidate is not committed or published and all later appends fail.
|
||||
|
||||
### Requirement: Record integrity and bounded readers
|
||||
Each record SHALL be exactly one top-level PDE OBJECT containing checksum, previousChecksum, sequence, stable short eventType, eventVersion and OBJECT payload in that order. Both hashes SHALL be 32 raw bytes. SHA-256 SHALL cover the original persisted fields 2..6; the first previousChecksum SHALL be SHA-256(empty). Each independent reader SHALL reuse one fixed 1024-byte buffer. The entire record including header MUST be at most 1024 bytes; reading SHALL reject declared excess or truncation immediately after framing and before payload decoding. Startup SHALL validate the entire history, checksums, chain, contiguous sequences, PDE, known codecs and exact schemas before succeeding.
|
||||
|
||||
#### Scenario: Corrupt history
|
||||
- **WHEN** a checksum, chain, sequence, schema, type/version or framing cannot be verified
|
||||
- **THEN** startup fails with the rejection layer identified; no record is skipped.
|
||||
|
||||
### Requirement: Event and codec contract
|
||||
Events SHALL represent facts, not commands. occurredAt SHALL be semantic time; note and correlationId are nullable. Correlation groups independently meaningful workflow events without atomicity or expected count; inseparable changes belong in one event. Codecs SHALL be registry components resolved by stable eventType and persistent version, never constructed in the journal. Domain class names MUST NOT encode persistent versions. Journal integrity, PDE validity and event schema validity SHALL remain separate layers.
|
||||
|
||||
#### Scenario: Historical codec
|
||||
- **WHEN** a known historical persistent version is read
|
||||
- **THEN** its codec yields the current domain event without changing the domain type name.
|
||||
|
||||
### Requirement: Owner payload
|
||||
AssetOwnerRegistered version 1 SHALL contain exactly seven positional values: signed long epoch seconds, nano-of-second, nullable UTF-8 note, nullable UUID correlationId, required UUID assetOwnerId, required textual assetOwnerType, required UTF-8 name. UUIDs SHALL be exactly 16 bytes, most-significant then least-significant long, both big endian. Nanos MUST be 0..999999999; owner type MUST be NATURAL_PERSON or LEGAL_PERSON, never ordinal. Required values, Instant range and exact field count SHALL be validated by the codec. No additional owner metadata SHALL be added.
|
||||
|
||||
#### Scenario: Extra payload value
|
||||
- **WHEN** an otherwise valid owner payload has an eighth value
|
||||
- **THEN** schema validation rejects it.
|
||||
|
||||
### Requirement: Format changes
|
||||
New journals SHALL implicitly start in format 1 with no header. JournalFormatVersionChanged SHALL be an ordinary event in the same envelope and hash chain, with a positive target version and common event fields. A change applies to subsequent records; returning to an earlier supported version SHALL be representable. Unsupported formats MUST fail closed; v1 implements only format 1.
|
||||
|
||||
#### Scenario: Chained format event
|
||||
- **WHEN** a format-1 change to format 1 is appended
|
||||
- **THEN** it is checksummed and replayed normally; unsupported target formats are rejected before append.
|
||||
|
||||
### Requirement: Registration and independent replay
|
||||
Registration SHALL atomically capture registrationHead, validate 1 <= replayFromSequence <= registrationHead + 1, defensively copy a non-null filter without null members and reject duplicate consumer identity (==). Empty filters SHALL mean all; other filters SHALL match exact runtime class only. Registration SHALL return without awaiting replay. Each subscription SHALL own a virtual-thread worker and independent reader, replay through registrationHead, close the reader and drain subsequent FIFO publications. Degraded registrations remain registered. registrationHead MUST NOT be public API.
|
||||
|
||||
#### Scenario: Late subscriber
|
||||
- **WHEN** another consumer registers while an existing one processes events
|
||||
- **THEN** only the new consumer replays; both receive their own ordered stream without gaps or duplicates.
|
||||
|
||||
### Requirement: Progress and barriers
|
||||
processedThrough SHALL mean all records through the cursor were successfully handled or examined as irrelevant, starting at replayFromSequence - 1. A barrier SHALL return whenever the safe cursor meets its minimum, including DEGRADED; otherwise wait while CATCHING_UP/ONLINE or throw DegradedException when DEGRADED. Progress and degradation SHALL wake waiters. The barrier MUST NOT imply a historical snapshot.
|
||||
|
||||
#### Scenario: Filtered progress
|
||||
- **WHEN** a consumer last handled 1000 and examined irrelevant records through 1100
|
||||
- **THEN** its cursor is 1100 and a barrier for 1100 succeeds.
|
||||
|
||||
### Requirement: States and atomic cutover
|
||||
Subscriptions SHALL begin CATCHING_UP, without requiring an initial state callback. After replay and pending events through an atomic cutover, the worker SHALL request ONLINE. Successful return establishes readiness to act; the callback should establish readiness without irreversible actions. Publication concurrent with cutover SHALL be processed before ONLINE or as the first normal event after ONLINE, with no gap. All callbacks for one consumer SHALL be sequential.
|
||||
|
||||
#### Scenario: Append during readiness callback
|
||||
- **WHEN** a new event arrives at the chosen cutover
|
||||
- **THEN** it is queued and processed next after readiness succeeds.
|
||||
|
||||
### Requirement: Isolated degradation and atomic models
|
||||
Callback or ONLINE transition failure SHALL retain the last safe cursor, mark only that subscription DEGRADED, stop delivery, wake barriers, and log the exception plus consumer identity, sequence and state. Consumers SHALL atomically apply each event or leave their externally visible model unchanged. Replay SHALL rebuild state without repeating trades, transfers or notifications. No OFFLINE, unregister or recovery SHALL be implemented.
|
||||
|
||||
#### Scenario: Consumer throws
|
||||
- **WHEN** one subscriber fails applying sequence N
|
||||
- **THEN** it retains its safe cursor, future barriers beyond that cursor fail, and other subscribers continue.
|
||||
@@ -0,0 +1,26 @@
|
||||
## Purpose
|
||||
|
||||
Provide reusable primitive binary encoding and bounded structural decoding for positional AssetAZ journal data.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Primitive representation
|
||||
The PDE layer SHALL implement Jenkov PDE integers, UTF-8, bytes, typed null and length-delimited OBJECT framing only. Integer magnitudes and lengths use little endian; negative integers encode -(value + 1). It MUST reject wrong types, malformed lengths, invalid UTF-8 and values outside the requested Java int/long range.
|
||||
|
||||
#### Scenario: Integer extremes
|
||||
- **WHEN** signed long extrema and nullable strings/bytes are encoded
|
||||
- **THEN** they round-trip without loss and null is distinguished from empty.
|
||||
|
||||
### Requirement: Bounded decoding
|
||||
Every decoder SHALL operate on byte[], offset and length; child objects SHALL have independent logical bounds. requireEnd SHALL reject unconsumed values using the logical limit, never backing-array size. Framing SHALL expose total object size before payload decoding.
|
||||
|
||||
#### Scenario: Reused backing storage
|
||||
- **WHEN** a short record follows a long record in one buffer
|
||||
- **THEN** stale bytes are invisible and truncated children are rejected.
|
||||
|
||||
### Requirement: Abstraction boundary
|
||||
PDE SHALL know nothing about UUID, Instant, events, sequence numbers or checksums. It MUST NOT implement object mapping, reflection, annotation schemas or PDL.
|
||||
|
||||
#### Scenario: Domain payload decoding
|
||||
- **WHEN** a UUID byte value is decoded
|
||||
- **THEN** PDE validates its binary framing while the codec validates its 16-byte schema.
|
||||
@@ -0,0 +1,23 @@
|
||||
## 1. Encoding and contracts
|
||||
|
||||
- [x] 1.1 Implement bounded PDE primitives and framing.
|
||||
- [x] 1.2 Add event/service contracts and registry codecs with exact schemas.
|
||||
|
||||
## 2. Journal and consumers
|
||||
|
||||
- [x] 2.1 Implement record hashing, bounded readers, startup validation and durable fail-closed append.
|
||||
- [x] 2.2 Implement independent virtual-thread replay, FIFO delivery, cutover and barriers.
|
||||
- [x] 2.3 Wire passive codecs and journal process in the Hub.
|
||||
|
||||
## 3. Verification
|
||||
|
||||
- [x] 3.1 Run build and existing tests; verify issue invariants and diagnostics.
|
||||
- [x] 3.2 Validate OpenSpec and review unstaged changes against scope.
|
||||
|
||||
## Validation results
|
||||
|
||||
- Normal Gradle build passed, including all four existing tests (zero failures/errors/skips).
|
||||
- 78 temporary checks outside the repository passed: PDE ranges and bounds, exact schemas, raw hashes, full startup validation, 1024/1025-byte boundary, replay/live ordering, filtering, safe cursors, barriers, isolated degradation and uncertain force failure before publication.
|
||||
- OpenSpec strict validation and git diff whitespace checks passed. No repository tests were added, following project policy.
|
||||
- The fresh checkout omitted ignored conf/detag.conf; build validation used a local symlink to the existing configuration without changing its definitions, and a workspace-local Gradle cache.
|
||||
- All changes remain unstaged and uncommitted on the fetched Gitea baseline.
|
||||
@@ -0,0 +1,69 @@
|
||||
# assetaz-journal Specification
|
||||
|
||||
## Purpose
|
||||
Define the authoritative AssetAZ fact journal and independent consumer reconstruction and causal freshness contracts.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Durable authoritative ordering
|
||||
The journal SHALL be append-only and serialize appends, assigning contiguous sequences from 1 independently of occurredAt. Fields 2..6 SHALL be encoded once, hashed, and copied into the final record. Only successful FileChannel.force(true) SHALL commit the candidate head, publish and return its sequence. Missing append codecs and oversized records MUST fail before any append. An uncertain write SHALL permanently fail closed in that JVM without retries, repair or truncation.
|
||||
|
||||
#### Scenario: Failed force
|
||||
- **WHEN** persistence may have left an uncertain tail
|
||||
- **THEN** the candidate is not committed or published and all later appends fail.
|
||||
|
||||
### Requirement: Record integrity and bounded readers
|
||||
Each record SHALL be exactly one top-level PDE OBJECT containing checksum, previousChecksum, sequence, stable short eventType, eventVersion and OBJECT payload in that order. Both hashes SHALL be 32 raw bytes. SHA-256 SHALL cover the original persisted fields 2..6; the first previousChecksum SHALL be SHA-256(empty). Each independent reader SHALL reuse one fixed 1024-byte buffer. The entire record including header MUST be at most 1024 bytes; reading SHALL reject declared excess or truncation immediately after framing and before payload decoding. Startup SHALL validate the entire history, checksums, chain, contiguous sequences, PDE, known codecs and exact schemas before succeeding.
|
||||
|
||||
#### Scenario: Corrupt history
|
||||
- **WHEN** a checksum, chain, sequence, schema, type/version or framing cannot be verified
|
||||
- **THEN** startup fails with the rejection layer identified; no record is skipped.
|
||||
|
||||
### Requirement: Event and codec contract
|
||||
Events SHALL represent facts, not commands. occurredAt SHALL be semantic time; note and correlationId are nullable. Correlation groups independently meaningful workflow events without atomicity or expected count; inseparable changes belong in one event. Codecs SHALL be registry components resolved by stable eventType and persistent version, never constructed in the journal. Domain class names MUST NOT encode persistent versions. Journal integrity, PDE validity and event schema validity SHALL remain separate layers.
|
||||
|
||||
#### Scenario: Historical codec
|
||||
- **WHEN** a known historical persistent version is read
|
||||
- **THEN** its codec yields the current domain event without changing the domain type name.
|
||||
|
||||
### Requirement: Owner payload
|
||||
AssetOwnerRegistered version 1 SHALL contain exactly seven positional values: signed long epoch seconds, nano-of-second, nullable UTF-8 note, nullable UUID correlationId, required UUID assetOwnerId, required textual assetOwnerType, required UTF-8 name. UUIDs SHALL be exactly 16 bytes, most-significant then least-significant long, both big endian. Nanos MUST be 0..999999999; owner type MUST be NATURAL_PERSON or LEGAL_PERSON, never ordinal. Required values, Instant range and exact field count SHALL be validated by the codec. No additional owner metadata SHALL be added.
|
||||
|
||||
#### Scenario: Extra payload value
|
||||
- **WHEN** an otherwise valid owner payload has an eighth value
|
||||
- **THEN** schema validation rejects it.
|
||||
|
||||
### Requirement: Format changes
|
||||
New journals SHALL implicitly start in format 1 with no header. JournalFormatVersionChanged SHALL be an ordinary event in the same envelope and hash chain, with a positive target version and common event fields. A change applies to subsequent records; returning to an earlier supported version SHALL be representable. Unsupported formats MUST fail closed; v1 implements only format 1.
|
||||
|
||||
#### Scenario: Chained format event
|
||||
- **WHEN** a format-1 change to format 1 is appended
|
||||
- **THEN** it is checksummed and replayed normally; unsupported target formats are rejected before append.
|
||||
|
||||
### Requirement: Registration and independent replay
|
||||
Registration SHALL atomically capture registrationHead, validate 1 <= replayFromSequence <= registrationHead + 1, defensively copy a non-null filter without null members and reject duplicate consumer identity (==). Empty filters SHALL mean all; other filters SHALL match exact runtime class only. Registration SHALL return without awaiting replay. Each subscription SHALL own a virtual-thread worker and independent reader, replay through registrationHead, close the reader and drain subsequent FIFO publications. Degraded registrations remain registered. registrationHead MUST NOT be public API.
|
||||
|
||||
#### Scenario: Late subscriber
|
||||
- **WHEN** another consumer registers while an existing one processes events
|
||||
- **THEN** only the new consumer replays; both receive their own ordered stream without gaps or duplicates.
|
||||
|
||||
### Requirement: Progress and barriers
|
||||
processedThrough SHALL mean all records through the cursor were successfully handled or examined as irrelevant, starting at replayFromSequence - 1. A barrier SHALL return whenever the safe cursor meets its minimum, including DEGRADED; otherwise wait while CATCHING_UP/ONLINE or throw DegradedException when DEGRADED. Progress and degradation SHALL wake waiters. The barrier MUST NOT imply a historical snapshot.
|
||||
|
||||
#### Scenario: Filtered progress
|
||||
- **WHEN** a consumer last handled 1000 and examined irrelevant records through 1100
|
||||
- **THEN** its cursor is 1100 and a barrier for 1100 succeeds.
|
||||
|
||||
### Requirement: States and atomic cutover
|
||||
Subscriptions SHALL begin CATCHING_UP, without requiring an initial state callback. After replay and pending events through an atomic cutover, the worker SHALL request ONLINE. Successful return establishes readiness to act; the callback should establish readiness without irreversible actions. Publication concurrent with cutover SHALL be processed before ONLINE or as the first normal event after ONLINE, with no gap. All callbacks for one consumer SHALL be sequential.
|
||||
|
||||
#### Scenario: Append during readiness callback
|
||||
- **WHEN** a new event arrives at the chosen cutover
|
||||
- **THEN** it is queued and processed next after readiness succeeds.
|
||||
|
||||
### Requirement: Isolated degradation and atomic models
|
||||
Callback or ONLINE transition failure SHALL retain the last safe cursor, mark only that subscription DEGRADED, stop delivery, wake barriers, and log the exception plus consumer identity, sequence and state. Consumers SHALL atomically apply each event or leave their externally visible model unchanged. Replay SHALL rebuild state without repeating trades, transfers or notifications. No OFFLINE, unregister or recovery SHALL be implemented.
|
||||
|
||||
#### Scenario: Consumer throws
|
||||
- **WHEN** one subscriber fails applying sequence N
|
||||
- **THEN** it retains its safe cursor, future barriers beyond that cursor fail, and other subscribers continue.
|
||||
@@ -0,0 +1,27 @@
|
||||
# pde-encoding Specification
|
||||
|
||||
## Purpose
|
||||
Provide reusable primitive binary encoding and bounded structural decoding for positional AssetAZ journal data.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Primitive representation
|
||||
The PDE layer SHALL implement Jenkov PDE integers, UTF-8, bytes, typed null and length-delimited OBJECT framing only. Integer magnitudes and lengths use little endian; negative integers encode -(value + 1). It MUST reject wrong types, malformed lengths, invalid UTF-8 and values outside the requested Java int/long range.
|
||||
|
||||
#### Scenario: Integer extremes
|
||||
- **WHEN** signed long extrema and nullable strings/bytes are encoded
|
||||
- **THEN** they round-trip without loss and null is distinguished from empty.
|
||||
|
||||
### Requirement: Bounded decoding
|
||||
Every decoder SHALL operate on byte[], offset and length; child objects SHALL have independent logical bounds. requireEnd SHALL reject unconsumed values using the logical limit, never backing-array size. Framing SHALL expose total object size before payload decoding.
|
||||
|
||||
#### Scenario: Reused backing storage
|
||||
- **WHEN** a short record follows a long record in one buffer
|
||||
- **THEN** stale bytes are invisible and truncated children are rejected.
|
||||
|
||||
### Requirement: Abstraction boundary
|
||||
PDE SHALL know nothing about UUID, Instant, events, sequence numbers or checksums. It MUST NOT implement object mapping, reflection, annotation schemas or PDL.
|
||||
|
||||
#### Scenario: Domain payload decoding
|
||||
- **WHEN** a UUID byte value is decoded
|
||||
- **THEN** PDE validates its binary framing while the codec validates its 16-byte schema.
|
||||
Reference in New Issue
Block a user