diff --git a/AGENTS-TASKS.md b/AGENTS-TASKS.md index 8bb3567..76863a8 100644 --- a/AGENTS-TASKS.md +++ b/AGENTS-TASKS.md @@ -78,11 +78,12 @@ format-level decisions before implementation. ## Phase 4 — Git-like reversible workflow -- [ ] Write and approve a design for working state, pseudo-staging, committed - epochs, and recovery semantics before implementing it. -- [ ] Define `diff`, `stage`, `unstage`/`restore --staged`, `commit`, `restore`, +- [ ] Approve the written design for working state, pseudo-staging, committed + epochs, and recovery semantics before implementing it. See + [the Phase 4 proposal](docs/staging-design.md). +- [x] Define `diff`, `stage`, `unstage`/`restore --staged`, `commit`, `restore`, `revert`, and `log` behavior, including entity-level selection. -- [ ] Define crash consistency, locking, concurrent-process behavior, and +- [x] Define crash consistency, locking, concurrent-process behavior, and repository format versioning. - [ ] Add migration fixtures for all prior supported repository versions. - [ ] Implement only after the design preserves current data and allows every diff --git a/docs/README.md b/docs/README.md index 97fc290..3a9262d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -7,6 +7,8 @@ behavior clearly. - [Repository format](repository.md): files, epochs, compatibility, backup, and recovery. - [Reversing changes](recovery.md): checkpoints and recovery for every mutation. - [Architecture](architecture.md): crate responsibilities, data flow, and quality gates. +- [Staging design](staging-design.md): proposed Phase 4 state, command, transaction, + and migration contract. - [Roadmap](roadmap.md): current work and stretch proposals. Start with the project [README](../README.md). Contributors should also read diff --git a/docs/architecture.md b/docs/architecture.md index 0d8322b..3f27156 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -19,6 +19,12 @@ writes with temporary-file plus rename; add locking; remove unsafe cache accessors; separate styling from semantic output; and retain unknown custom metadata as the model becomes organisation-agnostic. +Phase 4's proposed persistence boundary is specified in the +[staging, history, and recovery design](staging-design.md). It separates HEAD, +the index, and the editable worktree, and defines the atomic transaction and +migration rules that implementation must preserve. The proposal requires user +approval before repository-format code changes begin. + Library tests currently cover parsing, statistics, display snapshots, and parts of entity/epoch behavior. CLI integration coverage is the main gap. Routine CI should run formatting, all-target workspace tests, strict Clippy, and a release diff --git a/docs/roadmap.md b/docs/roadmap.md index dc8f704..2015cec 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -24,11 +24,11 @@ and every mutation has documented and tested recovery guidance. The next milestone is the reviewed repository-format design for working state, pseudo-staging, restore, and revert. Implementation begins only after those -compatibility and crash-consistency decisions are approved. - -Next comes a reviewed Git-like model for working changes, pseudo-staging, -committed epochs, restore, and revert. Organisation-agnostic custom fields and -types follow, with migrations and unknown-field retention designed first. +compatibility and crash-consistency decisions in the +[Phase 4 proposal](staging-design.md) are approved. Implementation then proceeds +through migration fixtures, transactions and locking, staging, and reversible +history. Organisation-agnostic custom fields and types follow, with migrations +and unknown-field retention designed first. ## Proposed stretch goals diff --git a/docs/staging-design.md b/docs/staging-design.md new file mode 100644 index 0000000..0b97a02 --- /dev/null +++ b/docs/staging-design.md @@ -0,0 +1,162 @@ +# Staging, history, and recovery design + +Status: **proposed for approval**. This document is the Phase 4 contract; it +does not describe behavior available in the current release. Persisted-format +implementation must not begin until this contract is approved. + +The goal is a familiar Git-like workflow without pretending that Epoch Forge +is Git. A repository has one editable timeline, every meaningful change can be +inspected before it becomes history, and ordinary mistakes are reversible from +the CLI. + +## State model + +Version 2 repositories have three intentionally separate states: + +1. **HEAD** is the immutable committed snapshot at + `epoch//epochinfo.json`. +2. **Index** is the complete proposed next snapshot in `.efo/index.json`. +3. **Worktree** is the complete editable snapshot in `.efo/worktree.json`. + +Complete snapshots keep recovery understandable and avoid reconstructing state +from a fragile chain of patches. Entity-level staging copies selected entities +between snapshots; it does not change the on-disk representation into partial +patches. Every snapshot carries a schema version and repository identity. + +The root `.epoch` manifest remains the small source of repository identity and +HEAD selection. Version 2 adds `format_version: 2` and a monotonic +`next_epoch_id`; existing identity, current epoch, version, and tag information +is retained. Epoch IDs are never reused, including after old future history is +pruned. + +Transient state lives under `.efo/`: + +```text +.epoch repository manifest and commit point +.efo/index.json staged complete snapshot +.efo/worktree.json editable complete snapshot +.efo/lock exclusive writer ownership +.efo/transaction.json recoverable in-progress operation journal +.efo/migrations/ byte-for-byte pre-migration backups +epoch//epochinfo.json immutable committed snapshots +``` + +## Command contract + +Commands use both plain text and decorative symbols: symbols and colour may +reinforce meaning, but never carry meaning alone. + +- `efo status` reports HEAD plus staged and unstaged changes separately. It + names entity kind and ID and gives the next useful command. +- `efo diff` compares index to worktree. `efo diff --staged` compares HEAD to + index. Output is deterministic and field-oriented; `--no-color` remains + readable and scriptable. +- `efo stage --all` copies the complete worktree to the index. Entity selectors + (`--kind ...`) copy only those entities, including deletion. + `efo stage --world` selects world-level metadata. Invalid or ambiguous + selectors change nothing. +- `efo restore --staged ...` copies selected state from HEAD to the index. + `efo restore --worktree ...` copies from the index to the worktree. One layer + flag is mandatory, and the same entity selectors and `--all` are supported. + This explicitness prevents accidental data loss. `unstage` may exist only as + a documented alias for `restore --staged`. +- `efo commit -m ` validates the index, creates one immutable epoch, + advances HEAD, then makes index and worktree equal to the new HEAD. An empty + commit is rejected with guidance. Commit messages are required and stored as + snapshot metadata. +- `efo epoch next` remains the simulation-oriented convenience operation. It + requires no unstaged changes, advances time in the index, validates it, and + commits it as one transaction. Already staged changes are included and shown + in the confirmation summary. +- `efo log` displays newest-first immutable history with epoch ID, in-world + date, message, and tags. Stable plain output is available independently of + Nerd Fonts and colour. +- `efo switch ` changes HEAD only when index and worktree are clean, then + resets both transient snapshots to the selected commit. `--force` is not a + shortcut; users must explicitly restore or commit first. +- `efo revert ` computes the inverse of that epoch relative to its + predecessor and applies it to the current index. It then creates a new commit + after confirmation. It never deletes history. Conflicts abort without + changing any layer and list affected fields. +- The destructive historical `reset` operation is renamed `prune-future`. + `reset` remains a deprecated alias for one release. Pruning always shows a + dry-run, requires confirmation unless `--yes` is supplied, never rewinds the + ID allocator, and leaves a migration-style backup that can be restored. + +Mutating commands refuse to proceed with an invalid layer. Errors state what +happened, what was left unchanged, and the safest recovery command. + +## Atomicity and crash recovery + +All mutations use one repository transaction: + +1. Create the writer lock exclusively. +2. Resolve the repository once, read all inputs, and validate preconditions. +3. Write replacement files into a transaction directory on the same + filesystem, flush them, and record old/new hashes plus ordered renames in + `.efo/transaction.json`. +4. Rename immutable epoch data first, transient snapshots second, and `.epoch` + last. The manifest rename is the commit point. +5. Flush affected directories, verify hashes and invariants, remove the journal, + then release the lock. + +Before the commit point, recovery rolls back transient replacements. After it, +recovery finishes the recorded operation. `efo recover` performs that decision +deterministically and explains it before changing files; normal commands refuse +to mutate a repository with an unfinished journal. Re-running recovery is safe. + +Readers may operate while no journal exists. Only one writer is allowed. The +lock records repository UUID, PID plus process-start token, hostname, command, +and creation time. A lock is never considered stale from age alone. `efo lock +inspect` explains ownership; `efo lock break` requires proof that the local +process is gone or an explicit confirmation for a remote/unverifiable owner. + +## Format migration + +Opening format 1 remains read-only compatible. The first write instructs the +user to run `efo migrate --to 2`; migration is never implicit. Migration is a +dry-run by default and requires confirmation (or `--yes`) to execute. + +The migration procedure is: + +1. Run format-1 integrity checks and reject ambiguity. +2. Copy every affected byte into `.efo/migrations/v1-/` and write a + hash manifest. +3. Create worktree and index from the selected epoch and set `next_epoch_id` to + one greater than the maximum ID ever observed, including future epochs. +4. Commit the version-2 manifest atomically, then run version-2 integrity + checks. + +Rollback from the backup is supported until the first version-2 commit. After +that, migration is one-way; export remains available. Fixtures must cover empty +and populated repositories, tags, a switched-back repository with future +epochs, relationships, legacy `Monster` naming, and malformed repositories +whose migration must be rejected without modification. + +## Compatibility invariants + +- Existing epoch contents, IDs, tags, dates, entity IDs, and relationships do + not change during migration. +- Unknown fields survive load/save and migration once the custom-field model is + introduced; Phase 4 must not reserve a representation that prevents this. +- Repository discovery remains independent of the current working directory. +- Every selector is stable by entity ID; display names are never identity. +- Machine-readable output will be versioned separately from human presentation. +- Branching, merging, and remote synchronization are deferred. The format may + gain parent references now, but the CLI remains a single linear timeline. + +## Approval checklist + +Approval of this design confirms these product choices: + +- direct entity edits modify the worktree; +- the index is a complete snapshot with entity-level selection; +- commits are immutable, linear, and use never-reused monotonic IDs; +- restore always names the layer it may overwrite; +- revert creates new history and aborts on conflicts; +- destructive pruning requires a preview, confirmation, and backup; +- format migration is explicit, validated, backed up, and initially reversible. + +After approval, implementation proceeds in small commits: format types and +fixtures, migration, transaction/locking infrastructure, state/diff/staging, +commit/log/revert, then compatibility cleanup and full release validation.