Initial commit
This commit is contained in:
@@ -0,0 +1,26 @@
|
||||
# GUPI OSINT Board roadmap
|
||||
|
||||
This list tracks domain and product work that follows the accepted exhibit model. It is not a substitute for migrations or implementation issues.
|
||||
|
||||
## Exhibit-schema cutover
|
||||
|
||||
- [ ] Add boards, exhibits, exhibit types, subtype tables, immutable template versions, and cloned mutable levels.
|
||||
- [ ] Migrate transitional `widgets`, `widget_relations`, and `playthrough_*` data with equivalence checks.
|
||||
- [ ] Implement template instantiation and “save level as template” as transactional clone operations.
|
||||
- [ ] Move the frontend to an exhibit/widget registry backed by the normalized API.
|
||||
|
||||
## Party exhibits
|
||||
|
||||
- [ ] Add a Party exhibit supertype representing an investigation participant.
|
||||
- [ ] Add a Person subtype with display name, normalized aliases, and extensible structured identity fields.
|
||||
- [ ] Add an Organization subtype covering businesses, public bodies, associations, and informal groups.
|
||||
- [ ] Add normalized many-to-many Party-to-Evidence associations with an optional explanatory note.
|
||||
- [ ] Add typed Party-to-Party relationships such as employment, ownership, membership, control, and representation.
|
||||
- [ ] Build distinct Person and Organization widgets that can open as dossiers and reveal associated evidence without using folder ownership semantics.
|
||||
- [ ] Include parties, aliases, evidence associations, and party relationships in template/level cloning.
|
||||
|
||||
## Events and narrative
|
||||
|
||||
- [ ] Implement Event exhibits with occurrence time and investigator-authored narrative text.
|
||||
- [ ] Implement normalized Event-to-Evidence links and their board visualization.
|
||||
- [ ] Present chronologically ordered events as the emerging investigation story.
|
||||
@@ -0,0 +1,330 @@
|
||||
# GUPI OSINT Board: canonical exhibit data model
|
||||
|
||||
Status: accepted design foundation.
|
||||
|
||||
## Vocabulary
|
||||
|
||||
- An **exhibit** is a durable investigation-domain object stored in PostgreSQL.
|
||||
- An **exhibit type** describes its domain behavior: folder, document, clipping, note, event, party, or conclusion.
|
||||
- A **widget** is the frontend visualization and interaction implementation selected for an exhibit type.
|
||||
- A **document type** specializes a document exhibit: image, PDF, web capture, email, article, filing, price list, text, or generic file.
|
||||
- A **board** is a neutral container for exhibits. Both mutable levels and immutable template versions own boards.
|
||||
- A **level** is a mutable board copy used for either play or authoring.
|
||||
- A **level template version** is an immutable board snapshot.
|
||||
|
||||
The frontend registry maps `exhibit_type` (and, for documents, `document_type`) to a widget component. Changing a widget must never change the meaning or storage of its exhibit.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
Templates are never edited in place.
|
||||
|
||||
1. A template family has one or more immutable versions.
|
||||
2. Starting work creates a new level and clones the selected template-version board into it.
|
||||
3. Every cloned exhibit receives a new ID, belongs to the new level board, and retains provenance through `origin_exhibit_id`.
|
||||
4. Playing and editing write to exactly the same level and exhibit tables.
|
||||
5. “Save as template” clones the current level board into a new immutable template version.
|
||||
6. Binary assets are immutable and shared; document exhibits are copied, asset bytes are not.
|
||||
|
||||
No long-term `playthrough_*` overlay is required. A level is already the user's isolated working copy.
|
||||
|
||||
## Core structure
|
||||
|
||||
```sql
|
||||
CREATE TABLE osint.boards (
|
||||
id UUID PRIMARY KEY,
|
||||
board_kind TEXT NOT NULL CHECK (board_kind IN ('level', 'template_version')),
|
||||
revision BIGINT NOT NULL DEFAULT 0,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE osint.level_templates (
|
||||
id UUID PRIMARY KEY,
|
||||
slug TEXT NOT NULL UNIQUE,
|
||||
name TEXT NOT NULL,
|
||||
current_version_id UUID,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE osint.level_template_versions (
|
||||
id UUID PRIMARY KEY,
|
||||
template_id UUID NOT NULL REFERENCES osint.level_templates(id),
|
||||
version INTEGER NOT NULL,
|
||||
board_id UUID NOT NULL UNIQUE REFERENCES osint.boards(id),
|
||||
title TEXT NOT NULL,
|
||||
subtitle TEXT NOT NULL DEFAULT '',
|
||||
created_from_level_id UUID,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
UNIQUE (template_id, version)
|
||||
);
|
||||
|
||||
CREATE TABLE osint.levels (
|
||||
id UUID PRIMARY KEY,
|
||||
slug TEXT NOT NULL UNIQUE,
|
||||
board_id UUID NOT NULL UNIQUE REFERENCES osint.boards(id),
|
||||
source_template_version_id UUID REFERENCES osint.level_template_versions(id),
|
||||
title TEXT NOT NULL,
|
||||
subtitle TEXT NOT NULL DEFAULT '',
|
||||
status TEXT NOT NULL,
|
||||
viewport_x DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
viewport_y DOUBLE PRECISION NOT NULL DEFAULT 28,
|
||||
viewport_zoom DOUBLE PRECISION NOT NULL DEFAULT 0.7,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
`boards.revision` is incremented transactionally and used for optimistic concurrency control.
|
||||
|
||||
## Exhibits
|
||||
|
||||
Common spatial and lifecycle properties belong to the base exhibit, not to its widget or subtype.
|
||||
|
||||
```sql
|
||||
CREATE TABLE osint.exhibit_types (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
is_spatial BOOLEAN NOT NULL DEFAULT TRUE
|
||||
);
|
||||
|
||||
CREATE TABLE osint.exhibits (
|
||||
id UUID PRIMARY KEY,
|
||||
board_id UUID NOT NULL REFERENCES osint.boards(id) ON DELETE CASCADE,
|
||||
exhibit_type_id TEXT NOT NULL REFERENCES osint.exhibit_types(id),
|
||||
origin_exhibit_id UUID REFERENCES osint.exhibits(id) ON DELETE SET NULL,
|
||||
xpos DOUBLE PRECISION NOT NULL,
|
||||
ypos DOUBLE PRECISION NOT NULL,
|
||||
width DOUBLE PRECISION NOT NULL,
|
||||
height DOUBLE PRECISION NOT NULL,
|
||||
rotation DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
z_index INTEGER NOT NULL DEFAULT 0,
|
||||
hidden BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
UNIQUE (board_id, id)
|
||||
);
|
||||
```
|
||||
|
||||
The common table is extended with one-to-one subtype tables. Domain fields do not live in an unstructured `config` document.
|
||||
|
||||
```sql
|
||||
CREATE TABLE osint.folder_exhibits (
|
||||
exhibit_id UUID PRIMARY KEY REFERENCES osint.exhibits(id) ON DELETE CASCADE,
|
||||
title TEXT NOT NULL,
|
||||
label_text TEXT NOT NULL DEFAULT '',
|
||||
is_open BOOLEAN NOT NULL DEFAULT FALSE
|
||||
);
|
||||
|
||||
CREATE TABLE osint.document_types (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE
|
||||
);
|
||||
|
||||
CREATE TABLE osint.document_exhibits (
|
||||
exhibit_id UUID PRIMARY KEY REFERENCES osint.exhibits(id) ON DELETE CASCADE,
|
||||
document_type_id TEXT NOT NULL REFERENCES osint.document_types(id),
|
||||
asset_id UUID REFERENCES osint.assets(id),
|
||||
title TEXT NOT NULL,
|
||||
published_at TIMESTAMPTZ,
|
||||
captured_at TIMESTAMPTZ,
|
||||
source_uri TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE osint.image_documents (
|
||||
exhibit_id UUID PRIMARY KEY REFERENCES osint.document_exhibits(exhibit_id) ON DELETE CASCADE,
|
||||
pixel_width INTEGER CHECK (pixel_width > 0),
|
||||
pixel_height INTEGER CHECK (pixel_height > 0),
|
||||
alt_text TEXT NOT NULL DEFAULT ''
|
||||
);
|
||||
|
||||
CREATE TABLE osint.note_exhibits (
|
||||
exhibit_id UUID PRIMARY KEY REFERENCES osint.exhibits(id) ON DELETE CASCADE,
|
||||
title TEXT NOT NULL,
|
||||
note_text TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE osint.event_exhibits (
|
||||
exhibit_id UUID PRIMARY KEY REFERENCES osint.exhibits(id) ON DELETE CASCADE,
|
||||
title TEXT NOT NULL,
|
||||
narrative_text TEXT NOT NULL,
|
||||
occurred_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
The service validates that every base exhibit has exactly one subtype row matching `exhibit_type_id`.
|
||||
|
||||
### Event semantics
|
||||
|
||||
An event is an investigator-authored assertion: **“this happened.”** It is not source evidence and must not silently inherit a document's publication time.
|
||||
|
||||
- `narrative_text` states what the investigator believes happened.
|
||||
- `occurred_at` places that assertion in reconstructed time.
|
||||
- One event may cite several supporting exhibits.
|
||||
- One exhibit may support several events.
|
||||
- Events ordered by `occurred_at` form the emerging case narrative; there is no duplicated story-text record.
|
||||
|
||||
Supporting evidence is an explicit normalized relationship:
|
||||
|
||||
```sql
|
||||
CREATE TABLE osint.event_evidence (
|
||||
event_exhibit_id UUID NOT NULL REFERENCES osint.event_exhibits(exhibit_id) ON DELETE CASCADE,
|
||||
evidence_exhibit_id UUID NOT NULL REFERENCES osint.exhibits(id) ON DELETE CASCADE,
|
||||
sort_order INTEGER NOT NULL DEFAULT 0,
|
||||
note TEXT,
|
||||
PRIMARY KEY (event_exhibit_id, evidence_exhibit_id),
|
||||
CHECK (event_exhibit_id <> evidence_exhibit_id)
|
||||
);
|
||||
```
|
||||
|
||||
A deferred constraint trigger verifies that both exhibits belong to the same board and that the evidence endpoint is not itself the same event. The optional `note` explains why that exhibit supports the event; it does not replace the event's narrative.
|
||||
|
||||
In the frontend, the Event widget shows its occurrence time, narrative text, and evidence count. Opening or selecting it reveals its supporting exhibits. Lines between an event and its evidence visualize `event_evidence`; they are not ordinary folder containment bands.
|
||||
|
||||
### Party semantics (planned)
|
||||
|
||||
A party is a person or organization that participates in the investigation. Businesses are organizations. A party is a first-class exhibit, not a special folder: identity and evidence association must not be represented as file ownership.
|
||||
|
||||
- A **person** has a display name and may have aliases and other structured identity fields.
|
||||
- An **organization** has a display name and an organization kind such as business, public body, association, or informal group.
|
||||
- A party may be linked to several supporting exhibits, and one exhibit may concern several parties.
|
||||
- Relationships between parties—employment, ownership, membership, control, representation, or an investigator-defined connection—are explicit typed relationships rather than containment.
|
||||
|
||||
The intended normalized shape is a `party_exhibits` supertype with one-to-one `person_parties` and `organization_parties` subtype tables, plus a many-to-many `party_evidence` association. Aliases and party-to-party relationships use child tables rather than arrays or widget configuration JSON.
|
||||
|
||||
The frontend provides distinct Person and Organization widgets through the exhibit registry. They may visually behave like dossiers—opening one can reveal associated evidence—but that interaction is derived from `party_evidence`; it does not turn the party into a folder or cause evidence to be owned by or disappear into the party.
|
||||
|
||||
## Assets and document content
|
||||
|
||||
`assets` stores immutable uploaded bytes, checksum, MIME type, original filename, and size. Multiple cloned document exhibits may reference one asset.
|
||||
|
||||
Document text and extractable regions are normalized separately:
|
||||
|
||||
```sql
|
||||
CREATE TABLE osint.document_content_blocks (
|
||||
id UUID PRIMARY KEY,
|
||||
document_exhibit_id UUID NOT NULL REFERENCES osint.document_exhibits(exhibit_id) ON DELETE CASCADE,
|
||||
sort_order INTEGER NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
UNIQUE (document_exhibit_id, sort_order)
|
||||
);
|
||||
|
||||
CREATE TABLE osint.document_regions (
|
||||
id UUID PRIMARY KEY,
|
||||
document_exhibit_id UUID NOT NULL REFERENCES osint.document_exhibits(exhibit_id) ON DELETE CASCADE,
|
||||
region_key TEXT NOT NULL,
|
||||
label TEXT NOT NULL,
|
||||
excerpt TEXT NOT NULL,
|
||||
occurred_at TIMESTAMPTZ,
|
||||
sort_order INTEGER NOT NULL DEFAULT 0,
|
||||
UNIQUE (document_exhibit_id, region_key)
|
||||
);
|
||||
```
|
||||
|
||||
## Folder ownership
|
||||
|
||||
Folder containment is a domain relationship, not a generic visual connection.
|
||||
|
||||
```sql
|
||||
CREATE TABLE osint.folder_memberships (
|
||||
board_id UUID NOT NULL REFERENCES osint.boards(id) ON DELETE CASCADE,
|
||||
folder_exhibit_id UUID NOT NULL REFERENCES osint.folder_exhibits(exhibit_id) ON DELETE CASCADE,
|
||||
child_exhibit_id UUID NOT NULL REFERENCES osint.exhibits(id) ON DELETE CASCADE,
|
||||
sort_order INTEGER NOT NULL DEFAULT 0,
|
||||
PRIMARY KEY (folder_exhibit_id, child_exhibit_id),
|
||||
UNIQUE (board_id, child_exhibit_id),
|
||||
CHECK (folder_exhibit_id <> child_exhibit_id)
|
||||
);
|
||||
```
|
||||
|
||||
For the POC, one exhibit has at most one owning folder. This makes open/close behavior deterministic. If the same uploaded image must appear in two folders, two document exhibits reference the same immutable asset.
|
||||
|
||||
Composite foreign keys or deferred constraint triggers enforce that folder, child, and membership all belong to the same board and that folders cannot contain themselves or form containment cycles.
|
||||
|
||||
The pale red folder band is derived from this table. Its geometry is not stored.
|
||||
|
||||
## Investigative connections and provenance
|
||||
|
||||
```sql
|
||||
CREATE TABLE osint.connection_types (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
directed BOOLEAN NOT NULL DEFAULT FALSE
|
||||
);
|
||||
|
||||
CREATE TABLE osint.exhibit_connections (
|
||||
id UUID PRIMARY KEY,
|
||||
board_id UUID NOT NULL REFERENCES osint.boards(id) ON DELETE CASCADE,
|
||||
connection_type_id TEXT NOT NULL REFERENCES osint.connection_types(id),
|
||||
from_exhibit_id UUID NOT NULL REFERENCES osint.exhibits(id) ON DELETE CASCADE,
|
||||
to_exhibit_id UUID NOT NULL REFERENCES osint.exhibits(id) ON DELETE CASCADE,
|
||||
label TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CHECK (from_exhibit_id <> to_exhibit_id)
|
||||
);
|
||||
|
||||
CREATE TABLE osint.exhibit_sources (
|
||||
exhibit_id UUID PRIMARY KEY REFERENCES osint.exhibits(id) ON DELETE CASCADE,
|
||||
source_document_exhibit_id UUID NOT NULL REFERENCES osint.document_exhibits(exhibit_id),
|
||||
source_region_id UUID REFERENCES osint.document_regions(id),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
Red investigative thread is derived from `exhibit_connections`. Extraction provenance is stored in `exhibit_sources`; it is not represented as a visual thread.
|
||||
|
||||
## Document metadata
|
||||
|
||||
Known, semantically important values remain real columns: `published_at`, `captured_at`, and `source_uri`. Truly author-defined fields use typed metadata definitions and values rather than JSONB or one untyped EAV table.
|
||||
|
||||
```sql
|
||||
CREATE TABLE osint.metadata_fields (
|
||||
id UUID PRIMARY KEY,
|
||||
key TEXT NOT NULL UNIQUE,
|
||||
label TEXT NOT NULL,
|
||||
value_type TEXT NOT NULL CHECK (value_type IN ('text', 'timestamp', 'number', 'boolean'))
|
||||
);
|
||||
|
||||
CREATE TABLE osint.document_metadata_text_values (
|
||||
document_exhibit_id UUID NOT NULL REFERENCES osint.document_exhibits(exhibit_id) ON DELETE CASCADE,
|
||||
field_id UUID NOT NULL REFERENCES osint.metadata_fields(id),
|
||||
value TEXT NOT NULL,
|
||||
PRIMARY KEY (document_exhibit_id, field_id)
|
||||
);
|
||||
```
|
||||
|
||||
Timestamp, number, and boolean values use equivalent type-specific tables. This lets PostgreSQL enforce actual value types and avoids nullable multi-type value columns.
|
||||
|
||||
## Derived presentation
|
||||
|
||||
The following are computed and must not become duplicate source-of-truth tables:
|
||||
|
||||
- Grey timeline projection: document exhibit position to `published_at` on the timeline.
|
||||
- Pale red containment band: folder position to contained exhibit position.
|
||||
- Folder flight animation: closed folder position to the exhibit's stored `xpos` and `ypos`.
|
||||
- Widget selection: `exhibit_type` plus optional `document_type` mapped through the frontend registry.
|
||||
|
||||
## Transactional operations
|
||||
|
||||
`instantiate_template_version(source_version_id)` performs one database transaction:
|
||||
|
||||
1. Create the level board and level row.
|
||||
2. Clone all exhibits and build an old-to-new ID map.
|
||||
3. Clone each subtype row.
|
||||
4. Clone content, typed metadata, and provenance through the map.
|
||||
5. Clone folder memberships, event-evidence links, and exhibit connections through the map.
|
||||
6. Reuse asset IDs.
|
||||
7. Commit only after all constraints pass.
|
||||
|
||||
`save_level_as_template(level_id)` runs the same clone operation into a new immutable template-version board.
|
||||
|
||||
## Migration direction
|
||||
|
||||
The current `widgets`, `widget_relations`, and `playthrough_*` tables are transitional. The cutover should:
|
||||
|
||||
1. Introduce boards, exhibit tables, subtype tables, templates, and typed metadata.
|
||||
2. Convert existing authored widgets to exhibits.
|
||||
3. Materialize each existing playthrough as its own cloned mutable level.
|
||||
4. Move document layout from relation JSON into document exhibit coordinates.
|
||||
5. Convert `contains` relations to folder memberships and generic connections to exhibit connections.
|
||||
6. Update the API to read and write exhibits directly.
|
||||
7. Remove the transitional widget/playthrough tables only after data equivalence checks pass.
|
||||
Reference in New Issue
Block a user