Division of labor: Qdrant proposes semantic candidates (dense + sparse), Neo4j decides typed meaning and topology. Collections, hybrid retrieval, and six exploration views.

title: "Qdrant + Neo4j Architecture for Latent-Structure Discovery" created: "2026-06-20" artifact_type: "system_architecture" status: "draft v0.1" purpose: "Specify how hybrid dense/sparse retrieval in Qdrant can cooperate with Neo4j graph topology to support epistemic emergence, hidden-structure prediction, and visual exploration."

Qdrant + Neo4j Architecture

0. Why Two Databases?

The system needs two different computational geometries.

Qdrant is for semantic retrieval

Use Qdrant for:

dense semantic search
sparse keyword/entity search
hybrid dense+sparse retrieval
multi-vector representations
chunk-level retrieval
source-family detection
analogy search
translation-equivalence discovery
near-duplicate detection
claim candidate retrieval

Qdrant is especially useful when language varies:

jinn / demon / archon / Mara / wetiko / intrusive thought / whisper / temptation

The surface words differ. Dense retrieval finds semantic kinship. Sparse retrieval preserves exact names, dates, and rare terms.

Neo4j is for typed structure

Use Neo4j for:

actors
events
claims
observations
bridges
mechanisms
canonical roles
graphlets
latent structures
posterior weights
causal constraints
temporal constraints
visual graph exploration
next-best-question topology

Neo4j is where semantic candidates become typed claims and edges.

1. Current Feature Assumptions

Qdrant supports collections with dense, sparse, and multivector configurations, including named vectors. Qdrant hybrid queries can combine dense and sparse prefetches and fuse them, for example with Reciprocal Rank Fusion.

Neo4j supports graph-native modeling, vector indexes for embeddings, and Graph Data Science node embeddings such as FastRP, GraphSAGE, Node2Vec, and HashGNN.

Architecture implication:

Qdrant = retrieval and semantic candidate generation
Neo4j = graph state, topology, inference, visualization

Do not force Qdrant to be the graph database.
Do not force Neo4j to be the primary chunk-retrieval engine.

2. Data Flow

Raw Source
  -> chunking / OCR / metadata extraction
  -> Qdrant points for chunks, claims, motifs, source excerpts
  -> LLM-assisted atomization into observations and claims
  -> Neo4j nodes/edges
  -> graphlet matching / latent structure prediction
  -> next-best-question generation
  -> Qdrant retrieval for evidence to fill graph cavities
  -> Neo4j update

3. Qdrant Collections

3.1 source_chunks

Chunk-level storage.

Payload:

chunk_id: string
source_id: string
source_family_id: string
document_title: string
author: string | null
date_created: date | null
date_observed: date
source_type: string
tradition: string | null
culture: string | null
language: string
chunk_text: string
page_or_section: string | null
ontology_layer_guess: list
risk_flags: list
neo4j_node_ids: list

Vectors:

dense_semantic: embedding(chunk_text)
sparse_lexical: BM25/SPLADE-style sparse vector
dense_claimlike: optional embedding optimized for proposition retrieval
dense_symbolic: optional embedding for mythic/motif similarity

3.2 claims

One point per atomized claim.

Payload:

claim_id: string
claim_text: string
claim_type: string
ontology_layer: string
status: string
source_ids: list
observation_ids: list
confidence_weight: float
salience_weight: float
structure_watch_score: float
fact_export_score: float
neo4j_node_id: string

Vectors:

dense_claim: embedding(claim_text)
sparse_claim: sparse vector for exact entity/date/term match

3.3 local_concepts

One point per local religious/cultural concept.

Payload:

local_concept_id: string
name: string
tradition: string
language: string
summary: string
canonical_role_candidates: list
neo4j_node_id: string

Vectors:

dense_concept: semantic embedding of concept description
sparse_concept: exact terms
dense_role_signature: embedding of mapped canonical roles

3.4 graphlets

One point per graphlet template or observed graphlet instance.

Payload:

graphlet_id: string
name: string
node_roles: list
edge_roles: list
required_edges: list
observed_case_ids: list
latent_predictions: list
neo4j_node_id: string

Vectors:

dense_graphlet_description
sparse_graphlet_terms

3.5 questions

One point per next-best question.

Payload:

question_id: string
question_text: string
target_uncertainty: string
hypotheses_discriminated: list
expected_information_gain: float
priority_score: float
status: open | answered | retired

Vectors:

dense_question
sparse_question

4. Hybrid Retrieval Strategy

4.1 Dense retrieval

Good for:

semantic similarity
cross-cultural analogy
paraphrase
conceptual recurrence
translation drift
motif discovery

Example query:

"hostile invisible influence that whispers destructive thoughts and is countered by awareness or prayer"

Dense search may retrieve:

waswasa
Mara
demonic temptation
intrusive thoughts
wetiko
archons
hungry ghosts

4.2 Sparse retrieval

Good for:

rare proper nouns
dates
institutions
titles
technical terms
exact phrases
legal names

Example query:

"Michael Aquino Presidio Privacy Act Stone MindWar"

Sparse search protects specificity.

4.3 Hybrid retrieval

Use hybrid when both concept and exactness matter.

Example:

"occult military psychological warfare Aquino Set MindWar"

Dense finds conceptually related PSYOP/cult material. Sparse anchors Aquino/Set/MindWar.

4.4 Multi-representation retrieval

For each source chunk, store several views:

literal text
claim summary
symbolic/motif summary
mechanism summary
historical metadata summary

This prevents a single embedding from flattening very different questions.

5. Qdrant → Neo4j Mapping

Qdrant retrieves candidates.

Neo4j decides what they mean.

Pipeline:

1. User asks question or graph predicts latent structure.
2. Query Qdrant hybrid collections.
3. Retrieve candidate chunks/claims/concepts.
4. Rerank by:
   - semantic score
   - sparse exactness
   - source independence
   - graph relevance
   - expected information gain
5. LLM or parser atomizes new observations.
6. Neo4j creates/updates:
   - Source
   - Observation
   - Claim
   - Bridge
   - Mechanism
   - LatentStructure
7. Graph scores update.

Mapping table:

6. Neo4j → Qdrant Mapping

Neo4j generates semantic retrieval targets.

Examples:

latent edge: Cult -> IntelligenceActor
missing document: court record or FOIA file
graphlet: manufactured criminality
unresolved claim: Dwyer CIA link
canonical role: whispering adversary

Neo4j converts these into retrieval prompts.

Example:

latent_structure_id: latent_edge_cult_intelligence_adjacency
retrieval_prompt: "religious cult political protection intelligence officer embassy anomalous official presence investigation"
sparse_terms:
  - CIA
  - embassy
  - deputy chief of mission
  - investigation
  - classified
dense_prompt:
  - "official state or intelligence adjacency to coercive religious-political cult"

Then Qdrant searches sources likely to fill that cavity.

7. User Experience

7.1 Main interface views

View A: Confidence vs salience map

Quadrants:

high confidence / high salience      core facts
high confidence / low salience       context
low confidence / high salience       watchlist
low confidence / low salience        archive

View B: Latent structure heatmap

Show predicted missing nodes/edges:

missing funding path
missing handler/intermediary
missing source family
missing translation equivalence
missing suppression event
missing ritual sequence

Each cell has:

latent_confidence
latent_salience
expected_information_gain
next-best question

View C: Graphlet recurrence matrix

Rows = subgraphs/cultures/cases.
Columns = graphlet templates.

Example:

                         Whisper  Inversion  Criminality  Suppression  Remedy
Christianity                .82       .40         .30          .55        .90
Islam                       .85       .25         .25          .45        .88
Buddhism                    .70       .20         .10          .30        .92
Jonestown                   .20       .45         .88          .80        .10
O9A                         .35       .75         .90          .55        .05

View D: Hypothesis dependency chain

Each big hypothesis shows required links.

Example:

Saturn neural influence
├── Saturn plasma environment                           strong
├── plasma self-organization                            medium
├── agency in plasma                                    weak
├── semantic signal propagation to Earth                very weak
├── coupling to human neural dynamics                   very weak
├── hostile semantic content                            very weak
└── historical cult interface                           very weak

View E: Source-family graph

Show whether evidence is independent or copied.

View F: Next-best-question queue

Ranked by expected information gain.

8. Workflow Examples

8.1 Cross-cultural motif discovery

User asks:

Find traditions with a whispering adversarial influence and awareness-based remedy.

System:

1. Qdrant dense retrieves concepts from many traditions.
2. Sparse search anchors terms: waswasa, Mara, demon, archon, wetiko.
3. Neo4j maps concepts to canonical roles.
4. Graphlet recurrence scored.
5. Interface shows convergence and divergences.

8.2 Hidden edge prediction

Neo4j notices:

Cult case has isolation/confession/transgression/identity-rebuild
but missing funding/intermediary edge.

System:

1. Creates LatentFundingPath node.
2. Generates search prompts.
3. Qdrant retrieves chunks mentioning donors, foundations, legal defense, foreign transfers.
4. Retrieved material either fills, weakens, or dissolves latent structure.

8.3 Consensus challenge

A mainstream source claims:

No intelligence relationship existed.

System stores this as:

Observation(role=denies)

It does not become final truth. It competes against:

witness claims
FOIA traces
court records
source omissions
timeline anomalies
later disclosures

The consensus hypothesis can lose if posterior evidence warrants.

9. Deployment Sketch

9.1 Ingestion pipeline

raw_docs/
  pdf
  html
  text
  markdown
  screenshots

extract/
  text
  metadata
  chunks
  source family detection

embed/
  dense vectors
  sparse vectors
  role-signature vectors

qdrant/
  upsert chunks
  upsert claims
  upsert concepts
  upsert graphlets

neo4j/
  upsert sources
  upsert observations
  upsert claims
  upsert edges
  recompute weights

9.2 Service components

ingest_service
claim_atomizer
role_mapper
qdrant_retriever
neo4j_writer
graphlet_matcher
latent_structure_detector
next_best_question_ranker
visualization_api

9.3 Minimal stack

Python
Qdrant
Neo4j
FastAPI
Pydantic schemas
LLM extraction layer
Graph visualization: Neo4j Bloom / custom D3 / Cytoscape
Notebook research layer

10. Why Qdrant May Be Better for Part of This

A pure Neo4j system will struggle with:

fuzzy language
cross-cultural paraphrase
large raw text corpora
symbolic similarity
translation drift
exact + semantic retrieval together

Qdrant solves the retrieval problem.

A pure Qdrant system will struggle with:

typed causality
temporal constraints
bridge-quality scoring
source-family dependency
graphlet recurrence
latent edge prediction
visual topology

Neo4j solves the structure problem.

The hybrid design is therefore natural:

Qdrant finds possible meaning.
Neo4j decides structural meaning.
Qdrant searches for missing evidence.
Neo4j updates posterior structure.

11. Open Design Questions

1. Should all Neo4j nodes also have embeddings stored in Neo4j vector indexes?
2. Should Qdrant own all embeddings, with Neo4j storing only Qdrant point IDs?
3. Should graph embeddings from Neo4j GDS be pushed back into Qdrant as structural vectors?
4. Should each claim have multiple embeddings: literal, mechanism, symbolic, and legal?
5. Should user exploration begin from search, graph, or next-best-question queue?

Best initial answer:

Store raw/chunk/claim embeddings in Qdrant.
Store graph structure in Neo4j.
Optionally compute Neo4j graph embeddings and push them into Qdrant as a separate "structural_embedding" vector.

12. Product Principle

The user should never be forced to choose between:

semantic search
graph traversal
evidence inspection
hypothesis testing
visual exploration

The interface should let them move fluidly:

search phrase
-> retrieved observations
-> claim nodes
-> graphlet recurrence
-> latent structure
-> next-best question
-> new retrieval

That loop is the research engine.