Emergent structure
The map nobody drew
A memory store is a pile. The interesting question is whether structure has to be imposed on it — tags, folders, an ontology someone maintains — or whether a large enough pile of one developer's working knowledge already has a shape that can simply be read off it. This page is the measurement, taken on a live 144,159-vector store, with no labels and no supervision anywhere in the method.
Method
No labels, no taxonomy, no human in the loop.
Everything below comes out of one unsupervised pass. Nothing was tagged, nothing was named, and the algorithm was never told what a topic is.
Every chunk of memory — a curated note, a captured observation, a spec — is embedded as a 1024-dimensional unit vector. Mini-batch k-means then partitions those vectors, seeded to give roughly one centroid per 300 vectors, and refines the centroids incrementally as the store grows. That is the entire method. There is no supervision, no label set, and no ontology to maintain.
477 centroids are allocated; 424 of them actually hold vectors, because k-means does not guarantee every seed wins members. The content analysis below covers the 142,292 vectors (98.7% of the index) that still join to readable text, in 412 clusters — the remainder are chunks whose source document has since been removed. Percentages are computed over that 412-cluster set and are stated as such rather than rounded into one headline number.
The clustering exists for a practical reason — it is what lets search read a fraction of the corpus instead of all of it, measured on the Recall page. What follows is a side effect: once the partition exists, you can look at it, and it turns out to say something about the shape of the knowledge rather than just about the speed of the query.
Result one
The distribution is heavy-tailed, not flat.
Knowledge does not arrive in equal portions. A handful of concerns dominate; a long tail holds everything that happened once.
| Measure | Value | Reading |
|---|---|---|
| Largest cluster | 3,138 vectors | One concern the work returned to constantly |
| 90th percentile | 1,004 | The dozen or so real subject areas |
| Median | 132 | A normal working topic |
| Smallest | 1 | Something that happened once and never again |
| Top 50 clusters | 49.8% of the corpus | Half the store is 12% of the clusters |
This matters for a practical reason. A system that assumed uniform topics — fixed-size buckets, an even tag distribution — would be wrong about this corpus in both directions at once: it would split the dominant concerns and it would drown the singletons. The tail is not noise to be trimmed. A cluster of one is a fact learned once, and those are precisely the ones a person forgets.
Result two
Meaning does not respect your folder structure.
The strongest result here, and the least expected: the partition almost entirely ignores the boundary developers actually organise by.
| Clusters that… | Share |
|---|---|
| span more than one repository | 87% |
| span three or more repositories | 77% |
| mix curated notes with captured observations | 72% |
| median share held by their dominant repository | 68% |
Only 13% of clusters live inside a single repository. The median cluster draws roughly a third of its content from outside whichever repository dominates it, and for the bottom decile the dominant repository accounts for just 44% — those clusters are majority-foreign to their own home.
A directory tree is a statement about where files live. It turns out to be a poor statement about where knowledge lives. The same deployment mistake, the same rounding rule, the same misunderstanding about a protocol recurs across projects that share no code at all, and the embedding notices because it was never looking at paths in the first place.
The obvious objection is that these numbers describe a person who works on many things at once, and that a focused single-project developer would see the opposite. Probably true — and it is exactly the point. The 87% is not offered as a universal constant. It is evidence that when knowledge crosses project boundaries, an unsupervised partition finds that crossing without being told to look for it, which a folder-shaped or tag-shaped memory structurally cannot.
Result three
Three kinds of cluster, and only one of them is a topic.
Inspecting the contents by hand, the partition is not doing one thing. It is doing at least three, and nothing in the method distinguishes them.
1 · Topical — cluster 16, 3,138 vectors, four repositories
Federation and fleet design, spread acrosscaptain-memo, captain-memo-fed, captain-hub and loose notes.The expected kind. A subject, held together by subject matter.
2 · Methodological — cluster 8, 2,488 vectors
A capacity claim about an EPON port, overturned by measurement.Two speech-transcription engines compared, the earlier verdict reversed.
A parser whose headline-exclusion rule was found to be wrong.
These share no subject matter whatsoever.
What binds them is the epistemic act: in every one, a measurement contradicted a confident claim. The embedding grouped them by the shape of the reasoning rather than by what the reasoning was about. No taxonomy anyone designs has a folder called "times I was confidently wrong", and it is arguably the most valuable folder in the store.
3 · Episodic — cluster 185, 13 vectors, one repository
A dead privacy-policy link, found in email templates.A fix by dynamic query → the discovery that it touched 34 templates, not 5.
The dynamic query replaced again by an explicit list.
A single task's arc, including the reversal in the middle. Not a topic — an episode, preserved with its dead ends intact. This is the shape most knowledge bases destroy, because they record the conclusion and discard the route.
That one unsupervised partition produces all three is the finding. A hand-built taxonomy has to choose: organise by subject, by method, or by chronology. Choose one and the other two become unfindable. The embedding is not choosing, because it never had a schema to choose within.
Result four
The deliberate note and the incidental log land together.
72% of clusters contain both — a fact someone thought worth writing down, sitting beside the automatic record of the work that produced it.
Captain Memo holds two very different things. Curated memory is written on purpose: a decision, a convention, a trap worth remembering. Observations are captured automatically as work happens, without anyone deciding they matter. They are normally kept apart, because one is authored and the other is exhaust.
The partition does not keep them apart. In nearly three quarters of clusters they interleave, and the pairing is the useful part: the rule and the incident that produced the rule end up adjacent, so recalling either surfaces the other. A note saying "never round in intermediate calculations" is worth more when the afternoon that taught it is one row below.
What it is for
Substrate, not interpretation.
Clusters are a partition. They are read by two things that do the actual work, and it is worth keeping the three separate.
Search reads it for speed
A query compares itself to 477 centroids, then reads only the nearest handful of clusters instead of all 144,159 vectors. Measured: 1166 ms → 47.5 ms at recall@10 of 0.938.
Dreaming reads it for meaning
Consolidation looks for the same standing fact learned in separate sessions weeks apart, and writes a theme when it finds one. That is an interpretation, and a model has to sign off on it.
The distinction is worth holding onto. A cluster is a geometric fact: these vectors are near each other. A theme is a claim about your work: you keep relearning this. The first is free and always true. The second costs a model call and can be wrong, which is why it is proposed rather than asserted, and why it can be undone.
Limits
What this measurement does not show.
Stated plainly, because a result presented without its boundaries is a claim, not a measurement.
| Limit | Consequence |
|---|---|
| One corpus, one person | Every number here describes a single working store. They are evidence that this structure can emerge, not that it always does. |
| k is chosen, not discovered | 412 clusters follows from one centroid per ~300 vectors. It is a resolution setting. It is not a claim that the corpus contains 412 topics. |
| A partition is not a taxonomy | Every vector belongs to exactly one cluster. Ideas belong to several. k-means cannot express that, and does not try. |
| Centroids drift | Clusters are refined as the store grows, so this map is a snapshot. Re-measured next month it would differ, most at the boundaries. |
| The three kinds are my reading | Topical, methodological and episodic are labels applied by inspection afterwards. The algorithm produced groups; the interpretation is human and could be argued with. |
What survives all of that is narrow and, I think, still worth saying: a large enough record of one developer's work has structure in it already, that structure crosses the boundaries the work was filed under, and finding it needs no ontology, no tagging discipline, and nothing leaving the machine.
Reproduce it
On your own store.
The index is on by default above 3,000 vectors and builds in the background. Once it has, the partition is a plain SQLite table you can query.
Cluster membership lives in vec_chunks_p in your vector database, joinable to chunk text by chunk_id. Nothing about this is a service, and none of it leaves your machine — which is also why the numbers on this page are from one store rather than from many.