Levandor

KB semantics phase 2 — implementation plan

14 min read

KB Semantics Phase 2 Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax.

Goal: Cluster manual sections that mean the same thing across manuals (local, HDBSCAN over section vectors), and expose the clusters as a /semantics web page in the knowledgebase app + a dashboard tile.

Architecture: Extend /home/levander/kb-vectors/ — pull phase-1 chunk vectors from Qdrant, mean-pool to section vectors, HDBSCAN cluster them, label locally from headings, write clusters.json. Add a read-only /semantics route to the running knowledgebase Flask app and a dashboard tile.

Tech Stack: Python venv (numpy, scikit-learn, hdbscan, qdrant-client — all in the kb-vectors venv), Qdrant manuals collection, the knowledgebase Flask app, the home-portal static dashboard.

Design: 2026-07-24-kb-semantics-phase2-design

Global Constraints

  • Code under /home/levander/kb-vectors/ (cluster pipeline) and edits to /home/levander/knowledgebase/app.py + /home/levander/home-portal/index.html. Runs as levander, NOT root (Qdrant reachable via docker as levander; app edits are levander-owned).
  • No comments/docstrings/annotations in any code (hard user rule). Self-explanatory naming only.
  • Do not run git commit. Each task ends in verification.
  • Read-only over Qdrant and the KB docs — must not modify the manuals collection, the vectors, or any manual file.
  • Reuse the existing kb-vectors venv; add deps: scikit-learn hdbscan numpy (numpy already present via sentence-transformers).
  • CPU + thread-capped: prefix heavy python with OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 and nice -n 10 (per the PSU-load lesson). Serialize vs other heavy jobs.
  • Section = one NNN-*.md page = points sharing (manual_id, page) payload. Section vector = L2-normalized mean of its chunk vectors.
  • HDBSCAN: metric="euclidean" on unit-norm vectors (≈ cosine), min_cluster_size=2 default. Noise label = -1, excluded from clusters but counted.
  • clusters.json lives at /home/levander/kb-vectors/clusters.json.
  • The knowledgebase app currently binds 127.0.0.1:8092 and serves the mkdocs site/ at / plus /upload, /jobs, /api/jobs. New routes must not shadow the static catch-all /<path:path> — register specific routes (Flask/Werkzeug orders them first, but verify).
  • Run commands on the box: /usr/bin/ssh levander@telep-mainframe. On this Mac bare ssh is broken — always /usr/bin/ssh.

Test command: cd /home/levander/kb-vectors && venv/bin/python -m unittest <module> -v

Task 1: Deps + sectionvecs.py (pull + mean-pool section vectors)

Files:

  • Create: /home/levander/kb-vectors/sectionvecs.py
  • Test: /home/levander/kb-vectors/test_sectionvecs.py

Interfaces:

  • Produces:

    • mean_pool(vectors) -> list[float] — L2-normalized mean of a list of equal-length vectors
    • group_sections(points) -> tuple[list[list[float]], list[dict]] — from a list of {"vector":[...], "payload":{manual_id,folder,page,page_url,heading}}, group by (manual_id, page) (sorted), return (section_vectors, section_meta) where each meta = {manual_id, folder, page, page_url, heading}
    • load_sections(client) -> tuple[list[list[float]], list[dict]] — scroll all points from Qdrant manuals (with vectors), call group_sections

  • Step 1: Install deps

/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/kb-vectors && venv/bin/pip install -q scikit-learn hdbscan && venv/bin/python -c "import sklearn, hdbscan, numpy; print(\"sklearn\", sklearn.__version__, \"hdbscan ok\", \"numpy\", numpy.__version__)"'

Expected: prints versions, no import error. (hdbscan may compile; be patient.)

  • Step 2: Write the failing test

Create /home/levander/kb-vectors/test_sectionvecs.py:

import math
import unittest
 
from sectionvecs import group_sections, mean_pool
 
 
def norm(v):
    return math.sqrt(sum(x * x for x in v))
 
 
class TestMeanPool(unittest.TestCase):
    def test_normalized(self):
        out = mean_pool([[3.0, 0.0], [0.0, 0.0]])
        self.assertAlmostEqual(norm(out), 1.0, places=6)
 
    def test_mean_direction(self):
        out = mean_pool([[1.0, 0.0], [0.0, 1.0]])
        self.assertAlmostEqual(out[0], out[1], places=6)
 
 
class TestGroupSections(unittest.TestCase):
    def _points(self):
        return [
            {"vector": [1.0, 0.0], "payload": {"manual_id": "m/a", "folder": "m", "page": "001-x", "page_url": "m/a/001-x/", "heading": "X"}},
            {"vector": [0.0, 1.0], "payload": {"manual_id": "m/a", "folder": "m", "page": "001-x", "page_url": "m/a/001-x/", "heading": "X"}},
            {"vector": [1.0, 0.0], "payload": {"manual_id": "m/b", "folder": "m", "page": "002-y", "page_url": "m/b/002-y/", "heading": "Y"}},
        ]
 
    def test_two_sections(self):
        vecs, meta = group_sections(self._points())
        self.assertEqual(len(vecs), 2)
        self.assertEqual(len(meta), 2)
 
    def test_meta_fields(self):
        vecs, meta = group_sections(self._points())
        self.assertEqual(meta[0]["manual_id"], "m/a")
        self.assertEqual(meta[0]["page_url"], "m/a/001-x/")
        self.assertEqual(meta[0]["heading"], "X")
 
    def test_pooled_normalized(self):
        vecs, meta = group_sections(self._points())
        self.assertAlmostEqual(norm(vecs[0]), 1.0, places=6)
 
    def test_sorted_deterministic(self):
        vecs, meta = group_sections(self._points())
        self.assertEqual([m["manual_id"] for m in meta], ["m/a", "m/b"])
 
 
if __name__ == "__main__":
    unittest.main()
  • Step 3: Run the test to verify it fails
/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/kb-vectors && venv/bin/python -m unittest test_sectionvecs -v'

Expected: FAIL with ModuleNotFoundError: No module named 'sectionvecs'

  • Step 4: Write the implementation

Create /home/levander/kb-vectors/sectionvecs.py:

import math
 
from index import COLLECTION
 
 
def mean_pool(vectors):
    dim = len(vectors[0])
    total = [0.0] * dim
    for vector in vectors:
        for i in range(dim):
            total[i] += vector[i]
    mean = [x / len(vectors) for x in total]
    length = math.sqrt(sum(x * x for x in mean))
    if length == 0:
        return mean
    return [x / length for x in mean]
 
 
def group_sections(points):
    groups = {}
    order = []
    for point in points:
        payload = point["payload"]
        key = (payload["manual_id"], payload["page"])
        if key not in groups:
            groups[key] = {"vectors": [], "meta": {
                "manual_id": payload["manual_id"],
                "folder": payload.get("folder", payload["manual_id"].split("/")[0]),
                "page": payload["page"],
                "page_url": payload["page_url"],
                "heading": payload.get("heading", ""),
            }}
            order.append(key)
        groups[key]["vectors"].append(point["vector"])
    order.sort()
    section_vectors = []
    section_meta = []
    for key in order:
        section_vectors.append(mean_pool(groups[key]["vectors"]))
        section_meta.append(groups[key]["meta"])
    return section_vectors, section_meta
 
 
def load_sections(client):
    points = []
    offset = None
    while True:
        batch, offset = client.scroll(
            COLLECTION, limit=512, offset=offset, with_vectors=True, with_payload=True
        )
        for record in batch:
            points.append({"vector": list(record.vector), "payload": dict(record.payload)})
        if offset is None:
            break
    return group_sections(points)
  • Step 5: Run the test to verify it passes
/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/kb-vectors && venv/bin/python -m unittest test_sectionvecs -v'

Expected: OK — 6 tests pass.

  • Step 6: Verify against the real corpus
/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/kb-vectors && OMP_NUM_THREADS=4 venv/bin/python -c "
from index import get_client
from sectionvecs import load_sections
v, m = load_sections(get_client())
print(\"sections:\", len(v), \"| dim:\", len(v[0]))
print(\"distinct manuals:\", len(set(x[\"manual_id\"] for x in m)))
"'

Expected: a few hundred sections, dim 1024, ~15+ distinct manuals.

Task 2: labels.py (representative pick + local label)

Files:

  • Create: /home/levander/kb-vectors/labels.py
  • Test: /home/levander/kb-vectors/test_labels.py

Interfaces:

  • Consumes: nothing

  • Produces:

    • representative_index(member_indices, vectors) -> int — the member index whose vector is nearest (max dot product) to the members’ normalized centroid
    • make_label(rep_meta) -> str — rep_meta["heading"] trimmed; if empty, a prettified page (strip NNN- prefix, hyphens→spaces, title-ish)

  • Step 1: Write the failing test

Create /home/levander/kb-vectors/test_labels.py:

import unittest
 
from labels import make_label, representative_index
 
 
class TestRepresentative(unittest.TestCase):
    def test_picks_central_member(self):
        vectors = [[1.0, 0.0], [0.9, 0.1], [-1.0, 0.0]]
        idx = representative_index([0, 1], vectors)
        self.assertIn(idx, [0, 1])
 
    def test_central_of_three(self):
        vectors = [[1.0, 0.0], [0.0, 1.0], [0.71, 0.71]]
        idx = representative_index([0, 1, 2], vectors)
        self.assertEqual(idx, 2)
 
 
class TestMakeLabel(unittest.TestCase):
    def test_heading_used(self):
        self.assertEqual(make_label({"heading": "Rear Axle", "page": "007-rear-axle"}), "Rear Axle")
 
    def test_empty_heading_fallback_to_page(self):
        self.assertEqual(make_label({"heading": "", "page": "007-rear-axle-service"}), "Rear Axle Service")
 
    def test_strips_numeric_prefix(self):
        self.assertEqual(make_label({"heading": "  ", "page": "012-brake-bleeding"}), "Brake Bleeding")
 
 
if __name__ == "__main__":
    unittest.main()
  • Step 2: Run the test to verify it fails
/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/kb-vectors && venv/bin/python -m unittest test_labels -v'

Expected: FAIL with ModuleNotFoundError: No module named 'labels'

  • Step 3: Write the implementation

Create /home/levander/kb-vectors/labels.py:

import math
import re
 
_PREFIX = re.compile(r"^\d+-")
 
 
def representative_index(member_indices, vectors):
    dim = len(vectors[member_indices[0]])
    centroid = [0.0] * dim
    for i in member_indices:
        for d in range(dim):
            centroid[d] += vectors[i][d]
    length = math.sqrt(sum(x * x for x in centroid))
    if length:
        centroid = [x / length for x in centroid]
    best = member_indices[0]
    best_score = None
    for i in member_indices:
        score = sum(vectors[i][d] * centroid[d] for d in range(dim))
        if best_score is None or score > best_score:
            best_score = score
            best = i
    return best
 
 
def make_label(rep_meta):
    heading = (rep_meta.get("heading") or "").strip()
    if heading:
        return heading
    page = _PREFIX.sub("", rep_meta.get("page", ""))
    return " ".join(word.capitalize() for word in page.replace("-", " ").split())
  • Step 4: Run the test to verify it passes
/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/kb-vectors && venv/bin/python -m unittest test_labels -v'

Expected: OK — 5 tests pass.

Task 3: cluster.py + kbclust.py (HDBSCAN + artifact + CLI)

Files:

  • Create: /home/levander/kb-vectors/cluster.py
  • Create: /home/levander/kb-vectors/kbclust.py
  • Test: /home/levander/kb-vectors/test_cluster.py

Interfaces:

  • Consumes: sectionvecs.load_sections, labels.representative_index, labels.make_label

  • Produces:

    • build_clusters(vectors, meta, min_cluster_size=2) -> tuple[list[dict], int] — returns (clusters, noise_count); each cluster {id, label, size, span, source_manuals, members:[{manual_id, page_url, heading}]}, sorted by span desc then size desc, id = position in that order
    • write_clusters(clusters, path) / load_clusters(path) -> list[dict]
    • CLUSTERS_PATH = "/home/levander/kb-vectors/clusters.json"

  • Step 1: Write the failing test

Create /home/levander/kb-vectors/test_cluster.py:

import unittest
 
from cluster import build_clusters
 
 
def meta(manual_id, page, heading):
    return {"manual_id": manual_id, "folder": manual_id.split("/")[0], "page": page, "page_url": manual_id + "/" + page + "/", "heading": heading}
 
 
class TestBuildClusters(unittest.TestCase):
    def _data(self):
        vectors = [
            [1.0, 0.0, 0.0], [0.99, 0.01, 0.0], [0.98, 0.02, 0.0],
            [0.0, 1.0, 0.0], [0.0, 0.99, 0.01],
        ]
        metas = [
            meta("vitara/workshop", "001-diff", "Differential"),
            meta("chevy/geo", "005-diff", "Rear axle"),
            meta("kick-fix/axles", "001-axle", "Axle"),
            meta("vitara/workshop", "010-brake", "Brakes"),
            meta("chevy/geo", "020-brake", "Brake bleed"),
        ]
        return vectors, metas
 
    def test_forms_clusters(self):
        vectors, metas = self._data()
        clusters, noise = build_clusters(vectors, metas, min_cluster_size=2)
        self.assertGreaterEqual(len(clusters), 2)
 
    def test_cross_manual_span(self):
        vectors, metas = self._data()
        clusters, noise = build_clusters(vectors, metas, min_cluster_size=2)
        diff = [c for c in clusters if any("diff" in m["page_url"] or "axle" in m["page_url"] for m in c["members"])][0]
        self.assertGreaterEqual(diff["span"], 2)
        self.assertEqual(diff["span"], len(diff["source_manuals"]))
 
    def test_sorted_by_span(self):
        vectors, metas = self._data()
        clusters, noise = build_clusters(vectors, metas, min_cluster_size=2)
        spans = [c["span"] for c in clusters]
        self.assertEqual(spans, sorted(spans, reverse=True))
 
    def test_ids_sequential(self):
        vectors, metas = self._data()
        clusters, noise = build_clusters(vectors, metas, min_cluster_size=2)
        self.assertEqual([c["id"] for c in clusters], list(range(len(clusters))))
 
    def test_member_shape(self):
        vectors, metas = self._data()
        clusters, noise = build_clusters(vectors, metas, min_cluster_size=2)
        m = clusters[0]["members"][0]
        self.assertEqual(set(m.keys()), {"manual_id", "page_url", "heading"})
 
 
if __name__ == "__main__":
    unittest.main()
  • Step 2: Run the test to verify it fails
/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/kb-vectors && venv/bin/python -m unittest test_cluster -v'

Expected: FAIL with ModuleNotFoundError: No module named 'cluster'

  • Step 3: Write cluster.py

Create /home/levander/kb-vectors/cluster.py:

import json
 
import hdbscan
import numpy as np
 
from labels import make_label, representative_index
 
CLUSTERS_PATH = "/home/levander/kb-vectors/clusters.json"
 
 
def build_clusters(vectors, meta, min_cluster_size=2):
    array = np.array(vectors, dtype=np.float64)
    labeler = hdbscan.HDBSCAN(min_cluster_size=min_cluster_size, min_samples=1, metric="euclidean")
    assignments = labeler.fit_predict(array)
    noise_count = int((assignments == -1).sum())
    by_label = {}
    for i, label in enumerate(assignments):
        if label == -1:
            continue
        by_label.setdefault(int(label), []).append(i)
    clusters = []
    for members in by_label.values():
        rep = representative_index(members, vectors)
        source_manuals = sorted(set(meta[i]["manual_id"] for i in members))
        clusters.append({
            "label": make_label(meta[rep]),
            "size": len(members),
            "span": len(source_manuals),
            "source_manuals": source_manuals,
            "members": [
                {"manual_id": meta[i]["manual_id"], "page_url": meta[i]["page_url"], "heading": meta[i]["heading"]}
                for i in members
            ],
        })
    clusters.sort(key=lambda c: (c["span"], c["size"]), reverse=True)
    for position, cluster in enumerate(clusters):
        cluster["id"] = position
    return clusters, noise_count
 
 
def write_clusters(clusters, path=CLUSTERS_PATH):
    with open(path, "w", encoding="utf-8") as handle:
        json.dump({"clusters": clusters}, handle, ensure_ascii=False, indent=1)
 
 
def load_clusters(path=CLUSTERS_PATH):
    try:
        with open(path, encoding="utf-8") as handle:
            return json.load(handle).get("clusters", [])
    except (OSError, ValueError):
        return []
  • Step 4: Run the test to verify it passes
/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/kb-vectors && venv/bin/python -m unittest test_cluster -v'

Expected: OK — 5 tests pass.

  • Step 5: Write kbclust.py

Create /home/levander/kb-vectors/kbclust.py:

import argparse
 
from cluster import CLUSTERS_PATH, build_clusters, load_clusters, write_clusters
from index import get_client
from sectionvecs import load_sections
 
 
def main():
    parser = argparse.ArgumentParser()
    sub = parser.add_subparsers(dest="command", required=True)
    build_parser = sub.add_parser("build")
    build_parser.add_argument("--min-cluster-size", type=int, default=2)
    sub.add_parser("list")
    show_parser = sub.add_parser("show")
    show_parser.add_argument("id", type=int)
    args = parser.parse_args()
    if args.command == "build":
        vectors, meta = load_sections(get_client())
        clusters, noise = build_clusters(vectors, meta, min_cluster_size=args.min_cluster_size)
        write_clusters(clusters)
        cross = sum(1 for c in clusters if c["span"] >= 2)
        print("sections:", len(vectors), "clusters:", len(clusters), "cross-manual:", cross, "noise:", noise)
        print("written:", CLUSTERS_PATH)
    elif args.command == "list":
        for c in load_clusters():
            print("#%d span=%d size=%d  %s" % (c["id"], c["span"], c["size"], c["label"]))
    elif args.command == "show":
        clusters = load_clusters()
        match = [c for c in clusters if c["id"] == args.id]
        if not match:
            print("no cluster", args.id)
            return
        c = match[0]
        print("#%d  %s  (span %d)" % (c["id"], c["label"], c["span"]))
        for m in c["members"]:
            print("  %-34s %s | %s" % (m["manual_id"], m["page_url"], m["heading"][:40]))
 
 
if __name__ == "__main__":
    main()
  • Step 6: Build clusters on the real corpus + quality proof
/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/kb-vectors && OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 nice -n 10 venv/bin/python kbclust.py build && echo "=== top cross-manual clusters ===" && venv/bin/python kbclust.py list | head -20'

Expected: a few hundred sections, a non-trivial cluster count with several cross-manual (span ≥ 2), a small noise count; the top of list shows high-span clusters (the dedup targets).

  • Step 7: Inspect a known-overlap cluster
/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/kb-vectors && for id in $(venv/bin/python kbclust.py list | grep -iE "diff|axle|brake|oxygen|timing" | head -3 | grep -oE "^#[0-9]+" | tr -d "#"); do echo "--- cluster $id ---"; venv/bin/python kbclust.py show $id; done'

Expected: at least one cluster whose members span multiple manuals for the same topic (e.g. a differential/axle cluster with workshop + geo-tracker + kick-fix/axles). This is the real proof the clustering is meaningful.

Task 4: /semantics + /api/clusters routes in the knowledgebase app

Files:

  • Modify: /home/levander/knowledgebase/app.py
  • Test: /home/levander/knowledgebase/test_semantics.py

Interfaces:

  • Consumes: clusters.json at /home/levander/kb-vectors/clusters.json

  • Produces: GET /api/clusters (JSON), GET /semantics (HTML)

  • Step 1: Write the failing test

Create /home/levander/knowledgebase/test_semantics.py:

import json
import os
import unittest
 
import app as kbapp
 
 
class TestSemantics(unittest.TestCase):
    def setUp(self):
        self.client = kbapp.app.test_client()
        self.sample = {"clusters": [
            {"id": 0, "label": "Differential", "size": 3, "span": 2,
             "source_manuals": ["chevy/geo", "vitara/workshop"],
             "members": [{"manual_id": "vitara/workshop", "page_url": "vitara/workshop/001-diff/", "heading": "Differential"}]}
        ]}
        os.makedirs(os.path.dirname(kbapp.CLUSTERS_PATH), exist_ok=True)
        self._backup = None
        if os.path.exists(kbapp.CLUSTERS_PATH):
            with open(kbapp.CLUSTERS_PATH) as handle:
                self._backup = handle.read()
        with open(kbapp.CLUSTERS_PATH, "w") as handle:
            json.dump(self.sample, handle)
 
    def tearDown(self):
        if self._backup is not None:
            with open(kbapp.CLUSTERS_PATH, "w") as handle:
                handle.write(self._backup)
 
    def test_api_returns_clusters(self):
        response = self.client.get("/api/clusters")
        self.assertEqual(response.status_code, 200)
        self.assertEqual(response.get_json()["clusters"][0]["label"], "Differential")
 
    def test_semantics_page_renders(self):
        response = self.client.get("/semantics")
        self.assertEqual(response.status_code, 200)
        self.assertIn(b"Differential", response.data)
        self.assertIn(b"vitara/workshop/001-diff/", response.data)
 
    def test_semantics_ok_when_missing(self):
        os.remove(kbapp.CLUSTERS_PATH)
        response = self.client.get("/semantics")
        self.assertEqual(response.status_code, 200)
 
 
if __name__ == "__main__":
    unittest.main()
  • Step 2: Run the test to verify it fails
/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/knowledgebase && venv/bin/python -m unittest test_semantics -v'

Expected: FAIL — AttributeError: module 'app' has no attribute 'CLUSTERS_PATH' (or route 404).

  • Step 3: Add the routes to app.py

In /home/levander/knowledgebase/app.py, add near the top (after the existing imports and app = Flask(__name__) config block):

import json as _json
 
CLUSTERS_PATH = "/home/levander/kb-vectors/clusters.json"
 
SEMANTICS_HTML = """<!doctype html><html><head><meta charset=utf-8>
<meta name=viewport content="width=device-width,initial-scale=1"><title>Szemantika</title>
<style>body{font-family:system-ui;max-width:60em;margin:1.5em auto;padding:0 1em;background:#0f1115;color:#e6e8ee}
a{color:#4f8cff}.c{border:1px solid #2a2f3a;border-radius:12px;padding:.8em 1em;margin:.7em 0;background:#1a1d24}
.h{font-weight:600;font-size:1.05em}.s{color:#9aa3b2;font-size:.85em;margin:.2em 0 .5em}
.m{font-size:.9em;margin:.15em 0}.b{display:inline-block;background:#2f6bff;color:#fff;border-radius:8px;padding:0 .5em;font-size:.8em;margin-left:.4em}</style>
</head><body><h1>Szemantika — kapcsolódó szakaszok</h1>
<p class=s>{{summary}}</p>{% for c in clusters %}<div class=c>
<div class=h>{{c.label}}<span class=b>{{c.span}} forrás</span></div>
<div class=s>{{c.source_manuals|join(' · ')}}</div>
{% for m in c.members %}<div class=m><a href="/{{m.page_url}}">{{m.manual_id}}</a> — {{m.heading}}</div>{% endfor %}
</div>{% endfor %}{% if not clusters %}<p>Nincs klaszter. Futtasd: <code>kbclust.py build</code></p>{% endif %}
<p><a href="/">← Vissza a tudásbázishoz</a></p></body></html>"""
 
 
def _load_clusters():
    try:
        with open(CLUSTERS_PATH, encoding="utf-8") as handle:
            return _json.load(handle).get("clusters", [])
    except (OSError, ValueError):
        return []

Then add the two routes (place them BEFORE the catch-all /<path:path> site route):

@app.route("/api/clusters")
def api_clusters():
    return jsonify({"clusters": _load_clusters()})
 
 
@app.route("/semantics")
def semantics():
    clusters = _load_clusters()
    cross = sum(1 for c in clusters if c.get("span", 0) >= 2)
    summary = "%d klaszter, ebből %d több kézikönyvet érint" % (len(clusters), cross)
    return render_template_string(SEMANTICS_HTML, clusters=clusters, summary=summary)

(jsonify and render_template_string are already imported in app.py — verify; if not, add them to the existing flask import line.)

  • Step 4: Run the test to verify it passes
/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/knowledgebase && venv/bin/python -m unittest test_semantics -v'

Expected: OK — 3 tests pass.

  • Step 5: Full knowledgebase suite still green (no route shadowing/regression)
/usr/bin/ssh levander@telep-mainframe 'cd /home/levander/knowledgebase && venv/bin/python -m unittest discover -p "test_*.py" 2>&1 | tail -3'

Expected: OK — all prior app tests still pass (26+) plus the 3 new ones.

  • Step 6: Restart the service + verify live over the tailnet
/usr/bin/ssh levander@telep-mainframe 'sudo systemctl restart knowledgebase && sleep 4 && systemctl is-active knowledgebase'
curl -s -o /dev/null -w "semantics=%{http_code}\n" --max-time 15 "https://knowledgebase.taild4189d.ts.net/semantics"
curl -s --max-time 15 "https://knowledgebase.taild4189d.ts.net/api/clusters" | head -c 200; echo
curl -s --max-time 15 "https://knowledgebase.taild4189d.ts.net/semantics" | grep -oE "forrás|klaszter" | head -1

Expected: semantics=200, the api returns the clusters JSON, and the page contains the Hungarian labels. Confirm a member link (e.g. copy a page_url from the api) resolves 200.

Task 5: Dashboard “Szemantika” tile

Files:

  • Modify: /home/levander/home-portal/index.html

  • Step 1: Add the tile

Add this tile into the <main> grid of /home/levander/home-portal/index.html, after the Tudásbázis tile:

    <a class="tile" href="https://knowledgebase.taild4189d.ts.net/semantics">
      <span class="icon">🧭</span>
      <span class="meta"><span class="name">Szemantika</span><span class="desc">Kapcsolódó szakaszok kézikönyvek között</span></span>
    </a>

Apply it:

/usr/bin/ssh levander@telep-mainframe 'python3 - <<PY
path="/home/levander/home-portal/index.html"
html=open(path,encoding="utf-8").read()
tile=(
"    <a class=\"tile\" href=\"https://knowledgebase.taild4189d.ts.net/semantics\">\n"
"      <span class=\"icon\">🧭</span>\n"
"      <span class=\"meta\"><span class=\"name\">Szemantika</span><span class=\"desc\">Kapcsolódó szakaszok kézikönyvek között</span></span>\n"
"    </a>\n"
)
anchor="https://knowledgebase.taild4189d.ts.net\">"
if "Szemantika" not in html:
    idx=html.index("</a>", html.index("Tudásbázis"))+len("</a>\n")
    html=html[:idx]+tile+html[idx:]
    open(path,"w",encoding="utf-8").write(html)
    print("tile added")
else:
    print("tile already present")
PY'
  • Step 2: Verify the tile is live
curl -s --max-time 15 "https://home.taild4189d.ts.net/" | grep -oE "Szemantika" | head -1

Expected: Szemantika (the static server serves the updated file directly; no restart needed).

  • Step 3: End-to-end click-path check
echo "dashboard -> semantics -> a KB section:"
curl -s -o /dev/null -w "dashboard=%{http_code}\n" --max-time 15 "https://home.taild4189d.ts.net/"
curl -s -o /dev/null -w "semantics=%{http_code}\n" --max-time 15 "https://knowledgebase.taild4189d.ts.net/semantics"
MEMBER=$(curl -s --max-time 15 "https://knowledgebase.taild4189d.ts.net/api/clusters" | python3 -c "import json,sys; c=json.load(sys.stdin)['clusters']; print(c[0]['members'][0]['page_url'] if c else '')" 2>/dev/null)
echo "first member: $MEMBER"
[ -n "$MEMBER" ] && curl -s -o /dev/null -w "member-page=%{http_code}\n" --max-time 15 "https://knowledgebase.taild4189d.ts.net/$MEMBER"

Expected: all 200 — the dashboard tile opens /semantics, and a cluster member links to a real KB section page.

Self-review notes

Spec coverage: section unit via (manual_id,page) grouping + mean-pool (Task 1); HDBSCAN min_cluster_size=2 with noise bucket (Task 3); local heading label + representative pick, pluggable (Task 2); cross-manual span ranking + clusters.json (Task 3); /semantics + /api/clusters on the existing app, no new service (Task 4); dashboard tile (Task 5); read-only, thread-capped, re-runnable (Global Constraints, verified in Task 3 build); quality proof that known-overlap topics cluster cross-manual (Task 3 Step 7).

Deliberate deviations: git commit steps omitted (user’s rule). Unit tests are on the pure logic (mean_pool, group_sections, representative_index, make_label, build_clusters on synthetic vectors); the real HDBSCAN-on-corpus quality is proven by the Task 3 build + inspect steps (not mockable meaningfully). CLI list/show are exercised in Task 3 Steps 6-7 rather than separate unit tests (thin argparse wrappers over tested functions).

Graph View

Backlinks