okflint.cohesion module

Semantic cohesion scoring for markdown sections (S202 calibration core).

Pure functions only: no I/O, no printing, no CLI. See scripts/calibrate_cohesion.py for the disposable scaffolding that drives this module on real files.

class okflint.cohesion.CohesionResult(sections, components)

Bases: object

Full output of a cohesion analysis for one document.

components: list[Component]

sections: list[Section]

class okflint.cohesion.Component(section_indices)

Bases: object

A connected component of sections linked by cosine similarity > tau.

section_indices: list[int]

class okflint.cohesion.Section(title, level, line, body, token_count)

Bases: object

A document section after splitting and micro-section merging.

body: str

level: int | None

line: int

title: str | None

token_count: int

okflint.cohesion.analyze_cohesion(content, *, tau=0.15, min_tokens=20, title_levels=None)

Run the full cohesion pipeline on one markdown file’s raw content.

Parameters:

• content (str) – Full raw file content, frontmatter included.

• tau (float) – Cosine similarity threshold above which two sections are linked.

• min_tokens (int) – Token floor for micro-section merging.

• title_levels (set[int] | None) – Heading levels treated as section boundaries (None = all).

Return type:

CohesionResult

Returns:

CohesionResult with merged sections and their connected components.

okflint.cohesion.build_similarity_matrix(vectors)

Build a symmetric cosine similarity matrix, quantized to 6 decimals.

Return type:

list[list[float]]

okflint.cohesion.compute_tfidf_vectors(bodies)

Compute a TF-IDF vector per section body (stdlib only, no stopwords).

Parameters:

bodies (list[str]) – Section body texts, in document order.

Return type:

list[dict[str, float]]

Returns:

List of term -> weight dicts, one per body, in the same order.

okflint.cohesion.find_components(matrix, tau)

Return connected components (as sorted index lists) linked by cos > tau.

Return type:

list[list[int]]

okflint.cohesion.merge_micro_sections(sections, min_tokens)

Fuse sections under the token floor into a neighbour until none remain.

A section below the floor is attached to the previous section, or to the next one if it is the first section. Nothing is discarded.

Parameters:

• sections (list[Section]) – Sections as produced by split_into_sections.

• min_tokens (int) – Minimum token count a section’s body must reach.

Return type:

list[Section]

Returns:

List of Section where every entry meets the floor (unless the whole document is a single section below it).

okflint.cohesion.split_into_sections(safe_body, title_levels=None)

Split a code-fence-masked body into sections at every heading line.

Parameters:

• safe_body (str) – Document body with frontmatter removed and code fences blanked.

• title_levels (set[int] | None) – Heading levels that count as section boundaries. None means all levels (H1-H6) are boundaries.

Return type:

list[Section]

Returns:

List of Section, including a leading preamble section (title=None).

okflint.cohesion.tokenize(text)

Lowercase, split on non-alphanumeric boundaries, drop single characters.

Return type:

list[str]