The shape of a conversation: one path per turn, colored by its speaker¶
Setup: install HyperTools (run this cell first on Colab)
[ ]:
# HyperTools setup: use 1.1 or newer; retain a current local checkout.
import importlib.util
from importlib.metadata import version, PackageNotFoundError
from packaging.version import Version
from pathlib import Path
try:
_hypertools_version = Version(version('hypertools'))
except PackageNotFoundError:
_hypertools_version = Version('0')
if _hypertools_version < Version('1.1.0'):
_spec = importlib.util.find_spec('hypertools')
if _spec and _spec.origin and (Path(_spec.origin).resolve().parents[1] / '.git').exists():
raise RuntimeError('Select a HyperTools 1.1 checkout/kernel before running this tutorial; the installer will not replace your checkout.')
%pip install -q "hypertools[interactive]>=1.1.0"
else:
print('Keeping HyperTools', _hypertools_version, 'in this kernel. Optional extras are loaded when requested.')
A conversation as geometry, in one hyp.plot call on raw dialogue. Each turn (a contiguous run of speech by one speaker) is cut into sliding word windows, and the turns are handed to hyp.plot as a list of lists of strings: the nesting is the grouping, so every turn becomes its own disjoint trajectory through one shared 3-D space. The call embeds every window with a sentence-transformer (vectorizer='all-MiniLM-L6-v2'), reduces them together with UMAP, colors each path by
speaker through a categorical hue= with a native legend, and order='serial' reveals the turns one at a time with chemtrails=True leaving the spoken path behind each head. title= carries one string per turn – just the words being spoken, wrapped onto two lines when a turn is long so nothing runs off the figure – and the library’s own reveal schedule advances it; the title never names the speaker, because the color does: title_color= takes one color per turn and the
library retints the title with the current speaker’s color every frame, while title_kwargs={'size': ...} sets its size (and the library now reserves the animated 3-D title’s own vertical margin, sized to the tallest wrapped title, so the figure never has to grow by hand); the legend maps colors to names.
The one effect written out by hand is a recency fade across turns: the current turn is opaque, earlier turns recede slowly (a turn keeps most of its opacity for several exchanges before settling at a visible floor, so the conversation’s recent past stays legible), and unspoken turns are hidden. 1.1 ships the same fade as one keyword – dataset_fade={'floor': FLOOR, 'decay': DECAY} gives every head and trail exactly these alphas – and this example keeps it as a hook to show what the
public on_frame hook can do: it reads the schedule the library publishes (ctx.current_index, ctx.revealed_counts) and assigns every artist on every frame. Before 1.1 this example monkeypatched ani._func and re-derived that schedule by hand; the hook replaces both. The clip runs 30 seconds with two full camera rotations.
Here the conversation is Lewis Carroll’s Mad Tea-Party (Alice in Wonderland). The turns are bundled inline – quoted verbatim from the Project Gutenberg text – so the example is fully offline and deterministic.
What gets embedded is SPOKEN TEXT ONLY. Every narrative attribution (“said the Hatter”, “Alice replied”) and all surrounding narration has been stripped, so the geometry reflects what the characters say, not how the narrator introduces them – otherwise every one of Alice’s turns would be pulled together by the repeated words “said Alice” rather than by their content. Who is speaking is carried by the color (paths and title alike) and the legend.
Embeddings. Sentence embeddings come from the [text] extra, which hypertools installs on demand the first time an embedding model is requested; nothing here checks for it. The test-suite drives the same pipeline (embed -> reduce -> disjoint per-turn paths -> serial reveal colored by speaker) through fixture_data, which embeds with vectorizer='TfidfVectorizer' so no test downloads a model.
1. Imports, the speakers, and the turns¶
The Mad Tea-Party turns, spoken text only, with each speaker’s color, the recency-fade constants, the title wrap width and the tail length.
[1]:
import textwrap
from typing import NamedTuple
import hypertools as hyp
VECTORIZER = 'all-MiniLM-L6-v2'
SPEAKER_COLOR = {
'Alice': '#E4572E', 'Hatter': '#3F72AF', 'March Hare': '#5B8C5A',
'Dormouse': '#B5537F',
}
# The turns are CURATED, SPOKEN TEXT ONLY: each entry is exactly what that
# character says out loud, quoted verbatim from the Gutenberg text (emphasis
# underscores dropped), with every narrative attribution ("said the Hatter",
# "Alice replied") and all surrounding narration removed. Automatic extraction
# was tried and rejected: it mis-merged speakers across adjacent quotes (e.g.
# attributing "There's plenty of room!" to the March Hare) and dragged
# narration into the embedded text. What is embedded IS the dialogue.
TURNS = [
('Alice', "There's plenty of room!"),
('March Hare', "Have some wine."),
('Alice', "I don't see any wine."),
('March Hare', "There isn't any."),
('Alice', "Then it wasn't very civil of you to offer it."),
('March Hare', "It wasn't very civil of you to sit down without being invited."),
('Alice', "I didn't know it was your table; it's laid for a great many more than three."),
('Hatter', "Your hair wants cutting."),
('Alice', "You should learn not to make personal remarks; it's very rude."),
('Hatter', "Why is a raven like a writing-desk?"),
('Alice', "I'm glad they've begun asking riddles. I believe I can guess that."),
('March Hare', "Do you mean that you think you can find out the answer to it?"),
('Alice', "Exactly so."),
('March Hare', "Then you should say what you mean."),
('Alice', "I do; at least I mean what I say - that's the same thing, you know."),
('Hatter', "Not the same thing a bit! You might just as well say that 'I see what I eat' is the same thing as 'I eat what I see'!"),
('March Hare', "You might just as well say that 'I like what I get' is the same thing as 'I get what I like'!"),
('Dormouse', "You might just as well say that 'I breathe when I sleep' is the same thing as 'I sleep when I breathe'!"),
('Hatter', "It is the same thing with you. What day of the month is it?"),
('Alice', "The fourth."),
('Hatter', "Two days wrong! I told you butter wouldn't suit the works!"),
('March Hare', "It was the best butter."),
('Hatter', "Yes, but some crumbs must have got in as well; you shouldn't have put it in with the bread-knife."),
('Alice', "What a funny watch! It tells the day of the month, and doesn't tell what o'clock it is!"),
('Hatter', "Why should it? Does your watch tell you what year it is?"),
('Alice', "Of course not; but that's because it stays the same year for such a long time together."),
('Hatter', "Which is just the case with mine."),
('Alice', "I don't quite understand you."),
]
WINDOW, STEP, MIN_WINDOWS = 6, 2, 3
# Recency fade across turns (see turn_alpha): the turn before the current one
# is drawn at FLOOR + (1 - FLOOR) * DECAY, the one before that at DECAY^2 of
# the way from FLOOR to opaque, and so on. 0.45/0.10 (the first cut) dropped a
# turn to half opacity the moment the next speaker began and to the floor two
# exchanges later, so the tails looked like they vanished; 0.7/0.18 keeps the
# previous turn at ~0.75, the one before at ~0.58, and five turns back still
# at ~0.32, so the recent past of the conversation lingers on screen.
FLOOR, DECAY = 0.18, 0.7
# Title size and wrap width. At 14 pt a character is ~8.7 px wide at 100 dpi,
# so a 64-character line is ~560 px over the 800 px-wide axes -- centered with
# clear margin either side -- and the longest turn (117 characters, 119 with
# the quotes the title adds) wraps to exactly two lines; no turn needs a
# third (verified by rendering turns 15-17 and 22, the long ones).
TITLE_SIZE, TITLE_WIDTH = 14, 64
# How many seconds of the current turn's reveal the opaque comet-head spans
# (hyp.plot's tail_duration; default 2). Raised so the head covers more of
# the words just spoken before they hand over to the trail artist.
TAIL_SECONDS = 6
2. Windows and the payload¶
Each turn becomes a list of sliding word windows; the payload carries the windows, the speakers, the spoken lines (for the titles) and the vectorizer. fixture_data embeds with TF-IDF so no test downloads a model.
[2]:
class Conversation(NamedTuple):
turns: list # per turn, its list of word windows
speakers: list # per turn, who speaks it
texts: list # per turn, the spoken line (for the title)
vectorizer: str # how the windows are embedded
source: str # which path produced them
def windows(text, size=WINDOW, step=STEP, min_windows=MIN_WINDOWS):
"""Sliding word windows over one turn.
``min_windows`` prevents a real rendering artifact: ``hyp.plot`` draws a
ONE-ROW dataset as a dot (there is no line through a single point), and
with a fixed 6-word window stepping by 2, 12 of the 28 turns above get
fewer than three windows -- 9 of them none at all, 3 a single window
that would show up as a stray speck. Shrinking the window, and the step
if needed, keeps every turn a real path.
"""
words = text.split()
n = len(words)
size = max(1, min(size, n - min_windows + 1))
step = step if (n - size) // step + 1 >= min_windows else 1
return [' '.join(words[i:i + size]) for i in range(0, n - size + 1, step)]
# --- the data half. Nothing here fetches: the text is inline, and the only
# --- network access an embedding can need (the model) is named, not run.
def embed_turns(spec=TURNS, vectorizer=VECTORIZER):
"""Window each turn and name the vectorizer the figure should use."""
return Conversation([windows(text) for _speaker, text in spec],
[speaker for speaker, _text in spec],
[text for _speaker, text in spec],
vectorizer, 'inline Gutenberg text')
def fixture_data():
"""The same payload embedded with a deterministic sklearn TF-IDF fit:
no model download. What the test-suite drives."""
return embed_turns(TURNS, 'TfidfVectorizer')
3. The recency fade, on the on_frame hook¶
turn_alpha is the fade’s formula and recency_fade assigns it to every head and trail on the public on_frame hook. It is the hand-written form of dataset_fade={'floor': FLOOR, 'decay': DECAY}, which gives exactly these alphas in one keyword. The title needs no hook: title_color= tints each turn’s title with its speaker’s color, and the library reserves the wrapped title’s margin.
[3]:
# --- the figure half: no network, deterministic given its input -------------
def turn_alpha(i, revealed, current):
"""How visible turn `i` should be while turn `current` is being drawn.
Assigns a value for EVERY dataset on EVERY frame, including turns not
yet spoken -- the portable callback rule (animation.rst): put the
condition in the VALUE, never around the assignment. A skipped
assignment leaves matplotlib's shared artists at whatever the previous
frame set, which is how a fade turns into a smear.
"""
if i > current or revealed < 2:
return 0.0 # unspoken, or a single stray point
if i == current:
return 1.0
return FLOOR + (1.0 - FLOOR) * DECAY ** (current - i)
def recency_fade(ctx):
"""The one bespoke effect left: earlier turns recede as the talk moves on.
``chemtrails``/``precog``/``bullettime`` fade WITHIN one trajectory;
this fades ACROSS already-revealed datasets. It is the hand-written
form of ``dataset_fade={'floor': FLOOR, 'decay': DECAY}`` (same
formula, heads and trails alike), kept here to show the public
per-frame hook reading the library's own published schedule instead of
re-deriving it.
``ctx.artists`` is NOT one artist per dataset. It is heads first, then
trails (animation_context.FrameContext), so with ``chemtrails=True`` it
holds 2N entries against ``revealed_counts``' N. Zipping the two
directly walks off the end of the counts. Split by role first.
"""
current = ctx.current_index
if current is None:
raise RuntimeError(
"recency_fade needs a serial reveal: ctx.current_index is None, "
"which means this plot is animating in parallel. Keep "
"order='serial' (or animate='serial') on the plot() call.")
n_datasets = len(ctx.revealed_counts)
heads = ctx.artists[:n_datasets]
trails = ctx.artists[n_datasets:]
# chemtrails=True is broadcast to every dataset, so this holds here. It
# is asserted rather than assumed because a dataset drawn marker-only
# gets no trail artist, and the mismatch would otherwise show up as a
# silently mis-paired head/trail rather than an error.
if len(trails) != n_datasets:
raise RuntimeError(
f"expected one trail artist per dataset, got {len(trails)} "
f"trails for {n_datasets} datasets")
for i, (head, trail, revealed) in enumerate(
zip(heads, trails, ctx.revealed_counts)):
alpha = turn_alpha(i, revealed, current)
head.set_alpha(alpha)
# NOT the library's 0.3x trail convention. On a serial reveal the
# trail IS the part of the current turn already spoken (measured:
# 821 of its points against a 6-point head), so at 0.3x the turn
# being spoken was the faintest thing on screen. It fades with its
# head, one value per turn.
trail.set_alpha(alpha)
4. One call¶
Raw dialogue in, one disjoint trajectory per turn, colored by speaker, revealed one turn at a time over thirty seconds and two camera rotations.
[4]:
def construct_artifact(data):
"""`data.turns` / `data.speakers` in, the animation out. Returns the
HyperAnimation wrapper: `.on_frame()` lives on it, not on the
FuncAnimation that `fig, ani = ...` would give."""
# category order is FIRST APPEARANCE, not alphabetical, so the palette
# must be listed in that order for each speaker to get their color
order = list(dict.fromkeys(data.speakers))
# the title is JUST the spoken line (who says it is the title's color),
# wrapped so the longest turn fits the figure on two lines
titles = [textwrap.fill(f'\u201c{text}\u201d', TITLE_WIDTH)
for text in data.texts]
# THE hypertools call: raw dialogue in, one disjoint trajectory per
# turn, colored by speaker, revealed ONE TURN AT A TIME. backend= is
# pinned because the hook sets matplotlib alphas, and on Colab the
# default backend would be plotly.
anim = hyp.plot(
data.turns, '-',
vectorizer=data.vectorizer, semantic=None, corpus=None,
reduce={'model': 'UMAP', 'kwargs': {'n_neighbors': 8, 'min_dist': 0.5,
'random_state': 1}},
ndims=3,
hue=[[speaker] * len(turn)
for speaker, turn in zip(data.speakers, data.turns)],
palette=[SPEAKER_COLOR[s] for s in order], legend=True,
linewidth=1.6,
animate=True, order='serial', chemtrails=True,
tail_duration=TAIL_SECONDS, title=titles,
title_kwargs={'size': TITLE_SIZE},
title_color=[SPEAKER_COLOR[s] for s in data.speakers],
duration=30, rotations=2, frame_rate=16, elev=16, size=(8, 8),
backend='matplotlib', show=False)
anim.on_frame(recency_fade)
return anim
5. Load the data and build the animation¶
[5]:
conversation = embed_turns()
print(f'conversation: {len(conversation.turns)} turns, '
f'{len(set(conversation.speakers))} speakers, '
f'{sum(len(t) for t in conversation.turns)} windows, embedded with '
f'{conversation.vectorizer}')
anim = construct_artifact(conversation)
fig = anim.figure
_ = anim.draw_frame(anim.n_frames - 1) # the fully revealed frame
conversation: 28 turns, 4 speakers, 127 windows, embedded with all-MiniLM-L6-v2
6. Save the animation¶
[6]:
anim.save('conversation_shape.mp4', dpi=100)
print('saved conversation_shape.mp4')
# Colab serves output frames separately from kernel files; embed movie bytes.
try:
from google import colab as colab
except ImportError:
pass # Local Jupyter/Sphinx uses the relative video below.
else:
from IPython.display import Video, display
display(Video('conversation_shape.mp4', embed=True))
saved conversation_shape.mp4