A quarter century of the market: six sectors, one space¶
Setup: install HyperTools (run this cell first on Colab)
[ ]:
# HyperTools setup: use 1.1 or newer; retain a current local checkout.
import importlib.util
from importlib.metadata import version, PackageNotFoundError
from packaging.version import Version
from pathlib import Path
try:
_hypertools_version = Version(version('hypertools'))
except PackageNotFoundError:
_hypertools_version = Version('0')
if _hypertools_version < Version('1.1.0'):
_spec = importlib.util.find_spec('hypertools')
if _spec and _spec.origin and (Path(_spec.origin).resolve().parents[1] / '.git').exists():
raise RuntimeError('Select a HyperTools 1.1 checkout/kernel before running this tutorial; the installer will not replace your checkout.')
%pip install -q "hypertools[interactive]>=1.1.0"
else:
print('Keeping HyperTools', _hypertools_version, 'in this kernel. Optional extras are loaded when requested.')
Twenty-seven large-cap stocks in six sectors, every month since 2000, drawn as seven paths through one shared 3-D space. Each sector is handed to the library as its own matrix – months down the rows, that sector’s stocks across the columns (four or five of them; the counts differ on purpose) – and each cell is the stock’s cumulative log return since the first month (a growth curve). Three library calls turn that into the figure:
hyp.reducetakes every sector from its own handful of stocks to three dimensions separately, so a sector is a trajectory in a space made of its own stocks, not a projection shared with the others;hyp.align(..., model='HyperAlign')hyperaligns the six trajectories into one common space, so that from then on a direction means the same thing for Energy as it does for Technology;hyp.plotdraws all six, plus a seventh, heavier path: the market, the point-by-point mean of the six aligned sector paths.
Color is the market’s composition. Every path is colored through the library’s mixture hue: hue= carries one row of weights per month and palette= the six sector colors, and each point is drawn in the weighted blend (hue_mode='mixture'). A sector’s weights are a one-hot row, so it keeps its own color; the market path’s weights are each sector’s share of the basket’s market capitalization that month (reported share counts x price), so its color shifts toward whichever
sectors dominate – away from technology red as the dot-com bubble deflates (about a third of the basket in mid-2000, a fifth by 2004), toward financial gold before 2008 (the largest sector in 2006), then back to technology, about half the basket through the 2020s. The title is the current date, tinted by the basket’s own trailing twelve-month return: red when the market is below where it stood a year earlier, green when it is above. The camera makes three turns over one minute. Only the
last six seconds of path are drawn at full strength, but nothing disappears: older path stays on as a faint trail, so the last frame is the whole quarter century.
Zero padding, verified. hyp.align zero-pads datasets with different numbers of columns to a common width automatically (its trim_and_pad step), so hyperaligning sectors of unequal size is fine. hyp.reduce on a list of unequal-width datasets deliberately refuses (it stacks them into one shared fit), which is exactly why the reduction here is per sector – the sectors do not share columns, and should not share a projection.
Data & graceful degradation. Daily closes and split events come from Yahoo Finance’s chart endpoint (full history, month-end decimated so no future observation reaches back into a bar). Both of its closes are split-adjusted – adjclose also reinvests dividends – so the share counts, which the SEC’s XBRL API (the per-concept endpoint, falling back to company facts; quarterly, from 2009) reports as they stood on the day, are multiplied by every later split before they meet the price.
Earlier months back-fill the first reported capitalization along the adjusted price. XOM’s counts are read from Exxon Mobil Corp’s filings (CIK 34088), not the holding company the SEC’s ticker map now names. HON’s 2026-06-30 count in the SEC data (316,940,010, from a 10-Q/A) is half its 2026-03-31 count (633,653,119). Everything is cached on disk. If the network is unavailable the example falls back to a seeded synthetic basket with the same sector structure and share counts, so it always
renders, and the technique is identical either way.
1. Imports, a disk cache, and the universe¶
Six sectors of four or five tickers each – unequal on purpose – with fixed sector colors, the animation length, a whitened PCA so a sector’s common growth does not flatten its path onto one line, and the colormap that tints the date red or green.
[1]:
import json
import os
import tempfile
import urllib.request
from typing import NamedTuple
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from matplotlib.colors import LinearSegmentedColormap, Normalize
from matplotlib.lines import Line2D
import hypertools as hyp
CACHE = os.path.join(tempfile.gettempdir(), 'hypertools_gallery_cache')
START = '2000-01-01' # first month drawn
PERIOD1, PERIOD2 = 915148800, 4102444800 # 1999-01-01 .. 2100-01-01, unix
WINDOW = 12 # months in a trailing return
# six sectors, four or five tickers each -- unequal on purpose (see docstring)
SECTORS = {
'Technology': ['AAPL', 'MSFT', 'ORCL', 'IBM', 'INTC'],
'Financials': ['JPM', 'BAC', 'GS', 'AXP'],
'Healthcare': ['JNJ', 'PFE', 'MRK', 'ABT', 'AMGN'],
'Energy': ['XOM', 'CVX', 'COP', 'SLB'],
'Consumer': ['KO', 'PG', 'WMT', 'MCD', 'PEP'],
'Industrials': ['BA', 'CAT', 'GE', 'HON'],
}
SECTOR_COLORS = ['#c1272d', '#f2a900', '#1b7f4f', '#2d5fa8', '#7d3c98',
'#00808a']
DURATION, FPS, ROTATIONS = 60, 20, 3 # seconds, frames per second, turns
TAIL, TITLE_SIZE = 6, 17 # bright window (s), title points
# whitened PCA: without it the common growth of a sector's stocks is nearly
# all of the variance, and its 3-D path collapses onto one line
REDUCE = {'model': 'PCA', 'kwargs': {'whiten': True}}
# the SEC's fair-access policy wants a declared agent with a contact; this is
# the project's public one (pyproject.toml)
USER_AGENT = 'hypertools-gallery/1.1 (Contextual Dynamics Lab, contextualdynamics@gmail.com)'
SEC = 'https://data.sec.gov/api/xbrl/companyconcept/CIK{cik:010d}/dei/EntityCommonStockSharesOutstanding.json'
SEC_FACTS = 'https://data.sec.gov/api/xbrl/companyfacts/CIK{cik:010d}.json'
# the title's tint: the basket's trailing-12-month return, red below zero,
# green above, saturating at +/-30 % (a range the 2008 and 2020 drawdowns
# both leave)
TITLE_CMAP = LinearSegmentedColormap.from_list(
'market', ['#b3261e', '#555555', '#1b7f4f'])
TITLE_NORM = Normalize(-0.3, 0.3, clip=True)
3. Growth curves per sector, market-cap weights, the basket’s return¶
What is plotted is each stock’s cumulative log return since the first month (a growth curve), one matrix per sector. Each sector’s share of the basket’s capitalization, month by month, becomes the color weights of the market path, and the cap-weighted trailing twelve-month return tints the title. load_market is the real path; fixture_data is the same payload from the seeded basket.
[3]:
def assemble(adjusted, close, shares, sectors, source):
"""Month-end trailing returns per sector, market-cap weights per sector
and the basket's own return, on one shared monthly index from START."""
# month-end levels; a month still in progress is DROPPED (resample
# stamps the last close so far at the month's END, which dated a clip
# rendered on 4 September "September 30" -- measured 2026-09-04)
levels = np.log(adjusted.resample('ME').last()).loc[:adjusted.index[-1]]
returns = (levels - levels.shift(WINDOW)).loc[START:].dropna()
months = returns.index
# what is PLOTTED: each stock's cumulative log return since the first
# month, so a sector's matrix is a set of growth curves and its 3-D path
# is a journey rather than a tangle of month-to-month noise
paths = levels.loc[months] - levels.loc[months[0]]
# market cap = split-adjusted close x split-adjusted shares; before the
# first filing, the first known cap is carried back along the ADJUSTED
# (dividend-reinvested) price
cap = close.resample('ME').last().reindex(months) * shares.reindex(months).ffill()
first = cap.apply(lambda col: col.first_valid_index())
for ticker in cap:
known = cap.loc[first[ticker], ticker]
adj = np.exp(levels.reindex(months)[ticker])
cap.loc[:first[ticker], ticker] = known * adj / adj[first[ticker]]
caps = pd.DataFrame({s: cap[t].sum(axis=1) for s, t in sectors.items()})
weights = caps.div(caps.sum(axis=1), axis=0)
market = (returns * cap.div(cap.sum(axis=1), axis=0)).sum(axis=1)
return Market({s: paths[t] for s, t in sectors.items()}, weights,
market, source)
def load_market(sectors=SECTORS):
"""The ONLY function here that may touch the network."""
prices = shares = None
try:
prices = fetch_prices(sectors)
if prices is not None:
shares = fetch_shares([t for ts in sectors.values() for t in ts], prices[2])
except RuntimeError:
pass
if prices is None or shares is None:
return assemble(*synthetic_market(sectors), sectors,
'synthetic basket (offline)')
return assemble(*prices[:2], shares, sectors,
'Yahoo Finance closes, SEC share counts')
def fixture_data():
"""The same payload from the seeded synthetic basket. No network."""
return assemble(*synthetic_market(), SECTORS, 'synthetic basket (fixture)')
4. Reduce per sector, hyperalign, draw seven paths¶
Three library calls: hyp.reduce per sector (its own stocks, its own space), hyp.align(..., model='HyperAlign') into one shared space, and hyp.plot on the six aligned paths plus their mean, colored through the mixture hue. The on_frame hook only sets the title: the date under the head, tinted by the basket’s return.
[4]:
# --- the figure half: no network, deterministic given its input -------------
def construct_artifact(data):
"""`data` in, the animation out. No network, no module globals.
Returns the HyperAnimation wrapper, never the unpacked pair."""
names = list(data.sectors)
n_months = len(data.weights)
# 1. each sector to 3-D in a space of ITS OWN stocks; 2. hyperalign the
# six trajectories into one shared space; 3. the market is their mean
reduced = [hyp.reduce(data.sectors[s], reduce=REDUCE, ndims=3) for s in names]
aligned = hyp.align(reduced, model='HyperAlign')
market = np.mean(aligned, axis=0)
# mixture hue: one row of weights per month, blended through the six
# sector colors. One-hot rows keep a sector its own color; the market
# path's rows are the sectors' shares of the basket's capitalization.
hue = [np.tile(np.eye(len(names))[i], (n_months, 1))
for i in range(len(names))] + [data.weights.to_numpy()]
# THE call: seven paths, one minute, three turns of the camera; a
# six-second bright head, and a chemtrail that keeps everything older.
months, basket = data.weights.index, data.market.to_numpy()
# the title is restyled per frame below; the library reserves its margin
# at build time from rcParams, so the size is declared here as well.
# backend='matplotlib': the hook and legend below use the matplotlib
# figure, and on Colab the default backend would be plotly
with plt.rc_context({'axes.titlesize': TITLE_SIZE, 'axes.titleweight': 'bold'}):
anim = hyp.plot(aligned + [market], '-', hue=hue, palette=SECTOR_COLORS,
hue_mode='mixture', linewidth=[1.1] * len(names) + [3.4],
manip='Smooth', animate=True, chemtrails=True,
tail_duration=TAIL, duration=DURATION, frame_rate=FPS,
rotations=ROTATIONS, colorbar=False,
title=f'{months[0]:%B} {months[0].day}, {months[0].year}',
size=(8, 8), backend='matplotlib', show=False)
ax = anim.figure.axes[0]
ax.legend(handles=[Line2D([], [], color=c, lw=2, label=s)
for s, c in zip(names, SECTOR_COLORS)]
+ [Line2D([], [], color='#666666', lw=3.4,
label='Market (average across sectors)')],
loc='upper left', fontsize=8, frameon=False)
def date_title(ctx):
"""The title is the date under the head, tinted by the basket's own
trailing return. Assigned on EVERY frame (the portable hook rule)."""
frac = ctx.frame / max(1, ctx.n_frames - 1)
when = months[0] + frac * (months[-1] - months[0])
ret = np.interp(frac, np.linspace(0, 1, n_months), basket)
ax.set_title(f'{when:%B} {when.day}, {when.year}', fontsize=TITLE_SIZE,
fontweight='bold', color=TITLE_CMAP(TITLE_NORM(ret)))
anim.on_frame(date_title)
return anim
5. Load the data and build the animation¶
[5]:
market = load_market()
print(f'market data: {len(market.weights)} months x '
f'{sum(f.shape[1] for f in market.sectors.values())} stocks in '
f'{len(market.sectors)} sectors, {market.weights.index[0]:%b %Y} - '
f'{market.weights.index[-1]:%b %Y} ({market.source})')
anim = construct_artifact(market)
fig = anim.figure
_ = anim.draw_frame(anim.n_frames - 1) # the fully revealed frame
market data: 317 months x 27 stocks in 6 sectors, May 2000 - Sep 2026 (Yahoo Finance closes, SEC share counts)
6. Save the animation¶
[6]:
anim.save('market_sectors.mp4', dpi=100)
print('saved market_sectors.mp4')
# Colab serves output frames separately from kernel files; embed movie bytes.
try:
from google import colab as colab
except ImportError:
pass # Local Jupyter/Sphinx uses the relative video below.
else:
from IPython.display import Video, display
display(Video('market_sectors.mp4', embed=True))
saved market_sectors.mp4