Transformer Encoder Playground
Transformer Encoder Playground
timeflieslikeanarrow
Token Embeddings + Sinusoidal Positional Encoding
time
••••••••••••••••••••••••••••••••
flies
••••••••••••••••••••••••••••••••
like
••••••••••••••••••••••••••••••••
an
••••••••••••••••••••••••••••••••
arrow
••••••••••••••••••••••••••••••••
Notes
- Deterministic random init (fixed seed) for reproducibility.
- Sinusoidal positional encoding added before attention.
- Self-attention is unmasked (encoder-style).
- Simplified post-norm LayerNorm for clarity.
- Adjust heads, model dimension, and FFN size to see qualitative changes.
Scope note
This is an educational AI Primitive / model-architecture learning tool, not an AI Product classifier. A Transformer model can participate in an AI Product constitution or be productized as a Model Product, but model architecture alone does not establish AIPS AI Product Family membership.
About this tool
Originally created as a learning project while studying BERT/Transformer architectures; retained as an educational visualization.
This page illustrates the encoder sub-architecture of a Transformer:
- Embeddings + Positional Encoding — converts tokens to vectors and injects order.
- Multi-Head Self-Attention — mixes information across the sequence.
- Residual Connections + LayerNorm — stabilize training and preserve signal.
- Position-wise Feed-Forward — non-linear projection per token.
- Output — encoded token representations for downstream tasks.
Roadmap ideas: CLS pooling, key padding masks, import/export weights (JSON), BPE/WordPiece tokenization, toy backprop views.