Skip to main content

Transformer Encoder Playground

Transformer Encoder Playground

timeflieslikeanarrow

Token Embeddings + Sinusoidal Positional Encoding

time
••••••••••••••••••••••••••••••••
flies
••••••••••••••••••••••••••••••••
like
••••••••••••••••••••••••••••••••
an
••••••••••••••••••••••••••••••••
arrow
••••••••••••••••••••••••••••••••

Notes

  • Deterministic random init (fixed seed) for reproducibility.
  • Sinusoidal positional encoding added before attention.
  • Self-attention is unmasked (encoder-style).
  • Simplified post-norm LayerNorm for clarity.
  • Adjust heads, model dimension, and FFN size to see qualitative changes.

Scope note​

This is an educational AI Primitive / model-architecture learning tool, not an AI Product classifier. A Transformer model can participate in an AI Product constitution or be productized as a Model Product, but model architecture alone does not establish AIPS AI Product Family membership.

About this tool​

Originally created as a learning project while studying BERT/Transformer architectures; retained as an educational visualization.

This page illustrates the encoder sub-architecture of a Transformer:

  • Embeddings + Positional Encoding — converts tokens to vectors and injects order.
  • Multi-Head Self-Attention — mixes information across the sequence.
  • Residual Connections + LayerNorm — stabilize training and preserve signal.
  • Position-wise Feed-Forward — non-linear projection per token.
  • Output — encoded token representations for downstream tasks.

Roadmap ideas: CLS pooling, key padding masks, import/export weights (JSON), BPE/WordPiece tokenization, toy backprop views.