M31Entropy
M31Entropy — experimental architecture exploration and pretraining checkpoints for large-vocabulary M31 variants (314M parameters).
- Experimental
- RESEARCH
- 220M-Experiment
- MIT
Early pretraining checkpoint (step-00000500). Experimental research artifact.
Experimental
Experimental research artifact
RESEARCH
Active research & evaluation
AI Model
314M
TIV · MIT
Tonmoy Infrastructure & Vision
About M31Entropy
M31Entropy — experimental architecture exploration and pretraining checkpoints for large-vocabulary M31 variants (314M parameters).
Why this system exists
Evaluating whether expanding tokenizer vocabulary to 128,000 tokens improves code representation density and reduces context fragmentation in multilingual codebases.
Engineering philosophy & approach
M31Entropy investigates large vocabulary sizes (128k) in a 314M parameter causal transformer architecture, examining pretraining convergence dynamics.
System architecture
314,281,856 parameters; vocab 128,000; hidden 896; 20 layers; 14 attention heads; 7 KV heads; intermediate 2,816; RMSNorm eps 1e-6; RoPE theta 10,000; tied word embeddings.
Core system capabilities
- Expanded 128,000 token vocabulary for code density
- 314M parameter causal language model architecture
- 14 query / 7 KV head grouped-query attention
- Step-00000500 pretrain safetensors checkpoint
Technical Model Specification
314,281,856 parameters; vocab 128,000; hidden 896; 20 layers; 14 attention heads; 7 KV heads; intermediate 2,816; tied embeddings; max position embeddings 2048.
Overview
M31Entropy is an experimental architecture exploration investigating large-vocabulary (128,000 tokens) tokenizer dynamics in compact causal transformers.
Fact Verification: Project Name & Scope
Note: In accordance with TIV's strict transparency rules, the project name M31Entropy does not imply an entropy-based or thermodynamics-based architecture. It is an experimental model identifier exploring representation density and pretraining dynamics.
Architecture
- —Parameter Count: 314,281,856 parameters
- —Vocabulary Size: 128,000 tokens
- —Hidden Size: 896
- —Layers: 20 layers
- —Attention Heads: 14 query heads, 7 KV heads
- —Intermediate Size: 2,816
- —Max Position Embeddings: 2,048 (1,024 training sequence length)
- —Tied Embeddings: Enabled
Public Source
The model card and checkpoint are hosted on Hugging Face at eshanized/M31Entropy.
Milestones & active developments
- Expanded tokenizer construction (128,000 vocab)
- Model configuration and parameter budgeting
- Initial pretraining run to step-00000500
- Hugging Face checkpoint publication
- Vocabulary compression ratio evaluation against standard BPE
Checkpoint lineage & milestones
Step-wise checkpoints and lineage updates for M31Entropy.
220M-Experiment — Verified Milestone
Experimental checkpoints verified against public repositories.