Stable LM Reviews & Product Details

Stable LM 2 12B is a 12.1 billion parameter decoder-only language model developed by Stability AI. Pre-trained on 2 trillion tokens from diverse multilingual and code datasets over two epochs, it is designed to generate coherent and contextually relevant text across various applications. The model employs a transformer decoder architecture with 40 layers, a hidden size of 5120, and 32 attention heads, supporting a sequence length of up to 4096 tokens. Key features include the use of Rotary Position Embeddings for improved throughput, parallel attention and feed-forward residual layers with a single input LayerNorm, and the removal of bias terms from feed-forward networks and grouped-query self-attention layers. Additionally, it utilizes the Arcade100k tokenizer, a BPE tokenizer extended from OpenAI's tiktoken.cl100k_base, with digits split into individual tokens to enhance numerical understanding. The primary value of Stable LM 2 12B lies in its ability to generate high-quality, contextually appropriate text, making it suitable for a wide range of natural language processing tasks, including content creation, code generation, and multilingual applications.