Model Card for MPT-7B

Basic Information

• Model Name: MPT-7B
• Developer/Creator: MosaicML
• Release Date: May, 2023
• Version: Initial release, with subsequent variant models including MPT-7B-Chat, MPT-7B-Instruct, and MPT-7B-StoryWriter-65k+
• Model Type: Decoder-style Transformer, part of the GPT-style large language model family

Description

Overview: MPT-7B represents MosaicML's leap into the open-source domain, aiming to democratize access to state-of-the-art transformer technology. It's designed for both generic and specific NLP tasks, with a particular emphasis on handling extremely long input sequences.

Key Features:

• Commercially Usable and Open Source: Licensed under Apache-2.0 for base and some variants, enabling wide accessibility and commercial application.
• Long Input Sequences: Utilizes ALiBi to manage unprecedented input lengths (up to 65k tokens), making it ideal for detailed text analysis and generation tasks.
• High Efficiency: Incorporates FlashAttention and FasterTransformer for accelerated training and inference, significantly reducing operational costs.
• Broad Accessibility: Integrated with HuggingFace for easy implementation, ensuring compatibility with existing machine learning workflows.

Intended Use:

The model is versatile, suitable for tasks ranging from machine learning research and application development to specific commercial uses in fields like tech and entertainment. Its variants are optimized for roles like conversational AI, narrative generation, and compliance with complex instructions.

Language Support:

Focused on English, incorporating a diverse array of text types, including technical and creative writing, to ensure robust language understanding.

Technical Details

Architecture:

Built as a decoder-only transformer with a configuration of 6.7 billion parameters, tailored for deep contextual understanding and generation.

Training Data:

The model's robustness stems from its training on 1 trillion tokens derived from a meticulously curated dataset combining text and code, ensuring a comprehensive linguistic and contextual grasp.

Data Source and Size:

Diverse sources including large-scale corpora like Books3, Common Crawl, and domain-specific datasets ensuring a rich mix of general and specialized content.

Knowledge Cutoff:

Includes the most recent and relevant data up to the year 2023, facilitating a contemporary understanding of language and context.

Diversity and Bias:

Carefully constructed to minimize bias by incorporating a wide range of text sources, genres, and styles, with ongoing evaluations to address and amend any emergent biases.

Performance Metrics

Accuracy:

Demonstrates high performance, matching and in some aspects surpassing that of contemporaries like LLaMA-7B across standardized benchmarks.

Robustness:

Proven capability to handle a variety of inputs and tasks, showcasing excellent generalization across numerous benchmarks and real-world applications.

Usage

Code Samples

Ethical Guidelines:

Adherence to ethical AI development practices, with an emphasis on transparency, fairness, and responsible use, highlighted in the documentation.

License Type:

Each variant of MPT-7B comes with specific licensing, from fully open Apache-2.0 to more restrictive CC-By-NC-SA-4.0 for certain uses, clearly delineated to inform appropriate usage.