Model Overview Card MPT-Chat (30B)

Basic Information

• Model Name: MPT-Chat (30B)
• Developer/Creator: MosaicML, part of Databricks
• Release Date: June 22, 2023
• Version: Initial Release
• Model Type: Text-based Language Model

Description

Overview

MPT-Chat (30B) is an advanced, open-source language model designed for a broad spectrum of natural language processing tasks, emphasizing efficiency, scalability, and ethical AI practices.

Key Features

• Decoder-only transformer architecture.
• Large model size with 30 billion parameters.
• Supports a context window of up to 8,192 tokens.
• Utilizes innovative techniques like FlashAttention and ALiBi.

Intended Use

This model is tailored for:

• Open-ended text generation.
• Question answering.
• Summarization.
• Code completion.

Language Support

Currently, detailed language support specifics are not provided; however, it typically includes major global languages given its extensive training data.

Technical Details

Architecture

MPT-Chat (30B) employs a decoder-only transformer architecture, similar to GPT models, enhanced with modern techniques like FlashAttention for efficient attention computation and ALiBi for positional biases, which facilitate better scaling and performance.

Training Data

The model was trained on a curated dataset of 1 trillion tokens, encompassing a wide range of internet text to ensure relevance and diversity.

Data Source and Size

The training dataset is significantly large at 1 trillion tokens, selected for its high quality and comprehensive coverage of various domains.

Knowledge Cutoff

The information incorporated in the model is up-to-date as of its last training cut-off in early 2023.

Diversity and Bias

MPT-Chat (30B) was developed using constitutional AI principles to align closely with human values and minimize biases, supported by rigorous testing to detect and mitigate any unintended biases.

Performance Metrics

• Accuracy: Not specified, but comparable to other models of similar size.
• Speed: Optimized for real-time applications through efficient training methods.
• Robustness: Exhibits strong zero-shot and few-shot learning capabilities, adapting well across diverse tasks and languages.

Usage

Code Samples

Ethical Guidelines

Outlined as part of the development process, focusing on responsible AI use and bias mitigation.

License Type

Apache 2.0 license, permitting both commercial and non-commercial use.

Conclusion

MPT-Chat (30B) sets a new benchmark in the realm of open-source language models by combining large-scale machine learning capabilities with a commitment to ethical AI practices, making it highly suitable for developers and researchers in the AI community.