Model Overview Card for Koala (7B)

Basic Information

• Model Name: Koala (7B)
• Developer/Creator: Berkeley Artificial Intelligence Research (BAIR) Lab
• Release Date: April 2023
• Version: 1.0
• Model Type: Large Language Model (LLM)

Description

Koala (7B) is an open-source large language model developed by the Berkeley Artificial Intelligence Research (BAIR) Lab. It is designed to be a high-quality chatbot that rivals popular models like ChatGPT in terms of performance and capabilities.Key Features:

• High-quality performance comparable to ChatGPT
• Open-source and freely available for research and development
• Efficient 7 billion parameter architecture
• Fine-tuned on carefully curated datasets

Intended Use:

Koala is primarily intended for research purposes and as a foundation for developing advanced conversational AI applications.Language Support: English (primary), with potential for multilingual capabilities.

Technical Details

Architecture

Koala (7B) is based on the LLaMA architecture, specifically utilizing the 7B parameter version of LLaMA as its foundation. The model employs a transformer-based architecture, which has become the standard for state-of-the-art language models.

Training Data

The Koala model was fine-tuned on a carefully curated dataset comprising:

1. Anthropic's Helpful and Harmless (HH) dataset: This dataset consists of 67,000 human-AI conversation samples, focusing on helpful and safe interactions.
2. Open-Assistant conversations: A dataset of 9,000 samples from the Open-Assistant project, which aims to create open-source AI assistants.
3. Stanford Alpaca data: A dataset of 52,000 instruction-following demonstrations, generated using self-instruct techniques.

Data Source and Size

The total fine-tuning dataset for Koala consists of approximately 128,000 samples, combining the aforementioned sources. This relatively small dataset size demonstrates the efficiency of the fine-tuning process.

Knowledge Cutoff

The knowledge cutoff date for Koala (7B) is not explicitly stated in the available information. However, given its release date in April 2023, it's reasonable to assume that the model's knowledge is current up to early 2023.

Diversity and Bias

While specific information on diversity and bias in Koala is not provided, it's important to note that the model inherits biases present in its base model (LLaMA) and the datasets used for fine-tuning. Researchers and developers should be aware of potential biases and conduct thorough evaluations before deployment in sensitive applications.

Performance Metrics

Accuracy

Koala (7B) has demonstrated impressive performance in various benchmarks:

1. Human evaluation: In blind tests, human evaluators preferred Koala's responses over those of ChatGPT in 50% of cases, indicating comparable performance.
2. TruthfulQA: Koala achieved a score of 47%, surpassing GPT-3.5 and approaching the performance of GPT-4.
3. MMLU (Massive Multitask Language Understanding): Koala scored 43.3% on this comprehensive benchmark, showcasing its broad knowledge and reasoning capabilities.

Speed

Specific inference speed metrics for Koala (7B) are not provided in the available information. However, as a 7 billion parameter model, it is generally expected to be more efficient and faster in inference compared to larger models with similar capabilities.

Robustness

Koala (7B) has shown strong performance across various tasks and domains, as evidenced by its scores on diverse benchmarks like TruthfulQA and MMLU. This suggests good generalization capabilities and robustness across different topics and types of queries.

Usage

Code Samples

Ethical Guidelines

Explicit ethical guidelines for Koala (7B) are not provided in the available information. However, as an open-source model intended for research purposes, users should adhere to general AI ethics principles, including:

1. Responsible use and deployment
2. Awareness of potential biases
3. Consideration of privacy and data protection
4. Transparency in AI-generated content

License Type

The Koala (7B) model is released under an open-source license, allowing for research and development use.