Basic Information

• Model Name: Llama
• Developer/Creator: Meta
• Release Date: April 2024
• Version: 3.0 (70B)
• Model Type: Large Language Model (LLM)

Description

Overview

Llama-3 (70B) is a state-of-the-art open-source language model developed by Meta AI. This 70-billion parameter model is designed to excel in reasoning, coding, and broad application across multiple languages and domains.

Key Features

• Instruction-tuned for dialogue/chat: Llama-3 outperforms many open-source chat models on common benchmarks.
• Improved reasoning and coding capabilities: The model demonstrates strong performance on reasoning and coding tasks.
• Multilingual support: Over 5% of the pretraining dataset consists of high-quality non-English data covering 30+ languages.

Intended Use

Llama-3 (70B) is intended for a wide range of natural language processing tasks, including:

• Text generation: Generating coherent and contextually relevant text.
• Question answering: Providing accurate answers to questions based on the provided context.
• Sentiment analysis: Determining the sentiment (positive, negative, or neutral) of a given text.
• Text classification: Categorizing text into predefined classes or topics.
• Named entity recognition: Identifying and extracting named entities (e.g., people, organizations, locations) from text.

Language Support

Llama-3 (70B) supports over 30 languages, with a focus on high-quality non-English data, which accounts for more than 5% of the pretraining dataset.

Technical Details

Architecture

Llama-3 (70B) uses an optimized transformer architecture with grouped-query attention for improved inference scalability. The model's architecture is designed to efficiently process and generate text while maintaining high performance.

Training Data

Llama-3 (70B) was pretrained on over 15 trillion tokens from publicly available sources, including web pages, books, and other text corpora. The dataset includes a significant amount of code, with 4x more code than Llama-2.

Data Source and Size

The training data for Llama-3 (70B) comes from a variety of publicly available sources, such as web pages, books, and other text corpora. The total size of the pretraining dataset is over 15 trillion tokens.

Knowledge Cutoff

The knowledge cutoff for Llama-3 (70B) is December 2023, meaning that the model's knowledge is current up to that date.

Diversity and Bias

Llama-3 (70B) was trained on a diverse dataset that includes content from various sources and perspectives. However, as with any large language model, there may be biases present in the training data that could be reflected in the model's outputs.

Performance Metrics

Llama-3 (70B) has demonstrated strong performance on various benchmarks and tasks. It performs on par with GPT-4o on certain tasks while being 15x cheaper.

Usage

Ethical Guidelines

Meta has invested in tools to enhance the safety of Llama-3 (70B) and reduce the risk of harmful outputs. The model's usage is subject to Meta's ethical guidelines and principles.

License Type

Llama-3 (70B) is available under a custom commercial license from Meta.