SynthGenAI-Package for Generating Synthetic Datasets using LLMs

SynthGenAI is a package for generating Synthetic Datasets. The idea is to have a tool which is simple to use and can generate datasets on different topics by utilizing LLMs from different API providers. The package is designed to be modular and can be easily extended to include some different API providers for LLMs and new features.

Important

The package is still in the early stages of development and some features may not be fully implemented or tested. If you find any issues or have any suggestions, feel free to open an issue or create a pull request.

Why SynthGenAI now? 🤔

Interest in synthetic data generation has surged recently, driven by the growing recognition of data as a critical asset in AI development. As Ilya Sutskever, one of the most important figures in AI, says: 'Data is the fossil fuel of AI.' The more quality data we have, the better our models can perform. However, access to data is often restricted due to privacy concerns, or it may be prohibitively expensive to collect. Additionally, the vast amount of high-quality data on the internet has already been extensively mined. Synthetic data generation addresses these challenges by allowing us to create diverse and useful datasets using current pre-trained Large Language Models (LLMs). Beyond LLMs, synthetic data also holds immense potential for training and fine-tuning Small Language Models (SLMs), which are gaining popularity due to their efficiency and suitability for specific, resource-constrained applications. By leveraging synthetic data for both LLMs and SLMs, we can enhance performance across a wide range of use cases while balancing resource efficiency and model effectiveness. This approach enables us to harness the strengths of both synthetic and authentic datasets to achieve optimal outcomes.

Tools used for building SynthGenAI 🧰

The package is built using Python and the following libraries:

uv, An extremely fast Python package and project manager, written in Rust.
LiteLLM, A Python SDK for accessing LLMs from different API providers with standardized OpenAI Format.
Langfuse, LLMOps platform for observability, tracebility and monitoring of LLMs.
Pydantic, Data validation and settings management using Python type annotations.
Huggingface Hub & Datasets, A Python library for saving generated datasets on Hugging Face Hub.
Gradio, A Python library for creating UIs for machine learning models.

Quick Start 🚀

To quickly start using the SynthGenAI, you need to have the package installed. You can install it using the following command:

pip install synthgenai

After installation, simply run the following command in your terminal:

synthgenai

This will launch the Gradio UI for generating synthetic datasets.

To create datasets, you need to set up the following fields in the UI:

LLM Model: The LLM model to use (e.g., model_provider/model_name).
Temperature: The temperature for the LLM.
Top P: The top_p value for the LLM.
Max Tokens: The maximum number of tokens for the LLM.
API Base: The API base URL (optional).
API Key: The API key (optional).
Dataset Type: The type of dataset to generate (e.g., Raw, Instruction, Preference, Sentiment Analysis, Summarization, Text Classification).
Topic: The topic of the dataset.
Domains: The domains for the dataset (comma-separated).
Language: The language of the dataset.
Additional Description: Additional description for the dataset (optional).
Number of Entries: The number of entries in the dataset.
Hugging Face Token: The Hugging Face token.
Hugging Face Repo Name: The Hugging Face repository name.
LLM Environment Variables: Comma-separated environment variables for the LLM (e.g., KEY1=VALUE1, KEY2=VALUE2).

Installation 🛠️

To install the package, you can use the following command:

pip install synthgenai

or you can install the package directly from the source code using the following commands:

git clone https://github.com/Shekswess/synthgenai.git
uv build
pip install ./dist/synthgenai-{version}-py3-none-any.whl

Requirements 📋

To use the package, you need to have the following requirements installed:

Python 3.10+
uv for building the package directly from the source code
Ollama running on your local machine if you want to use Ollama as an API provider (optional)
Langfuse running on your local machine or in the cloud if you want to use Langfuse for tracebility (optional)
Hugging Face Hub account if you want to save the generated datasets on Hugging Face Hub with generated token (optional)
Gradio for using the SynthGenAI UI (optional)

Usage 👨‍💻

The available API providers for LLMs are:

Groq
Mistral AI
Gemini
Bedrock
Anthropic
OpenAI
Hugging Face
Ollama
vLLM
SageMaker
Azure
Vertex AI

For observing the generated datasets, you can use Langfuse for tracebility and monitoring of the LLMs.

To use the LLMs from different API providers, to observe the generated datasets, and to save the generated datasets on Hugging Face Hub, you need to set the following environment variables:

# API keys for different LLM providers
GROQ_API_KEY=
MISTRAL_API_KEY=
GEMINI_API_KEY=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION=
AWS_PROFILE=
ANTHROPIC_API_KEY=
OPENAI_API_KEY=
AZURE_API_KEY
AZURE_API_BASE
AZURE_API_VERSION
AZURE_AD_TOKEN
AZURE_API_TYPE
GOOGLE_APPLICATION_CREDENTIALS
VERTEXAI_LOCATION
VERTEXAI_PROJECT
HUGGINGFACE_API_KEY=

# Langfuse API keys
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_HOST=

# Huggingface token for uploading datasets on Huggingface
HF_TOKEN=

Currently there are six types of datasets that can be generated using SynthGenAI:

Raw Datasets
Instruction Datasets
Preference Datasets
Sentiment Analysis Datasets
Summarization Datasets
Text Classification Datasets

The datasets can be generated:

Synchronously - each dataset entry is generated one by one
Asynchronously - batch of dataset entries is generated at once

Note

Asynchronous generation is faster than synchronous generation, but some of LLM providers can have limitations on the number of tokens that can be generated at once.

Raw Datasets 🥩

To generate a raw dataset, you can use the following code:

# For asynchronous dataset generation
# import asyncio
import os

from synthgenai import (
    DatasetConfig,
    DatasetGeneratorConfig,
    LLMConfig,
    RawDatasetGenerator,
)

# Setting the API keys
os.environ["LLM_API_KEY"] = ""

# Optional for Langfuse Tracebility
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_HOST"] = ""

# Optional for Hugging Face Hub upload
os.environ["HF_TOKEN"] = ""

# Creating the LLMConfig
llm_config = LLMConfig(
    model="model_provider/model_name", # Check liteLLM docs for more info
    temperature=0.5,
    top_p=0.9,
    max_tokens=2048,
)

# Creating the DatasetConfig
dataset_config = DatasetConfig(
    topic="topic_name",
    domains=["domain1", "domain2"],
    language="English",
    additional_description="Additional description",
    num_entries=1000
)

# Creating the DatasetGeneratorConfig
dataset_generator_config = DatasetGeneratorConfig(
    llm_config=llm_config,
    dataset_config=dataset_config,
)

# Creating the RawDatasetGenerator
raw_dataset_generator = RawDatasetGenerator(dataset_generator_config)

# Generating the dataset
raw_dataset = raw_dataset_generator.generate_dataset()

# Generating the dataset asynchronously
# raw_dataset = asyncio.run(raw_dataset_generator.agenerate_dataset())

# Name of the Hugging Face repository where the dataset will be saved
hf_repo_name = "organization_or_user_name/dataset_name" # optional

# Saving the dataset to the locally and to the Hugging Face repository(optional)
raw_dataset.save_dataset(
    hf_repo_name=hf_repo_name,
)

Example of generated entry for the raw dataset:

{
  "keyword": "keyword",
  "topic": "topic",
  "language": "language",
  "generated_entry": {
    "text": "generated text"
  }
}

Instruction Datasets 💬

To generate an instruction dataset, you can use the following code:

# For asynchronous dataset generation
# import asyncio
import os

from synthgenai import (
    DatasetConfig,
    DatasetGeneratorConfig,
    LLMConfig,
    InstructionDatasetGenerator,
)

# Setting the API keys
os.environ["LLM_API_KEY"] = ""

# Optional for Langfuse Tracebility
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_HOST"] = ""

# Optional for Hugging Face Hub upload
os.environ["HF_TOKEN"] = ""

# Creating the LLMConfig
llm_config = LLMConfig(
    model="model_provider/model_name", # Check liteLLM docs for more info
    temperature=0.5,
    top_p=0.9,
    max_tokens=2048,
)

# Creating the DatasetConfig
dataset_config = DatasetConfig(
    topic="topic_name",
    domains=["domain1", "domain2"],
    language="English",
    additional_description="Additional description",
    num_entries=1000
)

# Creating the DatasetGeneratorConfig
dataset_generator_config = DatasetGeneratorConfig(
    llm_config=llm_config,
    dataset_config=dataset_config,
)

# Creating the InstructionDatasetGenerator
instruction_dataset_generator = InstructionDatasetGenerator(dataset_generator_config)

# Generating the dataset
instruction_dataset = instruction_dataset_generator.generate_dataset()

# Generating the dataset asynchronously
# instruction_dataset = asyncio.run(instruction_dataset_generator.agenerate_dataset())

# Name of the Hugging Face repository where the dataset will be saved
hf_repo_name = "organization_or_user_name/dataset_name" # optional

# Saving the dataset to the locally and to the Hugging Face repository(optional)
instruction_dataset.save_dataset(
    hf_repo_name=hf_repo_name,
)

Example of generated entry for the instruction dataset:

{
  "keyword": "keyword",
  "topic": "topic",
  "language": "language",
  "generated_entry": {
    "messages": [
      {
        "role": "system",
        "content": "generated system(instruction) prompt"
      },
      {
        "role": "user",
        "content": "generated user prompt"
      },
      {
        "role": "assistant",
        "content": "generated assistant prompt"
      }
    ]
  }
}

Preference Datasets 🌟

To generate a preference dataset, you can use the following code:

# For asynchronous dataset generation
# import asyncio
import os

from synthgenai import (
    DatasetConfig,
    DatasetGeneratorConfig,
    LLMConfig,
    PreferenceDatasetGenerator,
)

# Setting the API keys
os.environ["LLM_API_KEY"] = ""

# Optional for Langfuse Tracebility
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_HOST"] = ""

# Optional for Hugging Face Hub upload
os.environ["HF_TOKEN"] = ""

# Creating the LLMConfig
llm_config = LLMConfig(
    model="model_provider/model_name", # Check liteLLM docs for more info
    temperature=0.5,
    top_p=0.9,
    max_tokens=2048,
)

# Creating the DatasetConfig
dataset_config = DatasetConfig(
    topic="topic_name",
    domains=["domain1", "domain2"],
    language="English",
    additional_description="Additional description",
    num_entries=1000
)

# Creating the DatasetGeneratorConfig
dataset_generator_config = DatasetGeneratorConfig(
    llm_config=llm_config,
    dataset_config=dataset_config,
)

# Creating the PreferenceDatasetGenerator
preference_dataset_generator = PreferenceDatasetGenerator(dataset_generator_config)

# Generating the dataset
preference_dataset = preference_dataset_generator.generate_dataset()

# Generating the dataset asynchronously
# preference_dataset = asyncio.run(preference_dataset_generator.agenerate_dataset())

# Name of the Hugging Face repository where the dataset will be saved
hf_repo_name = "organization_or_user_name/dataset_name" # optional

# Saving the dataset to the locally and to the Hugging Face repository(optional)
preference_dataset.save_dataset(
    hf_repo_name=hf_repo_name,
)

Example of generated entry for the preference dataset:

{
  "keyword": "keyword",
  "topic": "topic",
  "language": "language",
  "generated_entry": {
    "prompt": [
      { "role": "system", "content": "generated system(instruction) prompt" },
      { "role": "user", "content": "generated user prompt" }
    ],
    "chosen": [
      { "role": "assistant", "content": "generated chosen assistant response" }
    ],
    "rejected": [
      {
        "role": "assistant",
        "content": "generated rejected assistant response"
      }
    ]
  }
}

Sentiment Analysis Datasets 🎭

To generate a sentiment analysis dataset, you can use the following code:

# For asynchronous dataset generation
# import asyncio
import os

from synthgenai import (
    DatasetConfig,
    DatasetGeneratorConfig,
    LLMConfig,
    SentimentAnalysisDatasetGenerator,
)

# Setting the API keys
os.environ["LLM_API_KEY"] = ""

# Optional for Langfuse Tracebility
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_HOST"] = ""

# Optional for Hugging Face Hub upload
os.environ["HF_TOKEN"] = ""

# Creating the LLMConfig
llm_config = LLMConfig(
    model="model_provider/model_name", # Check liteLLM docs for more info
    temperature=0.5,
    top_p=0.9,
    max_tokens=2048,
)

# Creating the DatasetConfig
dataset_config = DatasetConfig(
    topic="topic_name",
    domains=["domain1", "domain2"],
    language="English",
    additional_description="Additional description",
    num_entries=1000
)

# Creating the DatasetGeneratorConfig
dataset_generator_config = DatasetGeneratorConfig(
    llm_config=llm_config,
    dataset_config=dataset_config,
)

# Creating the SentimentAnalysisDatasetGenerator
sentiment_analysis_dataset_generator = SentimentAnalysisDatasetGenerator(dataset_generator_config)

# Generating the dataset
sentiment_analysis_dataset = sentiment_analysis_dataset_generator.generate_dataset()

# Generating the dataset asynchronously
# sentiment_analysis_dataset = asyncio.run(sentiment_analysis_dataset_generator.agenerate_dataset())

# Name of the Hugging Face repository where the dataset will be saved
hf_repo_name = "organization_or_user_name/dataset_name" # optional

# Saving the dataset to the locally and to the Hugging Face repository(optional)
sentiment_analysis_dataset.save_dataset(
    hf_repo_name=hf_repo_name,
)

Example of generated entry for the sentiment analysis dataset:

{
  "keyword": "keyword",
  "topic": "topic",
  "language": "language",
  "generated_entry": {
    "prompt": "generated text",
    "label": "generated sentiment (which can be positive, negative, neutral)"
  }
}

Text Classification Datasets 🔠

To generate a text classification dataset, you can use the following code:

# For asynchronous dataset generation
# import asyncio
import os

from synthgenai import (
    DatasetConfig,
    DatasetGeneratorConfig,
    LLMConfig,
    TextClassificationDatasetGenerator,
)

# Setting the API keys
os.environ["LLM_API_KEY"] = ""

# Optional for Langfuse Tracebility
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_HOST"] = ""

# Optional for Hugging Face Hub upload
os.environ["HF_TOKEN"] = ""

# Creating the LLMConfig
llm_config = LLMConfig(
    model="model_provider/model_name", # Check liteLLM docs for more info
    temperature=0.5,
    top_p=0.9,
    max_tokens=2048,
)

# Creating the DatasetConfig
dataset_config = DatasetConfig(
    topic="topic_name",
    domains=["domain1", "domain2"],
    language="English",
    additional_description="Additional description",
    num_entries=1000
)

# Creating the DatasetGeneratorConfig
dataset_generator_config = DatasetGeneratorConfig(
    llm_config=llm_config,
    dataset_config=dataset_config,
)

# Creating the TextClassificationDatasetGenerator
text_classification_dataset_generator = TextClassificationDatasetGenerator(dataset_generator_config)

# Generating the dataset
text_classification_dataset = text_classification_dataset_generator.generate_dataset()

# Generating the dataset asynchronously
# text_classification_dataset = asyncio.run(text_classification_dataset_generator.agenerate_dataset())

# Name of the Hugging Face repository where the dataset will be saved
hf_repo_name = "organization_or_user_name/dataset_name" # optional

# Saving the dataset to the locally and to the Hugging Face repository(optional)
text_classification_dataset.save_dataset(
    hf_repo_name=hf_repo_name,
)

Example of generated entry for the text classification dataset:

{
  "keyword": "keyword",
  "topic": "topic",
  "language": "language",
  "generated_entry": {
    "prompt": "generated text",
    "label": "generated sentiment (which will be from a list of labels, created from the model)"
  }
}

Summarization Datasets 🧾

To generate a summarization dataset, you can use the following code:

# For asynchronous dataset generation
# import asyncio
import os

from synthgenai import (
    DatasetConfig,
    DatasetGeneratorConfig,
    LLMConfig,
    SummarizationDatasetGenerator,
)

# Setting the API keys
os.environ["LLM_API_KEY"] = ""

# Optional for Langfuse Tracebility
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_HOST"] = ""

# Optional for Hugging Face Hub upload
os.environ["HF_TOKEN"] = ""

# Creating the LLMConfig
llm_config = LLMConfig(
    model="model_provider/model_name", # Check liteLLM docs for more info
    temperature=0.5,
    top_p=0.9,
    max_tokens=2048,
)

# Creating the DatasetConfig
dataset_config = DatasetConfig(
    topic="topic_name",
    domains=["domain1", "domain2"],
    language="English",
    additional_description="Additional description",
    num_entries=1000
)

# Creating the DatasetGeneratorConfig
dataset_generator_config = DatasetGeneratorConfig(
    llm_config=llm_config,
    dataset_config=dataset_config,
)

# Creating the SummarizationDatasetGenerator
summarization_dataset_generator = SummarizationDatasetGenerator(dataset_generator_config)

# Generating the dataset
summarization_dataset = summarization_dataset_generator.generate_dataset()

# Generating the dataset asynchronously
# summarization_dataset = asyncio.run(summarization_dataset_generator.agenerate_dataset())

# Name of the Hugging Face repository where the dataset will be saved
hf_repo_name = "organization_or_user_name/dataset_name" # optional

# Saving the dataset to the locally and to the Hugging Face repository(optional)
summarization_dataset.save_dataset(
    hf_repo_name=hf_repo_name,
)

Example of generated entry for the summarization dataset:

{
  "keyword": "keyword",
  "topic": "topic",
  "language": "language",
  "generated_entry": {
    "text": "generated text",
    "summary": "generated summary"
  }
}

More Examples 📖

More examples with different combinations of LLM API providers and dataset configurations can be found in the examples directory.

Important

Sometimes the generation of the keywords for the dataset and the dataset entries can fail due to the limitation of the LLM to generate JSON Object as output (this is handled by the package). That's why it is recommended to use models that are capable of generating JSON Objects (structured output). List of models that can generate JSON Objects can be found here.

Generated Datasets 📚

Examples of generated synthetic datasets can be found on the SynthGenAI Datasets Collection on Hugging Face Hub.

Supported API Providers 💪

Contributing 🤝

If you want to contribute to this project and make it better, your help is very welcome. Create a pull request with your changes and I will review it. If you have any questions, open an issue.

License 📝

This project is licensed under the MIT License - see the LICENSE.md file for details.

Repo Structure 📂

.
├── .github
│   └── workflows
│       ├── build_n_release.yml
│       ├── docs.yml
│       └── tests.yml
├── docs
│   ├── assets
│   │   ├── favicon.png
│   │   ├── logo_header.png
│   │   ├── logo.svg
│   │   └── ui.png
│   ├── configurations
│   │   ├── dataset_configuration.md
│   │   ├── dataset_generator_configuration.md
│   │   └── index.md
│   ├── contributing
│   │   └── index.md
│   ├── datasets
│   │   ├── index.md
│   │   ├── instruction_dataset.md
│   │   ├── preference_dataset.md
│   │   ├── raw_dataset.md
│   │   ├── sentiment_analysis_dataset.md
│   │   ├── summarization_dataset.md
│   │   └── text_classification_dataset.md
│   ├── examples
│   │   └── index.md
│   ├── installation
│   │   └── index.md
│   ├── llm_providers
│   │   └── index.md
│   ├── ui
│   │   └── index.md
│   └── index.md
├── examples
│   ├── anthropic_instruction_dataset_example.py
│   ├── azure_ai_preference_dataset_example.py
│   ├── azure_summarization_dataset_example.py
│   ├── bedrock_raw_dataset_example.py
│   ├── gemini_langfuse_raw_dataset_example.py
│   ├── groq_preference_dataset_example.py
│   ├── huggingface_instruction_dataset_example.py
│   ├── mistral_preference_dataset_example.py
│   ├── ollama_preference_dataset_example.py
│   ├── openai_raw_dataset_example.py
│   ├── sagemaker_summarization_dataset_example.py
│   ├── vertex_ai_text_classification_dataset_example.py
│   └── vllm_sentiment_analysis_dataset_example.py
├── synthgenai
│   ├── data_model.py
│   ├── dataset_generator.py
│   ├── dataset.py
│   ├── __init__.py
│   ├── llm.py
│   ├── prompts.py
│   ├── ui.py
│   └── utils.py
├── tests
│   ├── test_dataset.py
│   └── test_llm.py
├── .gitignore
├── .python-version
├── LICENSE.txt
├── mkdocs.yml
├── pyproject.toml
├── README.md
└── uv.lock

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

SynthGenAI-Package for Generating Synthetic Datasets using LLMs

Why SynthGenAI now? 🤔

Tools used for building SynthGenAI 🧰

Quick Start 🚀

Installation 🛠️

Requirements 📋

Usage 👨‍💻

Raw Datasets 🥩

Instruction Datasets 💬

Preference Datasets 🌟

Sentiment Analysis Datasets 🎭

Text Classification Datasets 🔠

Summarization Datasets 🧾

More Examples 📖

Generated Datasets 📚

Supported API Providers 💪

Contributing 🤝

License 📝

Repo Structure 📂

About

Releases 6

Contributors 2

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 107 Commits
.github/workflows		.github/workflows
docs		docs
examples		examples
synthgenai		synthgenai
tests		tests
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
.python-version		.python-version
LICENCE.txt		LICENCE.txt
README.md		README.md
mkdocs.yml		mkdocs.yml
pyproject.toml		pyproject.toml
uv.lock		uv.lock

License

Shekswess/synthgenai

Folders and files

Latest commit

History

Repository files navigation

SynthGenAI-Package for Generating Synthetic Datasets using LLMs

Why SynthGenAI now? 🤔

Tools used for building SynthGenAI 🧰

Quick Start 🚀

Installation 🛠️

Requirements 📋

Usage 👨‍💻

Raw Datasets 🥩

Instruction Datasets 💬

Preference Datasets 🌟

Sentiment Analysis Datasets 🎭

Text Classification Datasets 🔠

Summarization Datasets 🧾

More Examples 📖

Generated Datasets 📚

Supported API Providers 💪

Contributing 🤝

License 📝

Repo Structure 📂

About

Topics

Resources

License

Stars

Watchers

Forks

Releases 6

Contributors 2

Languages