ParData

PyPI PyPI - Python Version PyPI - Implementation Gitter Runtime Tests Lint Docs Development Environment

ParData (homophone of partake) is a Python API that enables data consumers and distributors to easily use and share datasets, and establishes a standard for exchanging data assets. It enables:

  • a data scientist to have a simpler and more unified way to begin working with a wide range of datasets, and

  • a data distributor to have a consistent, safe, and open source way to share datasets with interested communities.

Quick Example

>>> import pardata
>>> pardata.list_all_datasets()
{'claim_sentences_search': ('1.0.2',),
 ..., 'wikitext103': ('1.0.1',)}
>>> pardata.load_dataset('wikitext103')
{...}  # Content of the dataset

Install the Package & its Dependencies

To install the latest version of ParData, run

$ pip install pardata

Alternatively, if you have downloaded the source, switch to the source directory (same directory as this README file, cd /path/to/pardata-source) and run

$ pip install -U .

Quick Start

Import the package and load a dataset. ParData will download WikiText-103 dataset (version 1.0.1) if it’s not already downloaded, and then load it.

import pardata
wikitext103_data = pardata.load_dataset('wikitext103')

View available ParData datasets and their versions.

>>> pardata.list_all_datasets()
{'claim_sentences_search': ('1.0.2',), ..., 'wikitext103': ('1.0.1',)}

To view your globally set configs for ParData, such as your default data directory, use pardata.get_config().

>>> pardata.get_config()
Config(DATADIR=PosixPath('dir/to/download/load/from'), ..., DATASET_SCHEMA_FILE_URL='file/to/load/datasets/from')

By default, pardata.load_dataset() downloads to and loads from ~/.pardata/data/<dataset-name>/<dataset-version>/. To change the default data directory, use pardata.init().

pardata.init(DATADIR='new/dir/to/download/load/from')

Load a previously downloaded dataset using pardata.load_dataset(). With the new default data dir set, ParData now searches for the Groningen Meaning Bank dataset (version 1.0.2) in new/dir/to/download/load/from/gmb/1.0.2/.

gmb_data = load_dataset('gmb', version='1.0.2', download=False)  # assuming GMB dataset was already downloaded

To learn more about ParData, check out the documentation and the tutorial.

API References

Top-Level

pardata

ParData package and high level functions.

pardata.dataset

Dataset downloading and loading functionality.

pardata.exceptions

Custom exceptions used by this package.

pardata.schema

Schema parsing and loading functionality.

Loaders

pardata.loaders

Loaders subpackage.

pardata.loaders.table

Tabular data loaders.

pardata.loaders.text

Text file loaders.

Schema File Format References

Miscellaneous