Fsspec configuration

A common fsspec configuration is a shared set of parameters and settings utilized by connectors responsible for managing cloud-based file-system content. These configurations enable connectors to interact with cloud storage systems consistently, specifying details such as authentication credentials, access permissions, storage locations, and optimization options. By standardizing these fsspec configurations, connectors can seamlessly interface with various cloud-based file systems, promoting interoperability and efficient management of remote data storage across different cloud providers.

​

File Storage Configuration

Fsspec configuration is an extension of the File Storage Config:

•   recursive: Default: False. If a directory path is provided, whether further folders should be traversed recursively.

•   remote_url: Path to the content on the remote system, prefixed with the protocol (for example, s3://...).

•   uncompress: Default: False. Whether to uncompress tar and zip files when downloaded. Will ignore these otherwise.

​

Configs

The following are configurations set on the fsspec config itself:

•   access_config: A dictionary of access keyword arguments (kwargs) that might be needed to provide access to the cloud provider associated with the data.

The following are generated for the user after the config is initialized:

•   dir_path and file_path: The raw remote path is parsed into the directory and file if it exists.

•   path_without_protocol: The file path on the remote source without the protocol.

•   protocol: The protocol is extracted from the raw remote path passed in (for example, s3 from s3://...).