khiops.core.internals.filesystems

Submodule of khiops.core.internals

Classes to interact with local and remote filesystems

CLASS	DESCRIPTION
`FilesystemResource`	Abstract Filesystem Resource
`LocalFilesystemResource`	A local filesystem resource
`GoogleCloudStorageResource`	Google Cloud Storage Resource
`AmazonS3Resource`	Amazon Simple Storage Service (S3) Resource
`AzureStorageResourceMixin`	Azure compatible Storage Resource Mixin
`AzureStorageFileResource`	Azure compatible Storage File Resource
`AzureStorageBlobResource`	Azure compatible Storage Blob Resource

FUNCTION	DESCRIPTION
`is_local_resource`	Checks if a URI or path is effectively a local path
`create_resource`	Factory method to create a FilesystemResource from an URI
`read`	Reads data in binary mode from the target resource
`write`	Writes data in binary mode to the target resource
`exists`	Checks the existence of the resource
`remove`	Removes the resource
`copy_from_local`	Copies the content of a local file to the resource
`copy_to_local`	Copies the content of the resource to a local path
`list_dir`	Lists a directory resource contents
`make_dir`	Creates a directory at the resource's path
`get_child_path`	Returns the child path of this URI at the specified child name
`get_parent_path`	Returns the parent path of this URI

Classes

`FilesystemResource(uri)`

Bases: ABC

Abstract Filesystem Resource

METHOD	DESCRIPTION
`create_child`	Creates a resource representing a child of this instance
`create_parent`	Creates a resource representing the parent of this instance

Methods:

`create_child(file_name)` `abstractmethod`

Creates a resource representing a child of this instance

RETURNS	DESCRIPTION
`FilesystemResource`	The specific resource type is that of the caller

`create_parent()` `abstractmethod`

Creates a resource representing the parent of this instance

RETURNS	DESCRIPTION
`FilesystemResource`	The specific resource type is that of the caller

`LocalFilesystemResource(uri)`

Bases: FilesystemResource

A local filesystem resource

`GoogleCloudStorageResource(uri)`

Bases: FilesystemResource

Google Cloud Storage Resource

By default, it reads the configuration from standard location.

`AmazonS3Resource(uri)`

Bases: FilesystemResource

Amazon Simple Storage Service (S3) Resource

The default configuration and credentials are read from the paths

~/.aws/configuration
~/.aws/credentials

The location of the configuration and credentials files may be overridden using the following environment variables:

AWS_CONFIG_FILE: location of the configuration file
AWS_SHARED_CREDENTIALS_FILE: location of the credentials file

If no configuration/credentials files are usable, Amazon SDK defaults apply. Individual settings such as endpoint URL or region can be used to override any of the available settings.

Other relevant environment variables:

AWS_S3_ENDPOINT_URL: sets the service endpoint URL
AWS_DEFAULT_REGION: sets the region to send requests to

Note

Operations with the s3 client are only verified by checking that the HTTP response code is in the 200 range.

`AzureStorageResourceMixin(uri)`

Azure compatible Storage Resource Mixin

See AzureStorageFileResource and AzureStorageBlobResource for more details.

Azure Storage Resource initializer common to Files and Blobs

ATTRIBUTE	DESCRIPTION
`uri_parts`	Build a list of path parts from a path as a str

Attributes

`uri_parts` `cached` `property`

Build a list of path parts from a path as a str

Methods:

`AzureStorageFileResource(uri)`

Bases: AzureStorageResourceMixin, FilesystemResource

Azure compatible Storage File Resource

As a prerequisite, the remote storage account and "file share" on Azure MUST have been created by the user.

For shared Files, the URI pattern of a resource is the following : https://.file.core.windows.net//... <0 to n folder name(s)>/. The first name after the netloc is the "share name" not a simple "folder name".

By default, this resource reads the configuration from standard location (environment variables for the moment)

Azure Storage Resource initializer for Files

METHOD	DESCRIPTION
`exists`	Check if the target resource exists (file or directory)
`remove`	Remove the target resource either it is a file or a directory.
`list_dir`	List the files (not the directories) of the current directory

Methods:

`exists()`

Check if the target resource exists (file or directory)

`remove()`

Remove the target resource either it is a file or a directory. If an error occurs an exception is raised.

`list_dir()`

List the files (not the directories) of the current directory

Note

This is not a recursive listing operation.

`AzureStorageBlobResource(uri)`

Bases: AzureStorageResourceMixin, FilesystemResource

Azure compatible Storage Blob Resource

As a prerequisite, the remote storage account and the blob "container" on Azure MUST have been created by the user.

For blobs, the URI pattern of a resource is the following: https://.blob.core.windows.net//... <0 to n virtual folder name(s)>/. The "virtual folder names" are part of the blob name but can help simulate a folder hierarchy. The first name after the netloc is the "container name" not a simple "virtual folder name".

By default, this resource reads the configuration from standard location (environment variables for the moment)

Azure Storage Resource initializer for Blobs

METHOD	DESCRIPTION
`list_dir`	List the files of the current directory

Methods:

`list_dir()`

List the files of the current directory

Functions:

`is_local_resource(uri_or_path)`

Checks if a URI or path is effectively a local path

Note

An URI with scheme of size 1 will be considered a local path. This is to take into account Windows paths such as C:\Some\Windows\Path.

RETURNS	DESCRIPTION
`bool`	True if a URI refers to a local path

`create_resource(uri_or_path)`

Factory method to create a FilesystemResource from an URI

PARAMETER	DESCRIPTION
`uri_or_path`	The resource's URI . Supported protocols/schemes: `file` or empty: Local filesystem resource `s3`: Amazon S3 resource `gs`: Google Cloud Storage resource `https`: Azure Storage resource (files or blobs) TYPE: `str`

RETURNS	DESCRIPTION
`FilesystemResource`	The URI resource object, its class depends on the URI.

`read(uri_or_path, size=None)`

Reads data in binary mode from the target resource

PARAMETER	DESCRIPTION
`uri_or_path`	The resource's URI or local filesystem path. TYPE: `str`
`size`	Number of bytes to read. TYPE: `int` DEFAULT: `None`

RETURNS	DESCRIPTION
`bytes`	A buffer containing the read contents.

RAISES	DESCRIPTION
`RuntimeError`	If there was a problem when reading.

`write(uri_or_path, data)`

Writes data in binary mode to the target resource

PARAMETER	DESCRIPTION
`uri_or_path`	The resource's URI or local filesystem path. TYPE: `str`
`data`	The data to be written. TYPE: `str or bytes`

RAISES	DESCRIPTION
`RuntimeError`	If there was a problem when writing.

`exists(uri_or_path)`

Checks the existence of the resource

PARAMETER	DESCRIPTION
`uri_or_path`	The resource's URI or local filesystem path. TYPE: `str`

RETURNS	DESCRIPTION
`bool`	True if the resource exists.

`remove(uri_or_path)`

Removes the resource

PARAMETER	DESCRIPTION
`uri_or_path`	The resource's URI or local filesystem path. TYPE: `str`

RAISES	DESCRIPTION
`RuntimeError`	If there was a problem when removing.

`copy_from_local(uri_or_path, local_path)`

Copies the content of a local file to the resource

PARAMETER	DESCRIPTION
`uri_or_path`	The resource's URI or local filesystem path. TYPE: `str`
`local_path`	The path of the local file to be copied. TYPE: `str`

RAISES	DESCRIPTION
`RuntimeError`	If there was a problem when copying.

`copy_to_local(uri_or_path, local_path)`

Copies the content of the resource to a local path

PARAMETER	DESCRIPTION
`uri_or_path`	The resource's URI or local filesystem path. TYPE: `str`
`local_path`	The path of the local file to be copied. TYPE: `str`

RAISES	DESCRIPTION
`RuntimeError`	If there was a problem when copying.

`list_dir(uri_or_path)`

Lists a directory resource contents

PARAMETER	DESCRIPTION
`uri_or_path`	The resource's URI or local filesystem path. TYPE: `str`

RETURNS	DESCRIPTION
`list`	A list of paths or URIs.

`make_dir(uri_or_path)`

Creates a directory at the resource's path

PARAMETER	DESCRIPTION
`uri_or_path`	The resource's URI or local filesystem path. TYPE: `str`

`get_child_path(uri_or_path, child_name)`

Returns the child path of this URI at the specified child name

PARAMETER	DESCRIPTION
`uri_or_path`	The resource's URI or local filesystem path. TYPE: `str`
`child_name`	Name of the child to be added to the URI or path. TYPE: `str`

RETURNS	DESCRIPTION
`str`	The URI or path of the child resource.

`get_parent_path(uri_or_path)`

Returns the parent path of this URI

PARAMETER	DESCRIPTION
`uri_or_path`	The resource's URI or local filesystem path. TYPE: `str`

RETURNS	DESCRIPTION
`str`	The URI or path of the parent's resource.

khiops.core.internals.filesystems

Classes

FilesystemResource(uri)

Methods:

create_child(file_name) abstractmethod

create_parent() abstractmethod

LocalFilesystemResource(uri)

GoogleCloudStorageResource(uri)

AmazonS3Resource(uri)

AzureStorageResourceMixin(uri)

Attributes

uri_parts cached property

Methods:

AzureStorageFileResource(uri)

Methods:

exists()

remove()

list_dir()

AzureStorageBlobResource(uri)

Methods:

list_dir()

Functions:

is_local_resource(uri_or_path)

create_resource(uri_or_path)

read(uri_or_path, size=None)

write(uri_or_path, data)

exists(uri_or_path)

remove(uri_or_path)

copy_from_local(uri_or_path, local_path)

copy_to_local(uri_or_path, local_path)

list_dir(uri_or_path)

make_dir(uri_or_path)

get_child_path(uri_or_path, child_name)

get_parent_path(uri_or_path)

`FilesystemResource(uri)`

`create_child(file_name)` `abstractmethod`

`create_parent()` `abstractmethod`

`LocalFilesystemResource(uri)`

`GoogleCloudStorageResource(uri)`

`AmazonS3Resource(uri)`

`AzureStorageResourceMixin(uri)`

`uri_parts` `cached` `property`

`AzureStorageFileResource(uri)`

`exists()`

`remove()`

`list_dir()`

`AzureStorageBlobResource(uri)`

`list_dir()`

`is_local_resource(uri_or_path)`

`create_resource(uri_or_path)`

`read(uri_or_path, size=None)`

`write(uri_or_path, data)`

`exists(uri_or_path)`

`remove(uri_or_path)`

`copy_from_local(uri_or_path, local_path)`

`copy_to_local(uri_or_path, local_path)`

`list_dir(uri_or_path)`

`make_dir(uri_or_path)`

`get_child_path(uri_or_path, child_name)`

`get_parent_path(uri_or_path)`