core.internals.filesystems¶

Submodule of khiops.core.internals

Classes to interact with local and remote filesystems

Functions¶

`copy_from_local`	Copies the content of a local file to the resource
`copy_to_local`	Copies the content of the resource to a local path
`create_resource`	Factory method to create a FilesystemResource from an URI
`exists`	Checks the existence of the resource
`get_child_path`	Returns the child path of this URI at the specified child name
`get_parent_path`	Returns the parent path of this URI
`is_local_resource`	Checks if a URI or path is effectively a local path
`list_dir`	Lists a directory resource contents
`make_dir`	Creates a directory at the resource's path
`read`	Reads data in binary mode from the target resource
`remove`	Removes the resource
`write`	Writes data in binary mode to the target resource

Classes¶

`AmazonS3Resource`	Amazon Simple Storage Service (S3) Resource
`AzureStorageBlobResource`	Azure compatible Storage Blob Resource
`AzureStorageFileResource`	Azure compatible Storage File Resource
`AzureStorageResourceMixin`	Azure compatible Storage Resource Mixin
`FilesystemResource`	Abstract Filesystem Resource
`GoogleCloudStorageResource`	Google Cloud Storage Resource
`LocalFilesystemResource`	A local filesystem resource

class khiops.core.internals.filesystems.AmazonS3Resource(uri)¶

Bases: FilesystemResource

Amazon Simple Storage Service (S3) Resource

The default configuration and credentials are read from the paths

~/.aws/configuration
~/.aws/credentials

The location of the configuration and credentials files may be overridden using the following environment variables:

AWS_CONFIG_FILE: location of the configuration file
AWS_SHARED_CREDENTIALS_FILE: location of the credentials file

If no configuration/credentials files are usable, Amazon SDK defaults apply. Individual settings such as endpoint URL or region can be used to override any of the available settings.

Other relevant environment variables:

- AWS_S3_ENDPOINT_URL: sets the service endpoint URL
- AWS_DEFAULT_REGION: sets the region to send requests to

Note

Operations with the s3 client are only verified by checking that the HTTP response code is in the 200 range.

create_child(file_name)¶

Creates a resource representing a child of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

create_parent()¶

Creates a resource representing the parent of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

class khiops.core.internals.filesystems.AzureStorageBlobResource(uri)¶

Bases: AzureStorageResourceMixin, FilesystemResource

Azure compatible Storage Blob Resource

As a prerequisite, the remote storage account and the blob “container” on Azure MUST have been created by the user.

For blobs, the URI pattern of a resource is the following: https://<storage-account-name>.blob.core.windows.net/<container name>/… <0 to n virtual folder name(s)>/<blob name>. The “virtual folder names” are part of the blob name but can help simulate a folder hierarchy. The first name after the netloc is the “container name” not a simple “virtual folder name”.

By default, this resource reads the configuration from standard location (environment variables for the moment)

list_dir()¶: List the files of the current directory

class khiops.core.internals.filesystems.AzureStorageFileResource(uri)¶

Bases: AzureStorageResourceMixin, FilesystemResource

Azure compatible Storage File Resource

As a prerequisite, the remote storage account and “file share” on Azure MUST have been created by the user.

For shared Files, the URI pattern of a resource is the following : https://<storage-account-name>.file.core.windows.net/<share name>/… <0 to n folder name(s)>/<file name>. The first name after the netloc is the “share name” not a simple “folder name”.

By default, this resource reads the configuration from standard location (environment variables for the moment)

exists()¶: Check if the target resource exists (file or directory)

list_dir()¶: List the files (not the directories) of the current directory

Note

This is not a recursive listing operation.

remove()¶: Remove the target resource either it is a file or a directory. If an error occurs an exception is raised.

class khiops.core.internals.filesystems.AzureStorageResourceMixin(uri)¶

Bases: object

Azure compatible Storage Resource Mixin

See AzureStorageFileResource and AzureStorageBlobResource for more details.

property uri_parts¶: Build a list of path parts from a path as a str

class khiops.core.internals.filesystems.FilesystemResource(uri)¶

Bases: ABC

Abstract Filesystem Resource

abstract create_child(file_name)¶

Creates a resource representing a child of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

abstract create_parent()¶

Creates a resource representing the parent of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

class khiops.core.internals.filesystems.GoogleCloudStorageResource(uri)¶

Bases: FilesystemResource

Google Cloud Storage Resource

By default, it reads the configuration from standard location.

create_child(file_name)¶

Creates a resource representing a child of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

create_parent()¶

Creates a resource representing the parent of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

class khiops.core.internals.filesystems.LocalFilesystemResource(uri)¶

Bases: FilesystemResource

A local filesystem resource

create_child(file_name)¶

Creates a resource representing a child of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

create_parent()¶

Creates a resource representing the parent of this instance

Returns:

FilesystemResource: The specific resource type is that of the caller

khiops.core.internals.filesystems.copy_from_local(uri_or_path, local_path)¶

Copies the content of a local file to the resource

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.
local_pathstr: The path of the local file to be copied.

Raises:

RuntimeError: If there was a problem when copying.

khiops.core.internals.filesystems.copy_to_local(uri_or_path, local_path)¶

Copies the content of the resource to a local path

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.
local_pathstr: The path of the local file to be copied.

Raises:

RuntimeError: If there was a problem when copying.

khiops.core.internals.filesystems.create_resource(uri_or_path)¶

Factory method to create a FilesystemResource from an URI

Parameters:

uri_or_pathstr

The resource’s URI . Supported protocols/schemes:

file or empty: Local filesystem resource
s3: Amazon S3 resource
gs: Google Cloud Storage resource
https: Azure Storage resource (files or blobs)

Returns:

FilesystemResource: The URI resource object, its class depends on the URI.

khiops.core.internals.filesystems.exists(uri_or_path)¶

Checks the existence of the resource

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.

Returns:

bool: True if the resource exists.

khiops.core.internals.filesystems.get_child_path(uri_or_path, child_name)¶

Returns the child path of this URI at the specified child name

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.
child_namestr: Name of the child to be added to the URI or path.

Returns:

str: The URI or path of the child resource.

khiops.core.internals.filesystems.get_parent_path(uri_or_path)¶

Returns the parent path of this URI

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.

Returns:

str: The URI or path of the parent’s resource.

khiops.core.internals.filesystems.is_local_resource(uri_or_path)¶

Checks if a URI or path is effectively a local path

Note

An URI with scheme of size 1 will be considered a local path. This is to take into account Windows paths such as C:\Some\Windows\Path.

Returns:

bool: True if a URI refers to a local path

khiops.core.internals.filesystems.list_dir(uri_or_path)¶

Lists a directory resource contents

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.

Returns:

list: A list of paths or URIs.

khiops.core.internals.filesystems.make_dir(uri_or_path)¶

Creates a directory at the resource’s path

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.

khiops.core.internals.filesystems.read(uri_or_path, size=None)¶

Reads data in binary mode from the target resource

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.
sizeint, optional: Number of bytes to read.

Returns:

bytes: A buffer containing the read contents.

Raises:

RuntimeError: If there was a problem when reading.

khiops.core.internals.filesystems.remove(uri_or_path)¶

Removes the resource

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.

Raises:

RuntimeError: If there was a problem when removing.

khiops.core.internals.filesystems.write(uri_or_path, data)¶

Writes data in binary mode to the target resource

Parameters:

uri_or_pathstr: The resource’s URI or local filesystem path.
datastr or bytes: The data to be written.

Raises:

RuntimeError: If there was a problem when writing.