Skip to main content

Storage Examples

This page describes storage examples using the StorageManager class. The storage examples include:

note

StorageManager supports http(s), S3, Google Cloud Storage, Azure, and file system folders.

StorageManager#

Downloading a File#

To download a ZIP file from storage to the global cache context, call the StorageManager.get_local_copy method, and specify the destination location as the remote_url argument:

# create a StorageManager instance
manager = StorageManager()
manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.zip")
note

Zip and tar.gz files will be automatically extracted to cache. This can be controlled with theextract_archive flag.

To download a file to a specific context in cache, specify the name of the context as the cache_context argument:

manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", cache_context="test")

To download a non-compressed file, set the extract_archive argument to False.

manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", extract_archive=False)

Uploading a File#

To upload a file to storage, call the StorageManager.upload_file method. Specify the full path of the local file as the local_file argument, and the remote URL as the remote_url argument.

manager.upload_file(local_file="/mnt/data/also_file.ext", remote_url="s3://MyBucket/MyFolder")

Setting Cache Limits#

To set a limit on the number of files cached, call the StorageManager.set_cache_file_limit method and specify the cache_file_limit argument as the maximum number of files. This does not limit the cache size, only the number of files.

new_cache_limit = manager.set_cache_file_limit(cache_file_limit=100)