Storage¶

This module provides helpers for working with Google Cloud Storage (GCS), including:

1. A Django Storage class allowing Django's FileField to use GCS as a storage backend. This incorporates the GCS-specific parts of django-storages.
2. A BlobField with an associated widget to facilitate direct uploads and provide more powerful ways of working with GCS features, including metadata and revisions.

The widget provides a better user experience for blankable and overwriting options.

Installation and authentication¶

First, follow the instructions to install, authenticate, and (if necessary) set your project.

Create bucket(s)¶

This library does not create buckets for you: infrastructure operations should be kept separate and dealt with using tools built for the purpose, like Terraform.

If you are setting up for the first time and do not want to get into infrastructure-as-code, manually create two buckets in your project:

• One with object-level permissions for media files.
• One with uniform, public permissions for static files.

<app>-<purpose>-<environment>-<media-or-static>

Set up media and static storage¶

The most common types of storage are for media and static files, using the storage backend. We provide a custom storage class for each, plumbed in via Django's STORAGES setting.

In your settings.py file, do:

STORAGES = {
    "default": {
        "BACKEND": "django_gcp.storage.GoogleCloudMediaStorage",
        "OPTIONS": {
            "bucket_name": "app-assets-environment-media",  # Or whatever name you chose
        },
    },
    "staticfiles": {
        "BACKEND": "django_gcp.storage.GoogleCloudStaticStorage",
        "OPTIONS": {
            "bucket_name": "app-assets-environment-static",  # Or whatever name you chose
        },
    },
}

# Point the urls to the store locations
#   You could customise the base URLs later with your own cdn, eg https://static.you.com
#   But that's only if you feel like being ultra fancy
MEDIA_URL = f"https://storage.googleapis.com/{STORAGES['default']['OPTIONS']['bucket_name']}/"
MEDIA_ROOT = "/media/"
STATIC_URL = f"https://storage.googleapis.com/{STORAGES['staticfiles']['OPTIONS']['bucket_name']}/"
STATIC_ROOT = "/static/"

Default and extra stores¶

Any number of extra stores can be added, each corresponding to a different bucket in GCS. Just add additional entries to the STORAGES dict — each top-level alias becomes a store_key you can use on a BlobField or pass to GoogleCloudStorage:

STORAGES = {
    "default": {
        "BACKEND": "django_gcp.storage.GoogleCloudMediaStorage",
        "OPTIONS": {"bucket_name": "app-assets-environment-media"},
    },
    "staticfiles": {
        "BACKEND": "django_gcp.storage.GoogleCloudStaticStorage",
        "OPTIONS": {"bucket_name": "app-assets-environment-static"},
    },
    "my-fun-store": {
        "BACKEND": "django_gcp.storage.GoogleCloudStorage",
        "OPTIONS": {"bucket_name": "all-the-fun-datafiles"},
    },
    "my-sad-store": {
        "BACKEND": "django_gcp.storage.GoogleCloudStorage",
        "OPTIONS": {"bucket_name": "all-the-sad-datafiles"},
    },
}

Then reference an extra store from a BlobField via its alias:

blob = BlobField(store_key="my-fun-store", get_destination_path=...)

With the STORAGES setting in place, default_storage is your Google Cloud media storage:

>>> from django.core.files.storage import default_storage
>>> print(default_storage.__class__)
<class 'django_gcp.storage.GoogleCloudMediaStorage'>

This way, if you define a new FileField, it will use that storage bucket:

>>> from django.db import models
>>> class MyModel(models.Model):
...     my_file_field = models.FileField(upload_to="pdfs")
...     my_image_field = models.ImageField(upload_to="photos")

BlobField storage¶

The benefit of a BlobField is that you can upload objects directly to the cloud. This allows you to accept uploads of files larger than 32 MB while on request-size-limited services like Cloud Run.

To enable this and other advanced features (like caching of metadata and blob version tracking), BlobFields intentionally do not maintain the FileField API. Under the hood, a BlobField is actually a JSONField, allowing properties other than just the blob name to be stored in the database.

We will flesh out these instructions later (pull requests accepted!), but in the meantime, see the example implementation. You will need to:

1. Add a django_gcp.storage.fields.BlobField field to a model.
2. Define a get_destination_path callback to generate the eventual name of the blob in the store.

Getting BlobField files on the server¶

Because a BlobField uploads directly to the cloud, its file never passes through your server. When you do need the contents server-side — most commonly to validate an uploaded file before saving the model — use the blob_to_temporary_file context manager, which downloads the blob to a temporary file (preserving the file extension) and removes that file on exit:

from django_gcp.storage.blob_utils import blob_to_temporary_file

with blob_to_temporary_file(instance, "my_blob_field") as f:
    if f is not None:  # None when the field is blank
        validate_contents(f)

Before the model instance is saved, the field value refers to the temporary ingress location, and the contents are downloaded from there; after save, they are downloaded from the blob's destination path. To work with the blob object itself rather than a local copy (for example to stream it, inspect its metadata, or generate signed URLs), use the other helpers in django_gcp.storage.blob_utils: get_blob, get_blob_name, get_path, get_signed_url, get_signed_download_url and get_console_url. All of these are also available as methods on models via django_gcp.storage.blob_utils.BlobFieldMixin.

FileField storage¶

The storage classes work as a standard drop-in storage backend. Standard file access options are available and work as expected:

>>> default_storage.exists("storage_test")
False
>>> file = default_storage.open("storage_test", "w")
>>> file.write("storage contents")
>>> file.close()

>>> default_storage.exists("storage_test")
True
>>> file = default_storage.open("storage_test", "r")
>>> file.read()
'storage contents'
>>> file.close()

>>> default_storage.delete("storage_test")
>>> default_storage.exists("storage_test")
False

The same applies to models. An object without a file has limited functionality:

>>> obj1 = MyModel()
>>> obj1.my_file_field
<FieldFile: None>
>>> obj1.my_file_field.size
Traceback (most recent call last):
...
ValueError: The 'my_file_field' attribute has no file associated with it.

Saving a file enables full functionality:

>>> obj1.my_file_field.save("django_test.txt", ContentFile("content"))
>>> obj1.my_file_field
<FieldFile: tests/django_test.txt>
>>> obj1.my_file_field.size
7
>>> obj1.my_file_field.read()
'content'

Files can be read a little at a time, if necessary:

>>> obj1.my_file_field.open()
>>> obj1.my_file_field.read(3)
'con'
>>> obj1.my_file_field.read()
'tent'
>>> "-".join(obj1.my_file_field.chunks(chunk_size=2))
'co-nt-en-t'

Saving another file with the same name appends extra characters (see the file_overwrite option below):

>>> obj2 = MyModel()
>>> obj2.my_file_field.save("django_test.txt", ContentFile("more content"))
>>> obj2.my_file_field
<FieldFile: tests/django_test_.txt>
>>> obj2.my_file_field.size
12

Controlling generated URLs¶

Keyword arguments passed to url() are forwarded to the underlying Blob.generate_signed_url call, giving control over the signed URL. For example, pass response_disposition="attachment" to make the browser download the file instead of displaying it inline:

>>> default_storage.url("storage_test", response_disposition="attachment")
'https://storage.googleapis.com/test-media/storage_test?...&response-content-disposition=attachment'

The store settings supply defaults for expiration, version and (where a custom endpoint is configured) bucket_bound_hostname; passing any of those keywords explicitly overrides the store default for that call. These keyword arguments only apply to signed URLs; they are ignored for stores that return public (unsigned) URLs (see the default_acl and querystring_auth options below).

For models using a BlobField, the helper functions in django_gcp.storage.blob_utils (for example get_signed_download_url) provide the same capability from a model instance.

Storage settings options¶

Each store can be set up with different options, passed via the OPTIONS dict for that alias in the STORAGES setting. For example, to set the media storage up so that files go to a location other than the root of the bucket:

STORAGES = {
    "default": {
        "BACKEND": "django_gcp.storage.GoogleCloudMediaStorage",
        "OPTIONS": {
            "bucket_name": "app-assets-environment-media",
            "location": "not/the/bucket/root/",
            # ... and whatever other options you want
        },
    },
    # ... other aliases ...
}

The full range of options (and their defaults) is as follows. These per-store options, and the library's other settings, are indexed in the settings reference.

project_id¶

Type: string or None

Default: None

The project this store's bucket belongs to, overriding the root-level GCP_PROJECT_ID setting for this store. If neither is set, the project is inferred from the credentials.

credentials¶

Type: a google.auth credentials object, or None

Default: None

An explicit credentials object for this store, overriding the root-level GCP_CREDENTIALS setting. Usually left unset in favour of environment-based authentication (see Authenticating the server).

gzip¶

Type: boolean

Default: False

Whether to enable gzipping of the content types specified by gzip_content_types.

gzip_content_types¶

Type: tuple

Default: ("text/css", "text/javascript", "application/javascript", "application/x-javascript", "image/svg+xml")

Content types which will be gzipped when gzip is True.

default_acl¶

Type: string or None

Default: None

ACL used when creating a new blob, from the list of predefined ACLs. (A "JSON API" ACL is preferred, but an "XML API/gsutil" ACL will be translated.)

For most cases, the blob needs the publicRead ACL for the file to be viewable. If default_acl is not set, the blob has the default permissions set by the bucket.

publicRead files return a public, non-expiring URL. All other files return a signed (expiring) URL. ACL options are: projectPrivate, bucketOwnerRead, bucketOwnerFullControl, private, authenticatedRead, publicRead, publicReadWrite.

querystring_auth¶

Type: boolean

Default: True

If set to False, forces URLs not to be signed. This setting is useful if your bucket is configured with uniform access control and public read: in that case, set querystring_auth = False and default_acl = None.

file_overwrite¶

Type: boolean

Default: True

By default, files with the same name overwrite each other. Set this to False to have extra characters appended to new files with duplicate names.

max_memory_size¶

Type: integer

Default: 0 (do not roll over)

The maximum amount of memory a returned file can take up (in bytes) before being rolled over into a temporary file on disk.

blob_chunk_size¶

Type: integer or None

Default: None

The size of blob chunks that are sent via resumable upload. If this is not set, the generated request must fit in memory. Recommended if you are going to be uploading large files.

object_parameters¶

Type: dict

Default: {}

Dictionary of key-value pairs mapping from blob property name to value. Use this to set parameters on all objects. To set them on a per-object basis, subclass the backend and override GoogleCloudStorage.get_object_parameters.

The valid property names are: acl, cache_control, content_disposition, content_encoding, content_language, content_type, metadata, storage_class.

If not set, the content_type property will be guessed. If set, acl overrides default_acl.

custom_endpoint¶

Type: string or None

Default: None

Sets a custom endpoint used instead of https://storage.googleapis.com when generating URLs for files.

location¶

Type: string

Default: ""

Subdirectory in which the files will be stored. Defaults to the root of the bucket.

expiration¶

Type: datetime.timedelta, datetime.datetime, or integer (seconds since epoch)

Default: timedelta(seconds=86400)

The time a generated URL is valid before expiration; the default is one day. Public files return a URL that does not expire. Files are signed by the credentials provided during authentication.

The value is handled by the underlying Google library, which supports timedelta, datetime, or integer seconds since epoch.

BlobField settings¶

Unlike the per-store options above, the following are root-level Django settings governing BlobField behaviour across all stores.

GCP_STORAGE_OVERRIDE_BLOBFIELD_VALUE¶

Type: boolean

Default: False

When True, values assigned to a BlobField are stored directly, bypassing the ingress (upload-then-move) process. This is intended for migrations and programmatic operations where the blob is already in its destination location — for example, the operations in django_gcp.storage.operations toggle it with override_settings while uploading blobs directly. Leave it False in normal operation.

GCP_STORAGE_BLOBFIELD_MAX_SIZE_BYTES¶

Type: integer

Default: unlimited

The maximum size, in bytes, of a file accepted by a BlobField direct upload.

Test-only callback overrides¶

GCP_STORAGE_OVERRIDE_GET_DESTINATION_PATH_CALLBACK and GCP_STORAGE_OVERRIDE_UPDATE_ATTRIBUTES_CALLBACK (both default None) replace every BlobField's get_destination_path and update_attributes callbacks respectively. They exist because patching a field's callback in a test framework is a struggle; do not use them outside tests.

Cleaning up temporary uploads¶

When direct uploads are ingressed to a temporary location but the corresponding model save fails, an orphaned upload is left behind. The cleanup_tmp_files management command lists temporary blobs older than 24 hours in a given store, and deletes them if passed the --delete flag:

python manage.py cleanup_tmp_files default --delete

Configuration in 0.24 and below¶

Prior to django-gcp 0.25, storage was configured via Django's deprecated DEFAULT_FILE_STORAGE and STATICFILES_STORAGE settings, with options held in three separate dicts: GCP_STORAGE_MEDIA, GCP_STORAGE_STATIC, and GCP_STORAGE_EXTRA_STORES. BlobField used store_key="media" and store_key="static" instead of "default"/"staticfiles".

That configuration looked like this:

# Set the default storage (for media files)
DEFAULT_FILE_STORAGE = "django_gcp.storage.GoogleCloudMediaStorage"
GCP_STORAGE_MEDIA = {
    "bucket_name": "app-assets-environment-media"  # Or whatever name you chose
}

# Set the static file storage
#   This allows `manage.py collectstatic` to automatically upload your static files
STATICFILES_STORAGE = "django_gcp.storage.GoogleCloudStaticStorage"
GCP_STORAGE_STATIC = {
    "bucket_name": "app-assets-environment-static"  # Or whatever name you chose
}

GCP_STORAGE_EXTRA_STORES = {
    "my_fun_store_key": {"bucket_name": "all-the-fun-datafiles"},
    "my_sad_store_key": {"bucket_name": "all-the-sad-datafiles"},
}

MEDIA_URL = f"https://storage.googleapis.com/{GCP_STORAGE_MEDIA['bucket_name']}/"
MEDIA_ROOT = "/media/"
STATIC_URL = f"https://storage.googleapis.com/{GCP_STORAGE_STATIC['bucket_name']}/"
STATIC_ROOT = "/static/"

To migrate to 0.25+:

1. Replace the legacy settings above with a single STORAGES dict (see Set up media and static storage). Each former extra-stores entry becomes a top-level alias under STORAGES — there is no separate "extra" wrapper anymore.
2. In every model, rename BlobField(store_key="media") to BlobField(store_key="default") and store_key="static" to store_key="staticfiles". Any custom store keys keep their names.
3. Run python manage.py makemigrations to capture the store_key change as an AlterField migration for each affected BlobField.
4. If you use the manage.py cleanup_tmp_files management command, update the positional store_key argument to the new alias name.