2.3. Projects and Containers

Projects (DXProject) are platform entities that serve as general-purpose containers for data, and are the unit of collaboration.

Containers (DXContainer) are special-purpose data containers that support a subset of the methods available to projects. Containers behave like projects with the PROTECTED flag unset (so that temporary intermediate data files can be deleted), except they cannot be explicitly created or destroyed, and their permissions are fixed.

DXProject is implemented as a subclass of DXContainer.

class dxpy.bindings.dxproject.DXContainer(dxid=None)[source]

Bases: DXObject

Remote container handler.

describe(**kwargs)[source]
Returns:

A hash containing attributes of the project or container.

Return type:

dict

Returns a hash with key-value pairs as specified by the API specification for the /project-xxxx/describe method. This will usually include keys such as “id”, “name”, “class”, “billTo”, “created”, “modified”, and “dataUsage”.

new_folder(folder, parents=False, **kwargs)[source]
Parameters:

  • folder (string) – Full path to the new folder to create

  • parents (boolean) – If True, recursively create any parent folders that are missing

Creates a new folder in the project or container.

list_folder(folder='/', describe=False, only='all', includeHidden=False, **kwargs)[source]
Parameters:

  • folder (string) – Full path to the folder to list

  • describe (bool or dict) – If True, returns the output of /describe on each object (see below for notes)

  • only (string) – Indicate “objects” for only objects, “folders” for only folders, or “all” for both

  • includeHidden (bool) – Indicate whether hidden objects should be returned

Returns:

A hash with key “objects” for the list of object IDs and key “folders” for the list of folder routes

Return type:

dict

Returns a hash containing a list of objects that reside directly inside the specified folder, and a list of strings representing the full paths to folders that reside directly inside the specified folder.

By default, the list of objects is provided as a list containing one hash {"id": "class-XXXX"} with the ID of each matching object. If describe is not False, the output of /describe is also included in an additional field “describe” for each object. If describe is True, /describe is called with the default arguments. describe may also be a hash, indicating the input hash to be supplied to each /describe call.

move(destination, objects=[], folders=[], **kwargs)[source]
Parameters:

  • destination (string) – Path of destination folder

  • objects (list of strings) – List of object IDs to move

  • folders (list of strings) – List of full paths to folders to move

Moves the specified objects and folders into the folder represented by destination. Moving a folder also moves all contained folders and objects. If an object or folder is explicitly specified but also appears inside another specified folder, it will be removed from its parent folder and placed directly in destination.

move_folder(folder, destination, **kwargs)[source]
Parameters:

  • folder (string) – Full path to the folder to move

  • destination (string) – Full path to the destination folder that will contain folder

Moves folder to reside in destination in the same project or container. All objects and subfolders inside folder are also moved.

remove_folder(folder, recurse=False, force=False, **kwargs)[source]
Parameters:

  • folder (string) – Full path to the folder to remove

  • recurse (bool) – If True, recursively remove all objects and subfolders in the folder

  • force (bool) – If True, will suppress errors for folders that do not exist

Removes the specified folder from the project or container. It must be empty to be removed, unless recurse is True.

Removal propagates to any hidden objects that become unreachable from any visible object in the same project or container as a result of this operation. (This can only happen if recurse is True.)

remove_objects(objects, force=False, **kwargs)[source]
Parameters:

  • objects (list of strings) – List of object IDs to remove from the project or container

  • force (bool) – If True, will suppress errors for objects that do not exist

Removes the specified objects from the project or container.

Removal propagates to any hidden objects that become unreachable from any visible object in the same project or container as a result of this operation.

clone(container, destination='/', objects=[], folders=[], parents=False, **kwargs)[source]
Parameters:

  • container (string) – Destination container ID

  • destination (string) – Path of destination folder in the destination container

  • objects (list of strings) – List of object IDs to move

  • folders (list of strings) – List of full paths to folders to move

  • parents (boolean) – Whether the destination folder and/or parent folders should be created if they do not exist

Clones (copies) the specified objects and folders in the container into the folder destination in the container container. Cloning a folder also clones all all folders and objects it contains. If an object or folder is explicitly specified but also appears inside another specified folder, it will be removed from its parent folder and placed directly in destination. No objects or folders are modified in the source container.

Objects must be in the “closed” state to be cloned.

class dxpy.bindings.dxproject.DXProject(dxid=None)[source]

Bases: DXContainer

Remote project handler.

new(name, summary=None, description=None, region=None, protected=None, restricted=None, download_restricted=None, contains_phi=None, tags=None, properties=None, bill_to=None, database_ui_view_only=None, external_upload_restricted=None, default_symlink=None, **kwargs)[source]
Parameters:

  • name (string) – The name of the project

  • summary (string) – If provided, a short summary of what the project contains

  • description – If provided, the new project description

  • region – If provided, the region that this project will be created in. The region must be among the permitted regions of the project’s billTo

  • protected (boolean) – If provided, whether the project should be protected

  • restricted (boolean) – If provided, whether the project should be restricted

  • download_restricted (boolean) – If provided, whether external file downloads and external access to database objects should be restricted

  • contains_phi (boolean) – If provided, whether the project should be marked as containing protected health information (PHI)

  • tags (list of strings) – If provided, tags to associate with the project

  • properties (dict) – If provided, properties to associate with the project

  • bill_to (string) – If provided, ID of the entity to which any costs associated with this project will be billed; must be the ID of the requesting user or an org of which the requesting user is a member with allowBillableActivities permission

  • database_ui_view_only (boolean) – If provided, whether the viewers on the project can access the database data directly

  • external_upload_restricted (boolean) – If provided, whether project members can upload data to project from external sources, e.g. outside of job

  • default_symlink – If provided, the details needed to have writable symlinks in the project. Dict must include drive, container, and optional prefix.

: type default_symlink: dict

Creates a new project. Initially only the user performing this action will be in the permissions/member list, with ADMINISTER access. See the API documentation for the /project/new method for more info.

update(name=None, summary=None, description=None, protected=None, restricted=None, download_restricted=None, version=None, allowed_executables=None, unset_allowed_executables=None, database_ui_view_only=None, external_upload_restricted=None, **kwargs)[source]
Parameters:

  • name (string) – If provided, the new project name

  • summary (string) – If provided, the new project summary

  • description – If provided, the new project description

  • protected (boolean) – If provided, whether the project should be protected

  • restricted (boolean) – If provided, whether the project should be restricted

  • download_restricted (boolean) – If provided, whether external downloads should be restricted

  • allowed_executables (list) – If provided, these are the only executable ID(s) allowed to run as root executions in this project

  • database_ui_view_only (boolean) – If provided, whether the viewers on the project can access the database data directly

  • external_upload_restricted (boolean) – If provided, whether project members can upload data to project from external sources, e.g. outside of job

  • version (int) – If provided, the update will only occur if the value matches the current project’s version number

Updates the project with the new fields. All fields are optional. Fields that are not provided are not changed. See the API documentation for the /project-xxxx/update method for more info.

invite(invitee, level, send_email=True, **kwargs)[source]
Parameters:

  • invitee (string) – Username (of the form “user-USERNAME”) or email address of person to be invited to the project; use “PUBLIC” to make the project publicly available (in which case level must be set to “VIEW”).

  • level (string) – Permissions level that the invitee would get (“VIEW”, “UPLOAD”, “CONTRIBUTE”, or “ADMINISTER”)

  • send_email (boolean) – Determines whether user receives email notifications regarding the project invitation

Invites the specified user to have access to the project.

decrease_perms(member, level, **kwargs)[source]
Parameters:

  • member (string) – Username (of the form “user-USERNAME”) of the project member whose permissions will be decreased.

  • level (string or None) – Permissions level that the member will have after this operation (None, “VIEW”, “UPLOAD”, or “CONTRIBUTE”)

Decreases the permissions that the specified user has in the project.

destroy(**kwargs)[source]

Destroys the project.

set_properties(properties, **kwargs)[source]
Parameters:

properties (dict) – Property names and values given as key-value pairs of strings

Given key-value pairs in properties for property names and values, the properties are set on the project for the given property names. Any property with a value of None indicates the property will be deleted.

Note

Any existing properties not mentioned in properties are not modified by this method.

2.3.1. Accessing App-Specific Containers

Apps have associated resource containers and project cache containers. To easily access these, the following utility functions are provided. These functions are meant to be called only by a job.

dxpy.bindings.dxapp_container_functions.load_app_resource(**kwargs)[source]
Parameters:

kwargs – keyword args for find_one_data_object(), with the exception of “project”

Raises:

DXError if “project” is given, if this is called with dxpy.JOB_ID not set, or if “DX_RESOURCES_ID” or “DX_PROJECT_CONTEXT_ID” is not found in the environment variables

Returns:

None if no matching object is found; otherwise returns a dxpy object handler for that class of object

Searches for a data object in the app resources container matching the given keyword arguments. If found, the object will be cloned into the running job’s workspace container, and the handler for it will be returned. If the app resources container ID is not found in DX_RESOURCES_ID, falls back to looking in the current project.

Example:

@dxpy.entry_point('main')
def main(*args, **kwargs):
    x = load_app_resource(name="Indexed genome", classname='file')
    dxpy.download_dxfile(x)
dxpy.bindings.dxapp_container_functions.load_from_cache(**kwargs)[source]
Parameters:

kwargs – keyword args for find_one_data_object(), with the exception of “project”

Raises:

DXError if “project” is given, if this is called with dxpy.JOB_ID not set, or if “DX_PROJECT_CACHE_ID” is not found in the environment variables

Returns:

None if no matching object is found; otherwise returns a dxpy object handler for that class of object

Searches for a data object in the project cache container matching the given keyword arguments. If found, the object will be cloned into the running job’s workspace container, and the handler for it will be returned.

Example:

@dxpy.entry_point('main')
def main(*args, **kwargs):
    x = load_from_cache(name="Indexed genome", classname='file')
    if x is None:
        x = compute_result(*args)
        save_to_cache(x)
dxpy.bindings.dxapp_container_functions.save_to_cache(dxobject)[source]
Parameters:

dxobject – a dxpy object handler for an object to save to the cache

Raises:

DXError if this is called with dxpy.JOB_ID not set, or if “DX_PROJECT_CACHE_ID” is not found in the environment variables

Clones the given object to the project cache.

Example:

@dxpy.entry_point('main')
def main(*args, **kwargs):
    x = load_from_cache(name="Indexed genome", classname='file')
    if x is None:
        x = compute_result(*args)
        save_to_cache(x)