IDA User Guide

This user guide is about the renewed IDA service, which will be available to end users later in summer 2018. The user instructions for the current IDA service can be found here: openscience.fi/ida-user-instructions

If you have any questions about particular features, or any other issue relating to the service, please contact IDA support at servicedesk (a) csc.fi

Hate reading manuals? Too busy to read the full user guide? Try the IDA Quick Start Guide

This guide covers the web UI of the renewed IDA service. Information about IDA’s command line tools will be soon available in GitHub repository. For the time being, please see the draft guide for the IDA command line tools.


Overview

Welcome to the IDA service!

IDA is a research data storage service organized by the Finnish Ministry of Education and Culture and produced by CSC — IT Center for Science.

The service is intended for stable research data, both raw data and processed data, and is offered free of charge to users affiliated with Finnish universities and polytechnics, Finnish research institutes and research funded by the Academy of Finland.

This guide provides an overview of the features and functions of the service, organized by major functional area and view. If you are new to the service, it is recommended that you read this guide fully before using the service.

Project Data Storage

Each project in IDA has an allocated quota of storage space. This storage space is divided into two distinct areas: a Staging area (read/write) and a Frozen area (read-only). Both areas share the same project quota, so there is no need to manage or reserve space for each use separately.

All project members have equal rights to add, remove, and operate on project data stored in the IDA service.

Users of the service may belong to one or more projects with which they collaborate.

In both areas project members can:

  • view directories, files and file sizes
  • download files and directories to their own computer
  • share files and directories outside the service via a temporary share link
  • delete files and directories

The staging area is intended for collecting and organizing research data in preparation for storage in the frozen area.

In the staging area:

  • files and folders can be added, rearranged, renamed, overwritten, and removed
  • file metadata are not saved in the metadata repository nor visible to other services
  • files are not replicated, so storing in the staging area doesn’t prevent data loss in case of a major hardware failure (single disk failure will not cause data loss)

The frozen area is intended for storing stable research data in an immutable state.

In the frozen area:

  • files are given unique identifiers, checksums are generated, and file metadata stored in a metadata repository
  • files are replicated on another media to prevent data loss in case of a major hardware failure
  • files are visible and accessible to other data preservation services, such as for describing datasets and for the digital preservation solution for research data

Data should be moved to the frozen area once no further changes are expected to be made. Data in the frozen area is available to other services supporting digital preservation and they can be included as a part of user-defined datasets using these other services.

All project members can move data from the staging area to the frozen area (as described below). Because of the procedures for data preservation, data in the frozen area can not be re-arranged, renamed or overwritten. Frozen data can moved back to the staging area or deleted entirely from the service, but users should be aware of the effects of these actions. If data is moved back to the staging area or deleted, the unique file identifiers and file metadata will be discarded. This means that any references to those files within any defined datasets and download links to data will become permanently invalid.

Please note that IDA’s frozen area is not digital preservation. The Digital Preservation Solution for Research Data (TPAS) is a separate service.

Project’s staging area Project’s frozen area
Intended use Collecting and organizing research data in preparation for frozen storage. Data can be added, moved, renamed, and deleted freely by all project members. Storing stable research data in an immutable state. Data can only be unfrozen or deleted. Data is available to other services supporting digital preservation.
Users can upload, rearrange, overwrite files and directories
Users can download and delete files/directories
Users can share data with temporary links
File metadata saved in metadata repository
Checksum calculated for files
Files have unique identifiers
Files replicated on another media (recovery from major hardware failure)

Getting started

Logging in

If you are not an IDA user yet, read more about the service at: Becoming an IDA user

You may log in to the IDA service using either the HAKA authentication service or using a personal CSC user account.

To log in to the IDA service using HAKA, click on the HAKA logo in the top left region of the log in screen and follow the instructions provided.

To log in to the IDA service using an personal CSC user account, enter your CSC user account name and password in the fields provided on the left hand side of the log in screen and click the Login button.

Note that personal CSC user accounts are created and managed via the SUI CSC Customer Portal, including changing or resetting your account password.

Upon successfully logging in to the IDA service, you will be presented with the top level files view, described below.

Logging out

To log out of the IDA service, click on the gear icon located in the top right corner of the view, which will open a pull-down menu, and select the “Logout” menu option.

Language settings

To change the language of the web UI, click on the gear icon located in the top right corner of the view, which will open a pull-down menu, and select the “Personal” menu option.

Locate and open the “Language” pull-down menu on the personal settings page, and select your desired language. The language of the web UI will change immediately after selection.


Files View

When you first log in to the IDA service, you will be presented with a file browser view showing the staging and frozen folders for all projects to which you have access. The name of the frozen folder for a project will correspond to the project name. The name of the staging folder for a project will correspond to the project name, followed by a plus “+” character. The plus character in the staging folder name serves as a reminder that new data is added to the staging area of the project.

The contents of a folder can be viewed by clicking on the name of the folder in the listing.

To navigate back to a parent folder, either use your web browser’s “Back” functionality, or click on the desired ancestor folder in the pathname above the listing.

To return to the top root of the files listing from a subfolder, click on the home icon at the far left of the pathname above the listing.

To return to the files view from any other view, click on the folder icon to the right of the IDA logo at the top left of the view.

Staging Area

New files are uploaded to the staging area of a project, which is fully contained within the staging folder of the project.

Files can be uploaded directly into the root of the staging folder; however, it is strongly recommended that files be organized by subfolders within the staging folder.

Organizing files by e.g. time period and other logical sructures can help make management of large numbers of data files easier, and freezing of data easier and more efficient, by selecting and freezing all files within a folder (see below about freezing files).

Learn more about best practices in naming and and structuring your data: research.csc.fi/files-and-file-formats

Creating Folders

To create a new folder, click on the “+” icon at the top of the view, to the right of the current folder path, and select the option “New folder”. Enter a name for the new folder and press “Return”.

Uploading Individual Files

To upload an individual file to the current folder, click on the “+” icon at the top of the view, to the right of the current folder path, and select the option “Upload file”, and using the file picker presented, navigate to and select a file on your local filesystem.

Uploading Folders and Multiple Files using Drag-and-Drop

It is possible to upload entire folders (and their subfolders) and/or multiple files at the same time by using drag-and-drop.

First navigate in the files view to the folder to which you wish to upload the data, creating new subfolders as required.

Then, open your local filesystem browser, select the desired folders and/or files to upload, and drag-and-drop them into the web browser window where the IDA service is open.

The selected folders and files will then be uploaded to the current folder in the files view, preserving the folder organization as stored on the local filesystem.

Moving Files or Folders

Files and folders within the staging area of a project may be freely moved elsewhere in the staging area.

To move a file or folder, navigate in the files view to where the file or folder is located and click on the ellipsis “…” menu to the right of the file or folder name, and select the option “Move”.

In the pop-up window presented, navigate to the desired target folder within the staging area and click “Choose”.

Renaming Files or Folders

Files and folders within the staging area of a project may be freely renamed.

To rename a file or folder, navigate in the files view to where the file or folder is located and click on the ellipsis “…” menu to the right of the file or folder name, and select the option “Rename”.

Edit the name as desired and press “Return”.

Downloading Files or Folders

Both individual files and entire folders can be easily downloaded from the IDA service. When downloading folders, the contents of the folder will be packaged for downloading as a single packaged archive file, which can then be unpacked after download.

To download a file or folder, navigate in the files view to where the file or folder is located and click on the ellipsis “…” menu to the right of the file or folder name, and select the option “Download”.

Follow your browser’s instructions regarding how and where to save the file on your local machine.

Deleting Files or Folders

Files and folders within the staging area of a project may be freely deleted.

To delete a file or folder (and all of its content), navigate in the files view to where the file or folder is located and click on the ellipsis “…” menu to the right of the file or folder name, and select the option “Delete”.

In the confirmation dialog presented, click “OK”.

WARNING: The delete action cannot be undone and the deleted data will be permanently and irretrievably removed from the IDA service!

Freezing Files

When files reach sufficient maturity such that they are considered final and should no longer change, they may be frozen, as described in detail below, which will render them immutable and also make them visible to other data preservation services, such as for inclusion in a published dataset.

Both individual files and entire folders can be selected for freezing. When selecting a folder, all files within that folder, and any subfolders, will be frozen.

Freezing will move each selected file to the frozen area, preserving its relative pathname within the staging area, and making them immutable. The freeze action cannot be terminated before it is complete. Depending on the size and number of files being frozen, the background operations may take several hours. The status of the pending freeze action can be viewed in the pending actions listing as described below.

Frozen files are replicated to separate physical storage to guard against loss of data due to hardware failure. Replication employs checksum validation to ensure the integrity of the frozen file copies. If frozen files are later unfrozen or deleted, all replicated copies will be removed.

To freeze a file or all files within the scope of a folder, navigate in the files view to the location in the staging area of the file or folder and click on the snowflake icon to the right of the file or folder name, which will open the action tab on the right side of the view. Then, in the action tab, click on the button “Freeze”.

In the confirmation dialog presented, if you are absolutely sure you wish to freeze the selected file(s), click “Yes”.

Once the freeze action has been successfully initiated, you will be presented with the option to remain in the file view where you initiated the action, open the file view in the frozen area where the newly frozen files are now located, or by clicking on the pending action link shown in the dialog, you may open the action summary (described in greater detail below).

Note that the freeze action will be rejected if any pathname of any of the files to be frozen conflicts with either an already frozen file, or a file belonging to a pending action. This is to ensure against corruption of already frozen (immutable) files, and their associated metadata and identifiers.

If you wish to freeze a file which shares the same relative pathname as a frozen file, you must either rename the newer file or move it to a different location (pathname) within the staging area, or you must unfreeze or delete the previously frozen file (described further below).

If you wish to freeze a file which shares the same relative pathname as a file in a pending action, you must first wait for the pending action to be completed and then try freezing the file again.

Frozen Area

Frozen files are located in the frozen area of a project, which is fully contained within the frozen folder of the project.

Frozen files are immutable, and may not be freely updated, renamed, moved, or otherwise modified the same as files in the staging area. All that may be done to frozen files is to download them, unfreeze them, or delete them entirely from the service; where the latter two options will result in their identity and metadata becoming invalid and no longer available to other data preservation services.

NOTE: Files are not stored persistently in the IDA service until they are frozen.

Frozen File Details

Frozen files are assigned persistent identifiers, have their checksums generated, their details stored as metadata in a centralized repository and they are replicated to separate physical storage.

To view the details of a particular file, navigate in the files view to where the file is located and click on the snowflake icon to the right of the file name, which will open the action tab on the right side of the view. The file details will be shown at the top of the action tab.

Note that if the file is part of a pending action, not all details will be visible, as they may not have been generated yet by the ongoing background processing.

Checksums for frozen files are generated automatically by the service, using the SHA256 algorithm, and are displayed as “sha256:” URIs in hexidecimal form.

After downloading a frozen file, a local checksum can be generated for comparison with the checksum displayed, using the following command:

 sha256sum  | awk '{print $1}' | tr '[A-Z]' '[a-z]'

 

Unfreezing Frozen Files

If one or more frozen files need to be modified in any way, including updating, renaming, or moving, it is possible to unfreeze both individial files as well as all files within the scope of a particular folder.

Unfreezing will move each selected file back to the staging area, preserving its relative pathname within the frozen area, and making them again fully editable. The unfreeze action cannot be terminated before it is complete. Depending on the size and number of files being unfrozen, the background operations may take several hours. The status of the pending unfreeze action can be viewed in the pending actions listing as described below.

WARNING: Unfrozen files will no longer be accessible to other services, making all external references to them invalid. If frozen files are unfrozen, all replicated copies will be removed. The unfreeze action, once initiated, cannot be undone!

To unfreeze a file or all files within the scope of a folder, navigate in the files view to the location in the frozen area of the file or folder and click on the snowflake icon to the right of the file or folder name, which will open the action tab on the right side of the view. Then, in the action tab, click on the button “Unfreeze”.

In the confirmation dialog presented, if you are absolutely sure you wish to unfreeze the selected file(s), click “Yes”.

Once the unfreeze action has been successfully initiated, you will be presented with the option to remain in the file view where you initiated the action, open the file view in the staging area where the newly unfrozen files are now located, or by clicking on the pending action link shown in the dialog, you may open the action summary (described in greater detail below).

Note that the unfreeze action will be rejected if any pathname of any of the files to be unfrozen conflicts with either any file in the staging area, or a file belonging to a pending action. This is to ensure against corruption of any files files in staging, and to permit all essential background processing to finish before like named files are refrozen, to avoid potential confusion or disruption in other services which may be utilizing frozen file metadata in some manner.

If you wish to unfreeze a file which shares the same relative pathname as a file in staging, you must either rename the file in staging or move it to a different location (pathname) within the staging area, or you must delete the file in staging.

If you wish to unfreeze a file which shares the same relative pathname as a file in a pending action, you must first wait for the pending action to be completed and then try unfreezing the file again.

Deleting Frozen Files

If one or more frozen files need to be permanently removed from the IDA service, it is possible to delete both individual frozen files as well as all frozen files within the scope of a particular folder.

The delete action cannot be terminated before it is complete. Depending on the size and number of frozen files being deleted, the background operations may take several hours. The status of the pending delete action can be viewed in the pending actions listing as described below.

WARNING: Deleted files will be permanently and irretrievably removed from the IDA service, and will no longer be accessible to other services, making all external references to them invalid. If frozen files are deleted, all replicated copies will be removed. The delete action, once initiated, cannot be undone!

To delete a frozen file or all frozen files within the scope of a folder, navigate in the files view to the location in the frozen area of the file or folder and click on the snowflake icon to the right of the file or folder name, which will open the action tab on the right side of the view. Then, in the action tab, click on the button “Delete”.

In the confirmation dialog presented, if you are absolutely sure you wish to delete the selected file(s), click “Yes”.

Once the delete action has been successfully initiated, you will be presented with a confirmation, and by clicking on the pending action link shown in the dialog, you may open the action summary (described in greater detail below).

Note that the delete action will be rejected if any pathname of any of the files to be unfrozen conflicts with a file belonging to a pending action. This is to permit all essential background processing to finish before like named files are deleted, to avoid potential confusion or disruption in other services which may be utilizing frozen file metadata in some manner.

If you wish to delete a file which shares the same relative pathname as a file in a pending action, you must first wait for the pending action to be completed and then try deleting the file again.

You may create temporary share links to either individual files or entire folders, which will be valid for a maxiumum of 30 days, and may optionally be assigned a required password to access. Anyone provided the link, including those who do not belong to any project nor have any IDA account will be able to download the particular file or folder, but will not be able to modify or upload any data.

Temporary links may be created for files or folders in either the staging or frozen area.

This is useful for providing temporary access to data, either during final review of data in the staging area, prior to freezing, or data in the frozen area prior to inclusion in and publication of a dataset.

WARNING: Temporary share links should not be published nor used in citations or as public download links! They are intended for limited, short-term, unofficial use only.

To create a temporary share link for a file or folder, navigate in the files view to the location of the file or folder and click on the share icon to the right of the file or folder name, which will open the sharing tab on the right side of the view. Then, in the sharing tab, click the checkbox “Temporary share link”.

After checking the “Temporary share link” checkbox, the temporary share link is created and displayed in the field immediately below the checkbox. Simply copy and provide the link to whomever you wish to allow temporary access to the data via that link.

To optionally require a password to access the data, click the “Password protect” checkbox and enter the desired password in the field provided, and press “Return”.

To shorten the period of time during which the link is valid, which defaults to 30 days, click on the expiration date field and select a new expiration date in the calendar dialog presented.

To disable an existing temporary share link, uncheck the “Temporary share link” checkbox. The link will be immediately deactivated and discarded.

WARNING: There is no confirmation before disabling a temporary share link. Disabling a temporary share link cannot be undone and the previous share link cannot be retrieved. If you still desire the file or folder to have a temporary share link, a new share link will need to be created by checking the option, and provided to all persons who should use the link.

Note that temporary share links are specific to each IDA user, and cannot be seen nor modified by other project members. It is the responsibility of each project member to inform other project members about, and coordinate the creation and management of, temporary share links, and to observe any organization and/or project-specific policies concerning such links.

Favorites

Files or folders which are frequently in use can be marked as “favorites” so that they can be accessed more easily and directly than browsing to their location in the files view.

Favorite files and folders are easily accessed by selecting the “Favorites” view from the left hand navigation of the files view.

To mark a file or folder as “favorite”, click the star icon to the left of the file or folder name. The star will become highlighted and the file or folder will be included in the “Favorites” view.

To remove a file or folder from the “Favorites” view, click the highlighted star icon to the left of the file or folder name.

Hidden Files

To make hidden files visible, open the “Settings” menu located in the bottom left corner of the “Files” view and select the option “Show hidden files”. To hide hidden files, unselect the option.

 


Action Listings

The IDA service performs numerous background operations to maximize the responsiveness of the service and the efficiency of processes, and ultimately minimize the amount of time users need to spend interacting with the service.

The IDA service provides easy access to listings of actions, ordered by most recently initiated to least recently initiated, and organized by status as described below.

To view all available action listings, click on “ACTIONS” in the top navigation bar.

The default action listing will be displayed, showing all pending actions.

To view the details of a particular action shown in a particular listing, click on the action identifier link. The time at which each individual background operation was completed will be shown in the top right of the summary. If the action is a failed action, the cause for the failure will be clearly indicated in red. The files associated with the action will be listed in the lower part of the summary.

Pending Actions

Once initiated, an action is pending until all background operations are completed, or some unrecoverable error is encountered. As individual operations are completed, the status of the pending action is updated accordingly, and can be monitored by regularly updating the summary view for the action.

To view all pending actions, open the actions listing section by clicking on “ACTIONS” in the top navigation bar as described above, or if already in the action listings section and another action status listing is shown, click “Pending Actions” in the left hand navigation.

Completed Actions

After all background operations have been completed successfully, an action is complete.

To view all completed actions, open the actions listing section by clicking on “ACTIONS” in the top navigation bar as described above, if not already in the action listings section, and click “Completed Actions” in the left hand navigation.

Failed Actions

If any background operation fails to complete successfully, the action is flagged as a failed action, and the cause of the failure will be clearly indicated in the action summary.

To view all failed actions, open the actions listing section by clicking on “ACTIONS” in the top navigation bar as described above, if not already in the action listings section, and click “Failed Actions” in the left hand navigation.

Clearing or Retrying Failed Actions

Failed actions may be either cleared (abandoned) or retried, such that the sequence of background operations will be resumed from the point that the previous operation failed. This assumes that the underlying cause of the failure has been resolved, possibly with the assistance of the IDA support staff.

When retrying a failed action, the original failed action will be marked as “cleared” and a new action will be created, based on the failed action, and initiated. The original action summary will then show a link to this new retry action. In this manner, an action may be retried any number of times, with the multiple actions linked to each attempt accordingly (as the retried action may either succeed and be marked completed, or fail for the same or some other reason).

To clear a failed action, either click “Clear” to the right of the action in the failed action listing, or within the action summary of the failed action.

To retry a failed action, either click “Retry” to the right of the action in the failed action listing, or within the action summary of the failed action.

Cleared Actions

Cleared actions are failed actions which have either been retried or simply abandoned. If the action was retried, a link to the retry action will be present in the action summary.

To view all cleared actions, open the actions listing section by clicking on “ACTIONS” in the top navigation bar as described above, if not already in the action listings section, and click “Cleared Actions” in the left hand navigation.


Command Line Tools

Tools for interacting with a project’s staging area of the IDA service via the command line, compatible with most UNIX-like platforms (Linux, OS-X, etc), will soon be available for download from a public GitHub repository. For the time being, please see the draft guide for the IDA command line tools.

Uploading, downloading, moving (renaming), and deleting of files and folders to/from a project’s staging area are supported.

Freezing files and unfreezing or deleting frozen files, however, require use of the IDA web UI as described above, and these actions cannot be performed from the command line.