Dynamic federations: storage aggregation using open tools and protocols

Dynamic federations: the Uniform Generic Redirector (UGR) (Dynamic federations: storage aggregation using open tools and protocols)

The Unifor Generic Redirector (UGR) is the core of the Storage federation system that we designed and built to match with the existing and upcoming Grid-related data management architectures. The system is able to federate storage sites and metadata endpoints that expose a suitable data access protocol, into a transparent, high performance storage federation that exposes a unique name space. The architecture can accommodate LFN/PFN algorithmic name translations without the need of catalogues. On the other hand, if catalogues are needed, several of them can be accommodated into the same federation. The idea is to allow applications to access a globally distributed repository, to which sites participate. The applications would be able to efficiently access data that is spread through different sites, by means of a redirection mechanism that is supported by the data access protocol that is used. The focus is on standard protocols for data access, like HTTP and WebDAV, and NFS can be considered as well. The architecture and the components of such a system are anyway detached from the actual protocol that is used. The focus of our design is on the fact that a federation may be composed by distant sites, and the redirection choices have to take this into account, without imposing the need of partitioning a federation into smaller ones on a geographical basis, or partitioning the name space. Another point that is important for our design, is efficiency and performance for the data access and also in the browsing case, e.g. allowing an user to list the content of a federated directory in a fast and reliable way that does not impact the performance of the whole system.

 

 

Category

Web service or application

 

Scientific areas

High performance computing

High Energy Physics

 

Main features

Storage federations

Metadata federations

high performance file metadata access

 

License

Apache License 2.0 (Apache-2.0)

 

Supported Operating Systems

Linux

 

Supported CPU Architectures

 

Programming languages

 

Build tools

 

Test tools