SG-OSG Transition FAQ
Please contribute answers to the questions below, add your own questions if you don't see it listed or add comments on the Talk page if you wish to discuss options or provide details.
What is SURAgrid?
From the SURAgrid web site:
SURAgrid is a consortium of organizations collaborating and combining resources to help bring grid technology to the level of seamless, shared infrastructure. The vision for SURAgrid is to orchestrate access to a rich set of distributed capabilities in order to meet diverse users' needs. Capabilities to be cultivated include locally contributed resources, project-specific tools and environments, highly specialized or HPC access, and gateways to national and international cyberinfrastructure.
What is OSG?
Open Science Grid is a national distributed computing grid for data intensive research. From the learn more about OSG site:
- OSG brings together computing and storage resources from campuses and research communities into a common, shared grid infrastructure over research networks via a common set of middleware.
- OSG offers participating research communities low-threshold access to more resources than they could afford individually,via a combination of dedicated, scheduled and opportunistic alternatives.
- OSG is a consortium of software, service and resource providers and researchers, from universities, national laboratories and computing centers across the U.S., who together build and operate the OSG project. The project is funded by the NSF and DOE, and provides staff for managing various aspects of the OSG.
- OSG Consortium members' independently owned and managed resources make up the distributed facility, agreements between them provide the glue for it, their requirements drive its evolution, and they contribute their effort to make it happen.
- OSG's Virtual Data Toolkit provides packaged, tested and supported collections of software for installation on participating compute and storage nodes and a client package for end-user researchers. Individual research communities, the 'virtual organizations', add services according to their scientists' needs.
- OSG works with an expanding set of research communities to help them evaluate their cyberinfrastructure needs and plan their solutions both locally across the campus and as part of national or international efforts. OSG works jointly with partners to create worldwide interoperable systems for cutting edge-research, for example the World Wide LHC Computing Grid for the upcoming experiments at CERN.
- OSG provides training through hands-on workshops and focused engagement with the community, helping new users to run applications on the infrastructure and resource owners to make their compute and storage resources accessible to the grid.
How does a SURAgrid Virtual Organization (SGVO) improve my life?
The goal of creating a SURAgrid Virtual Organization in OSG (Open Science Grid) is to make it easier to deploy, run, and maintain scientific applications on SURAgrid. Leveraging the OSG VO model will also enable many existing OSG resource provides to add support for SGVO opening the door for a significant increase to the compute platforms available to members of SGVO. Operating as an OSG VO interfaces model aso improves the lives of resource providers by adopating a well-defined and widely deployed grid computing software stack that supports many application domains. This means that resource providers will be part of a larger community of peers for supporting the software stack locally but will also be able to easily add support for existing domain-specific OSG VOs that may be of interest to researchers on their campus.
How does SGVO complicate my life?
The benefits gained from operating an SGVO are many but benefits are rarely cost free. The issues for new members of SGVO are expected to be minimal since these participants will adopt the operating practices common to all OSG VOs as they participate in SGVO.
Existing SURAgrid participants will notice changes, some significant, from traditional SURAgrid operating practices as they move to participate in the SGVO. The impact can be broken down into three areas of interest: 1) resource providers, 2) application users, and 3) community operations. Please see the SG-OSG Transition Document.
Most significantly, the existing SURAgrid certificate infrastructure is not compatible with the wider OSG community. This will require resource providers and users to obtain certificates from recognized OSG certificate authorities in order to access the broadest range of services in OSG.
In an era of clouds, do grids matter?
Clouds have greatly enhanced our ability to package services for consumption by a broad spectrum of users. Grid services are a vital component of this framework and provide a proven platform for adding large amounts of compute and storage capacity to your applications. An important goal of the SGVO is to simplify access to these compute resources so they can enhance the performance of applications available to your campus community and across the cloud.
Organization and Membership Questions
Is OSG a single organization?
OSG is composed of many participating VOs and each may have a role in the governance of OSG, adhere to an agreed upon standard for inter-VO operations, and defined individual operating practices as needed to support their science communities. You can learn more about OSG at their website.
Does operating an OSG VO mean that SURAgrid is subordinate to OSG?
In OSG terminology, a Virtual Organization (VO) is an abstraction designed to facilitate resource sharing between sites. It is does not define how those organizations operate pursuant to their missions. There are many examples of independent, real-world organizations that leverage OSG conventions to facilitate resource sharing for example LIGO, LHC, and a number of universities.
Is there a SG-OSG roadmap?
Yes and you can help define it further. Original Roadmap concept is outlined in the SURAgrid Strategic Plan, was described in the SG-OSG Statement of Shared Interest and has been further developed via working groups within SURAgrid. The road map is now being expanded via this transition document.
How does this fit with the SURAgrid Starategic Plan?
The SURAgrid Strategic Plan 2008-2012 defines five major goals. The SGVO is seen a key solution to building an operational infrastructure and specifically satisfies:
- GOAL 1: will significantly expand our outreach potential.
- GOAL 5: Strengthen our partnership.
Currently SURAgrid membership is determined by contributing resources. Is this the same?
De facto, by being a supporter of SGVO – i.e. providing a resource that can be used by SGVO – SURAgrid membership is obtained. Perhaps SGC should review/adjust the SURAgrid Membership criteria (http://www.sura.org/programs/docs/SGMembershipOct07.pdf) and consider a shift of emphasis to value the effort in community participation and contribution.NB: Alan Sill: Contribute resource and/or submit SG approved application.
Will SURAgrid still have All-Hands Meetings of its own?
SURAgrid has traditionally held a Spring and Fall All-Hands meeting at the SURA offices in Washington D.C. SURAgrid has always tried to co-schedule our All-Hands meeting with other meetings of interest to our membership, for example Internet2 and CASC. Establishing an SGVO will provide us yet another opportunity to address the interests and travel needs of our members with the opportunity to co-schedule our all hands meetings with OSG All-Hands meetings. The SURAgrid community will continue to schedule meetings that best address our members needs.
What is the SG Membership database?
OSG uses VOMs. Much like the SURAgrid LDAP server, VOMS provides a needed mechanism to manage the user base.
Working Group Questions
How does the Access Management Working Group change?
The AMWG will continue as a technical requirements forum that we think would work well within the OSG working groups; and the AMWG will also continue as a high-level in supporting the access management infrastructure used by SGVO.
What OSG working groups exist?
Many questions have been raised about certificate use in SURAgrid and OSG. This section will attempt to answer the common questions and address the perspective of an existing SURAgrid participant migrating to SGVO.
What are certificates and why do I need one?
A certificate is an identity document that is signed by an authority that is trusted to validate the identity expressed in the document. Certificates are central to how participants in grid computing environments identify each other. Grid computing environment traditionally expect users to work with certificates directly but this is not a strict requirement. Since all popular grids, including OSG and SURAgrid, make this assumption, these FAQ entries will take that perspective too. For more information on grid computing and certificates please see...
What certificate infrastructure does SURAgrid use?
SURAgrid was established to help campus adopt grid computing and share computing resources with other campuses. In order to simplify identity validation for users of SURAgrid and avoid the challenges of operating a centralized certificate authority (CA), a distributed identity trust network was constructed to allow on-campus contacts to verify the identity of SURAgrid users and issue a certificate to those users from an campus-operated CA. This network was constructed using a bridge certificate authority (BCA) to build an identity validation path rooted at the campus CA. This infrastructure allows each campus to run it's own CA and assert identities for which they are authoritative.
A bridge certificate authority (BCA) supports building a federated identity network by establishing trusts between campus-operation CAs. The SURAgrid BCA manages the trusts between campus CAs. Exploration of the BCA fabric used by SURAgrid is documented in The Case for Using Bridge Certificate Authorities for Grid Computing. Addtional background on BCAs is available Bridge Certification Authorities: Connecting B2B Public Key Infrastructures.
What certificate infrastructure does OSG use?
As with any grid, OSG relies on a trusted collection of CAs to assert the identity of individuals consuming grid services. In order to facilate trusting the identity of users that may come from many number of organization, OSG has adopted the convention requiring any CA that wishes to be part of the curated set of CAs distribted in an easy-to-install bundle to adhere to the policies of the International Grid Trust Federation (IGTF). Individual sites and VOs are free to use any identity trust model that meets their operational requirements; it simply requires adjusting configuration settings in the software components. In practice, however, only certficates issued by an IGTF compliant CA will be trusted across the broadest set of OSG resource providers. In order to maximize the compute and storage resources available to users, it is best to use an IGTF accredited CA.
What is the IGTF?
The IGTF is the Internation Grid Trust Federation. It is a federation of policy management authorities around the globe that seeks to define interoperable certificate profiles and minimum identity standards to ensure the broadest possible reach of certificates issued by member CAs. From their web site:
The International Grid Trust Federation (IGTF) is a body to establish common policies and guidelines between its Policy Management Authorities (PMAs) members and to ensure compliance to this Federation Document amongst the participating PMAs. The IGTF does not provide identity assertions but instead ensures that within the scope of the IGTF charter the assertions issued by accredited authorities of any of its member PMAs meet or exceed an authentication profile relevant to the accredited authority.
What certificates will SGVO use?
The goal of establishing SGVO is to simplify running applications on the grid by streamlining the operation of grid resources and increasing the compute capacity available to users. In order to accomplish this goal, it is strongly recommended that resource owners and grid users adhere to common operating conventions in OSG. This includes using resource and user certificates issued by an IGTF accredited CA.
This recommendation implies a noticable departure from existing resource and user certificate practices in SURAgrid. Resource owners and grid users will need to acquire certificates from IGTF accredited CAs. We are working with OSG to piggy-back on OSG's existing CA issuing process in the near term and exploring options for longer term solutions that can scale to handle larger campus communities.
The certificate infrastructure of SURAgrid and OSG are technically compatible, however, the SURAgrid BCA and the campus CAs that are in use on SURAgrid are not certified by the IGTF and will not be incorporated into the the default CA bundle in common use on OSG. This means that only sites that explicitly trust the SURAgrid BCA bundle will accept requests from users who use certificates issued by these CAs. Adding this trust bundle to the resource configuration is likely viable for sites which only intend to serve SURAgrid users, however, the adoption of these trusts is likely limited outside the scope of SGVO.
The state of the art in higher-education trust federations is undergoing significant change. The InCommon Federation has established a large trust federation across institutions. Also, CILogon can issue certificates directly to end users to participants in InCommon or OpenID providers participating in the US Identity, Credential, and Access Mechanism trust framework (e.g. Google). Additionally, CILogon operates an IGTF CA that will soon be able to issue end-user certificates to InCommon Silver participants. In short, there are a number of emerging technologies which promise a scalable solutions to the certificate issuance process.
Does the OSG stack come with installed certs?
The OSG stack does not comes with any certs installed. During the installation sites choose to install the CA roots that they trust. Most sites will install the curated bundle of IGTF accredited CAs that is maintained by OSG. This is the most convenient solution and has a solid policy foundation, therefore it is most the most common. This is the motivation for the recommendation to use certificates from IGTF accredited CAs.
Where can I learn more about these issue?
The SURAgrid Access Management Working Group is the place where these issues are debated and solutions developed. You can also address questions to the the SURAgrid users list.
Service Provider Questions
How do I make my cluster available to SGVO?
The definitive answer to this question is available on the OSG website. For the purposes of this FAQ, the steps will involve installing OSG VDT-based grid software stack an configuring it appropriately. As discussed above, configuring your resource with a certificate issued by an IGTF accredited CA will provide the greatest interoperability for your resource with all OSG particpants. This will also allow you to leverage OSG's availability and accounting services to advertise your resource and review usage.
Can I be a member of SGVO using the original simplified install SURAgrid stack?
Possibly, although some services that lead to a fully configured OSG resource are not included in the SURAgrid simplified install stack. The OSG stack is a superset of SURAgrid simplified install stack. Both are based on the VDT stack. Technically, OSG does not require the use of a packaged stack. Installing components individually is more complex and best done by seasoned grid resource providers. Due to the common VDT heritage of both stacks, the OSG stack install should be familiar to sites using the SURAgrid simplified stack.
What are some of the differences between the stacks?
The SURAgrid simplified software stack basically includes web services GRAM, with option pre-WebServices GRAM, and a recommendation to install GSI-SSH to simplify building and deploying software to remote resources.
The OSG stack will support pre-WebServices GRAM and does not install GSI-SSH. The Engage VO has defined software packaging and maintenance processes that support deploying software used by a VO without requiring traditional logins on target resources. These methods will be explored further. The OSG stack also has reporting tools, extra configuration options, Gracia accounting tool each of which adds functionality. SGVO operations can be enhanced by adopting these reporting/accounting functions.
Run the OSG stack on an Internet accessible gateway and hook the grid job submission components into your site scheduling fabric on the backend. This solutions offers a traditional DMZ/internal arrangement of services that should satisfy site security requirements.
Do I have to run a grid gateway?
No. You can run the OSG stack directly on your cluster head node, so long as the head node runs a supported operating systems (typically Linux).
Why do I have to run a resource monitor?
You don't, but you'll be glad you did.
Yes. OSG defines a number of resource types including compute and storage services.
What if I don't have a Linux cluster (a.k.a. what about AIX support)?
The best way to run the OSG stack is to run it on a supported operating system. If your cluster is not Linux-based, you can run a Linux-based gateway node that will accept grid job requests and submit them to your cluster.
What is the role of a system admin?
The goal of establishing SGVO is to simplify running applications on the grid. One aspect of this simplification is to free system administrators from having to support every application. Instead, application packaging and distribution process are adopted the enable application administrators to maintain and curate applications for a VO. These applications can then be distributed to resources that provide cycles to SGVO without the manual intervention of the system administrator. This keeps the site system administrator from being overwhelmed by the need to support large numbers of applications and frees them on keeping the resources operation at peek efficiency.
VO Operation Questions
How will services like the VO user database be operated an maintained?
Running an organization, real or virtual, requires a certain amount of technology infrastructure. With the advances in infrastructure service models brought about by cloud computing, it is easier than ever to maintain services as a community on virtual platforms and have them hosted at member sites. Maintaining these services and providing cycles to run them is a key contributions which members will make to the SURAgrid community.
Is VOMS the SGVO membership database?
Yes, in part. The reason for using the term "membership database" is because this is a more accurate description of the service SURAgrid will maintain. As with any organization, a list of members is crucial for effective operations. Knowing the members of your community allows you to identify participates and the roles they fill in your organization. This information can be used to grant access broadly to all members or restrict access to privileged operations based on their responsibilities. A membership database is a core component of any service organization and SURAgrid currently maintains membership in a veriety of places, both electronically in LDAP and in various static web lists. Our operations will improve as we improve the quality and capabilities of our membership database.
The OSG VO Membership Service (VOMS) provides information about a user to the service provider from which they are requesting compute cycles. This allows that site to know more about the user so that they can provide the services at the request about SURAgrid members. Running a VOMS server is the easiest way to package user information from our membership database so that other OSG sites can understand it. Running a VOMS server does not mean we have to use it's user interfaces. It can be used as an externally facing interface for the express purpose of exposing information to other OSG sites.
We can develop what ever user facing tools for our membership database that are practical and meet the operational needs of SURAgrid. In fact, it is reasonable to expect that other interfaces to our membership database would exist in the future. For example, Shibboleth could be used to add a SAML interface to allow communication with service providers in the InCommon Federation.
There is a welth of information available for running applications on OSG. Establishing SGVO is meant to simply running applicaitons on the grid by being able to adopt these practices on our own resources. To learn more about getting an application running on OSG please see getting your application running on OSG.
How do I register an application with SGVO?
TBA (to be answered)
How will I run an application on SGVO?
What is an application maintainer?
OSG runs single processor jobs, can I run MPI jobs?