OGCE SciDAC2010 Tutorial

SciDAC 2010 tutorial slides.

OGCE SciDAC2010 Tutorial

View more presentations from marpierc.

OGCE TeraGrid 2010 ASTA Support

Presentation on advanced support activities by the IU Science Gateway Group at TeraGrid 10 (accepted presentation).

OGCE TeraGrid 2010 ASTA Support

View more presentations from marpierc.

OGCE TeraGrid 2010 Science Gateway Tutorial Intro

Tutorial intro slides at TG10 in Pittsburgh, August 2-5, 2010

OGCE TeraGrid 2010 Science Gateway Tutorial Intro

View more presentations from marpierc.

Building Science Gateways with Gadgets and OpenSocial

Check out this SlideShare Presentation:

Building Science Gateways with Gadgets and OpenSocial

View more presentations from marpierc.

Twitter, SlideShare

OGCE's Twitter feed is http://twitter.com/ogceproject.  We have many slide presentations at SlideShare: http://www.slideshare.net/marpierc/presentations.

OGCE's XBaya Running a GridChem Workflow

The movie shows the XBaya workflow composer creating and executing a computational chemistry workflow in the GridChem Science Gateway.

OGCE Demos and Presentations at SC10 in New Orleans

Please stop by and see us at SC10 November 15-18 in New Orleans at the following times and locations:

• We have a mini-workshop Tuesday from 10:00 am - 12:00 noon at the IU booth.
• Monday-Thursday at the IU Booth, we have walk-up demos on demand at the interactive touch screen kiosk.
• Monday-Thursday at the Purdue Booth, we also have on demand demos.
• Wednesday from 1:00-2:00 pm, we have on demand presentations and demos at the NCSA booth.

GCE10 Program Now Available

The GCE10 workshop program is now available from http://www.collab-ogce.org/gce10/index.php/Program.  GCE10 is being held Sunday, November 14th at SC10 in New Orleans.  Pre-prints and slides will be made available from the GCE10 web site.

GCE10 Deadline Extension

The new submission deadline for GCE10 papers is September 13th.  The call for papers is available from www.collab-ogce.org.

TG10 Tutorial Agenda

Here is the agenda for the OGCE tutorial at TG10 on Monday, August 2nd.  Tutorial material will be available for download.

1:00-2:15  

• Introduction
• computational chemistry workflow demo
• build demo

2:15-2:30: Break
2:30-5:00 

• Hands on demo of OREChem workflow (using XBaya composer)
• Hands data mining tutorial (using XRegistry and Experiment Builder gadgets)
• Gadget container overview

Agenda is subject to improvisation based on audience interest and participation.

Sample Build Instructions Movie for OGCE Portal and Services

This link shows how to get the OGCE code base and build the portal and services: http://ogceportal.iu.teragrid.org/tutorials/CodeBuild.htm

GCE10 Workshop Accepted for Supercomputing 2010

The Gateway Computing Environments workshop (GCE10) has been accepted at SC10. The workshop will be held in room 387 in the New Orleans Convention center on Sunday, Nov 14^th from 9-5:30.  More details coming soon. For previous proceedings of this series, please see http://www.collab-ogce.org/gce10/index.php/Main_Page.

Condor jDRMAA Example

Notes on using the DRMAA java implementation to submit Condor Jobs.

The Condor jDRMAA implementation can be downloaded from their sourceforge page using svn

svn co https://condor-jdrmaa.svn.sourceforge.net/svnroot/condor-jdrmaa

and use ant to build the jar or alternatively download the jar from

http://sourceforge.net/projects/condor-jdrmaa/files/condor-jdrmaa/0.2/condor-drmaa-0.2.jar/download

Version 0.2 uses the condor command line clients to submit and manage jobs.

Hence you must make sure that the condor binaries are on path and CONDOR_CONFIG is set if your Condor config file is not in /etc/condor.

The notes here pertain to Condor 7.4.2 on Mac OS X but should work on any platform that supports Condor and Java.

140-182-140-134:0.3 archit$ condor_version

$CondorVersion: 7.4.2 Mar 29 2010 BuildID: 227044 $

$CondorPlatform: I386-OSX_10_4 $

Here is the Java Class I used to perform the Job Submission.

Note that it is in the package org.ogce.jobsub.api

To replicate my setup do

mkdir -p org/ogce/jobsub/api

Create org/ogce/jobsub/api/DRMAAJobSubmission.java with the following using your favorite editor.

package org.ogce.jobsub.api;

import java.util.ArrayList;

import java.util.LinkedList;

import java.util.List;

import java.util.Set;

import org.ggf.drmaa.*;

import net.sf.igs.*;

public class DRMAAJobSubmission {

public static void main(String[] args) {

Session session = null;

String jobPath = "/bin/echo";

session = SessionFactory.getFactory().getSession();

String jobId = null;

JobTemplate jt = null;

JobInfo status = null;

ArrayList argslist = new ArrayList();

argslist.add("Hello");

try {

System.out.println(session.getDrmSystem());

System.out.println(session.getContact());

System.out.println(session.getDrmaaImplementation());

session.init("localhost");

jt = session.createJobTemplate();

jt.setRemoteCommand(jobPath);

jt.setWorkingDirectory("/tmp");

jt.setJoinFiles(true);

jt.setOutputPath("/tmp/hello.out" );

jt.setArgs(argslist);

jt.setTransferFiles(new FileTransferMode(false, false, false));

} catch (DrmaaException e) {

e.printStackTrace();

}

try {

jobId = session.runJob(jt);

status = session.wait(jobId, Session.DONE);

if (status.wasAborted()) {

System.out.println("job " + jobId + " never ran");

} else if (status.hasExited ())

{

System.out.println("job " + jobId +

"finished successfully with Exit Status "

+ status.getExitStatus());

} else if (status.hasSignaled ())

{

System.out.println("job " + jobId + " finished due to signal" +

status.getTerminatingSignal ());

} else

{

System.out.println("job " + jobId +

" finished with unclear conditions");

}

} catch (DrmaaException e) {

System.out.println("Internal Exception: " + e.getMessage());

e.printStackTrace();

}

catch (org.ggf.drmaa.InternalException ie)

{

System.out.println("Internal Exception: " + ie.getMessage());

ie.printStackTrace();

}

}

}

To Compile the file use

javac -cp ./condor-drmaa-0.2.jar org/ogce/jobsub/api/DRMAAJobSubmission.java

and to submit the echo job do

java -cp ./condor-drmaa-0.2.jar:. org.ogce.jobsub.api.DRMAAJobSubmission

This assumes you have the condor-drmaa jar in the current folder.

You should see something like

Condor

Condor

job 25.0 finished successfully with Exit Status 0

Session.init("localhost") call is used to initiate the Condor DRM adaptor and tell it to connect to the scheduler running locally.

The JobTemplate class is used to create the job description which gets translated into a condor submit file which is used as an argument to condor_submit.

The Session class has a number of static members that are used to indicate Job Status. The JobInfo class returned by Session.wait is used to retrieve the current status of the job after it has been submitted using the session.runJob method. Session.wait takes as its second argument the time until which the call must block. In this case the call blacks until the job Completes as indicated by Session.DONE.

In Order to specify job attributes not supported by the Getters and Setters we can use the setNativeSpecification method of the JobTemplate class.

So In order to submit a Condor-G job all we need to do is add the line

jt.setNativeSpecification("Universe=Grid \n globusscheduler=qb1.loni.org/jobmanager");

before submitting the job.

Please note that the attributes must be separated by a new line (\n) character so that the submit file is generated correctly. Also it must be noted that jDRMAA sets the Universe to Vanilla by default, but since the Native Specification is written in the submit file after the Universe=Vanilla line, it can be used to override the Universe setting.

Gadget Container, Registry, and Experiment Builder Movie Links

The links below show movie tutorials for the OGCE Gadget Container and two major new gadgets in the "ogce-all" release described in the previous post. 

• OGCE Gadget Container: portal software for running secure Open Social gadgets:  http://ogceportal.iu.teragrid.org/tutorials/Gadget_Container.htm
• XRegistry Gadget: a gadget for registering host computers and scientific application services into the OGCE XRegistry service: http://ogceportal.iu.teragrid.org/tutorials/XregistryInterface.htm
• Experiment Builder Gadget and XBaya Workflow Composer: the experiment builder gadget allows you to build, execute, and monitor experiments from services and workflow stored in XRegistry: http://ogceportal.iu.teragrid.org/tutorials/ExperimentBuilder.htm

OGCE "All" Release Candidate

An initial release candidate of the next generation of the OGCE software stack is now available.  The download is available by anonymous SVN checkout using the command.

svn co  https://ogce.svn.sourceforge.net/svnroot/ogce/ogce-all-rc1

This is a preview release that will need to undergo additional acceptance testing, but it is stable for early users.  Future updates can be obtained using the usual "svn update" command, and the release can be modularly rebuilt.

This download integrates several software components:

1. The OGCE Gadget Container: an iGoogle-style portal container for science gateways.
2. Experiment Builder and XRegistry gadgets for interfacing with the OGCE GFAC and XRegistry services and the XBaya workflow composer
3. XRegistery and GFAC services for registering, launching, monitoring, and sharing application services and workflows.
4. A collection of third party gadgets.

Everything can be built with a single command: mvn clean install.  Maven profiles also allow you to build subcomponents: 

1. mvn clean install -Dservices-only builds only GFAC and XBaya.  These services can run on separate servers from the gadget container, and you can run more than one instance of these services.
2. mvn clean install -Dgadget-container builds just the gadget container.
3. mvn clean install -Dgadgets-and-clients builds the gadget container, registry gadget, and experiment builder gadget.

Updated TeraGrid CA Files

(Reposting from general TeraGrid mailing lists)

Changes have been made to the list of Teragrid recognized Certificate Authorities.

The new CA tarball, v1.10, can be found on repo. This tarball adds the new PSC MyProxy CA, with hash value 4b2783ac. You can obtain a copy of the tarball and signatures file from the software WG site at:
http://software.teragrid.org/security/teragrid-certs.tar.gz
http://software.teragrid.org/security/teragrid-certs.tar.gz.asc

Please incorporate the new list as specified (RPs, Science Gateways and users within 24 hours)

New OGCE Discussion List, Google Group

We have created a new, open group for OGCE discussion questions and announcements.  The group URL is http://groups.google.com/group/ogce-discuss, and the group email address is ogce-discuss@googlegroups.com. This email address replaces the older discuss@ogce.org mailing list.

You are welcome to join, and we also allow anonymous viewing and moderated posts by group non-members.  Only project managers can see the member list.

The project RSS feed is http://groups.google.com/group/ogce-discuss/feed/rss_v2_0_msgs.xml.

Build and Deploy to Maven2 Repo

OGCE components use Maven2 for build and Archiva for repository management. For legacy reasons some dependencies are exist in extreme lab maven repository. To upgrade these components and release a version and deploy a jar file to extreme lab maven 2 repository, add the following distributionManagement tag to the pom.xml.

<distributionmanagement>
<repository>
<uniqueversion>false</uniqueversion>
<id>xRepo</id>
<name>Extreme Lab Repository</name>
<url>scp://mneme.extreme.indiana.edu/l/extreme/java/repository</url>
<layout>default</layout>
</repository>
</distributionmanagement>

mvn deploy will deploy the generated jar files to the repository but will prompt to enter the password and will also assume local account name. To avoid password prompt and to specify a different user name the following settings.xml can be placed in the $HOME/.m2 directory. For security reasons, never include the settings.xml in any of the packages, this file should be protected by file system security and is strongly advised to set the file permissions to 600.

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xsi="http://www.w3.org/2001/XMLSchema-instance" schemalocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<servers>
<server>
<id>xRepo</id>
<username>smarru</username>
<privatekey>${user.home}/.ssh/id_dsa</privatekey> <!-- if ssh keys are not available the password can be set as <password>my_password</password> -->
<passphrase>some_passphrase</passphrase>
<filepermissions>664</filepermissions>
<directorypermissions>775</directorypermissions>
<configuration></configuration>
</server>
</servers>
</settings>

Gadget Container Theme Customization

The OGCE gadget container has been updated to simplify theme customization. See http://www.collab-ogce.org/ogce/index.php/OGCE_Gadget_Container_Theme_Customization for more information.

OGCE Messaging Service Released

We are pleased to announce the release of our Axis 2 based Messaging Service. This is a standalone publish/subscribe service that is also being integrated into the OGCE workflow suite.

For more information and download instructions, see http://www.collab-ogce.org/ogce/index.php/Messaging.

OGCE Simple Monitoring Script

A simple perl based service monitoring script is available to check out from sourceforge svn at Service Monitoring Dashboard. This is a cleaned up version of the LEAD SOA Status Page authored by Suresh Marru and Marcus Christie.

The perl script periodically pings the registered services and if a service is done, sends an emails to the registered contacts. The scripts also writes our a html page with the following color convention:

• Green - Service is up
• Red - Service is down
• Yellow - Request timed out (i.e., took longer than the default 20 seconds)

The OGCE Team has deployed and will maintain the script. The status of the services can be viewed at Monitoring Dash Board. All services are pinged every 10 mins and the page is refreshed. Additionally, XSUL based services are instrumented with finer details like uptime and total requests served. In the future Axis2 based services will also be explored for more instrumented data. OGCE hosted gateway (demo/tutorial) services as well as gateway services will be monitored. To add a service to the monitoring list, send an email to {smarru or mpierce} at cs.indiana.edu (Note that only project supported gateways will be monitored).

Gateways can checkout the script and deploy and maintain on their own servers. To add a service to the script, use the following template:

push (@services, {
$NAME => "OGCE Gateway Portal",
$ID => "OGCEPortal",
$URL => "https://ogceportal.iu.teragrid.org:8443/gridsphere/gridsphere",
$CONTACTS => [admin@gateway.org],
$GATEWAYS_R_US => $OGCE
});

The GATEWAYS_R_US is a variable to categorize the services. For instance, to generate a separate listing of OGCE Hosted Services and a list of Gateway Services. The perl script is commented to explain these terminologies. The documentation is very naive but will be enhanced more.

Questions or comments, email smarru at cs.indiana.edu

Overview of OGCE Science Gateway Support

The slideshow below was presented at the Indiana University Pervasive Technology Institute and focuses on IU OGCE activities.  For fuller overviews, see for example http://www.collab-ogce.org/ogce/index.php/Tutorials  
OGCE's Advanced Science Gateway Support

View more presentations from marpierc.

Analytics Reports for OGCE Web Site

Images below are from our Google Analytics traffic tracking from March 1, 2009-March 1, 2010. The first shows overall site traffic. The second shows global traffic sources. Click the image to see a larger version.

Download Java Server Faces Components for Amazon Cloud Services

We have Amazon EC2 user interface components built with Java Server Faces available for download or SVN checkout.  

The tar.gz version is available from http://ogce.svn.sourceforge.net/viewvc/ogce/ogce-amazon-gadgets/awsGadget.tar.gz?view=tar.

To check the code out from SVN, use the command

svn checkout  https://ogce.svn.sourceforge.net/svnroot/ogce/ogce-amazon-gadgets/awsGadget

This will create a directory $HOME/awsGadget.  To build the gadgets, use the command 

mvn clean install -Dcatalina.home=/path/to/your/tomcat

For example, if you download (separately) Apache Tomcat 6.0.18 and put it in your awsGadget directory, you would run the command

mvn clean install -Dcatalina.home=./apache-tomcat-6.0.18/  

Start your tomcat server and point your browser to http://localhost:8080/AWSGadget/index.jsp.

Note the login page requires your Amazon secret key, so you will need to run this over HTTPS.  This code is provided as is under our usual SourceForge license.  

OGCE Jira URL

We are opening up our Jira project and bug tracking system for anonymous viewing.  The URL is http://grids.ucs.indiana.edu:8080/jira.

Using GFac to wrap an application as a service

Launching applications on Super Computers and managing the application during its lifetime and its output after it ends are challenging tasks. GFac aims at being able to provide simple GUI tools that can help launch and manage the application on remote supercomputers.

This post aims at serving as a quickstart to configuring a XSul based GFac persistent service.

The first step is to install GFac and XRegistry components. Good documentation to do this can be found on the OGCE web pages.


After the installation is complete set the GFAC_HOME environment variable to point to the GFac's target/dist-bin folder:

export GFAC_HOME=/Users/archit/ogce/ogce-xsul-services-1.0-RC1/sgfac/target/dist-bin

Now create a folder to house the configuration files that will be used by our service.

cd ogce-xsul-services-1.0-RC1/sgfac
mkdir helloworld
cd helloworld

Now use your favorite editor to create the three XML configuration files needed by GFac to start the persistent service namely the App description document, the host description document and service description document.

Application Description File helloworldapp.xml:

<ApplicationDescription xmlns="http://www.extreme.indiana.edu/namespaces/2004/01/gFac" xmlns:lead="http://www.extreme.indiana.edu/lead">
<applicationName
targetNamespace="http://www.extreme.indiana.edu/lead">HelloWorldApp</applicationName>
<jobType>single</jobType>
<project> <projectName>TG-XXXXXXXXXX</projectName>
<projectDescription>
TG staff
</projectDescription>
</project>
<queue>
<queueName>SERIAL</queueName>
<queueDescription>
BigRed queue
</queueDescription>
</queue>
<maxWallTime>30</maxWallTime>
<count>1</count>
<rslparm name="project">TG-XXXXXXXXXX</rslparm>
<rslparm name="queue">SERIAL</rslparm>
<deploymentDescription>
<hostName>
bigred.iu.teragrid.org
</hostName>
<executable>/N/u/tg-archi/BigRed/helloworld</executable>
<workdir>
/N/u/tg-archi/BigRed/gfactmp
</workdir>
<!-- Tmp directory for this application.
if not specified, the tmp directory of the host will be used-->
<tmpDir>
/N/u/tg-archi/BigRed/gfactmp
</tmpDir>
<applicationEnv>
<name>FIRST_NAME</name>
<value>Archit
</value>
</applicationEnv>
</deploymentDescription>
</ApplicationDescription>

ServiceDescription file: helloworldsvc.xml

<ServiceMap xmlns="http://www.extreme.indiana.edu/namespaces/2004/01/gFac"
xmlns:lead="http://www.extreme.indiana.edu/lead">
<service>
<serviceName targetNamespace="http://www.extreme.indiana.edu/lead">HelloWorldService</serviceName>
</service>
<lifeTime>
<notAfterInactiveMinutes>-1</notAfterInactiveMinutes>
</lifeTime>
<portType>
<method stageOutputDataFiles="true" forceFileStagingToWorkDir="true">
<methodName>Run</methodName>
<application paramValuesOnly="true" useLEADNameListFile="true">
<applicationName targetNamespace="http://www.extreme.indiana.edu/lead">HelloWorldApp</applicationName>
</application>
<inputParameter>
<parameterName>lastname</parameterName>
<parameterType>String</parameterType>
</inputParameter>
<inputParameter>
<parameterName>inputfile</parameterName>
<parameterType>URI</parameterType>
</inputParameter>
<outputParameter>
<parameterName>friendCount</parameterName>
<parameterType>String</parameterType>
</outputParameter>
<outputParameter>
<parameterName>friendListFile</parameterName>
<parameterType>URI</parameterType>
</outputParameter>
</method>
</portType>
</ServiceMap>

Host Description File: bigred.xml

<HostDescription xmlns="http://www.extreme.indiana.edu/namespaces/2004/01/gFac" xmlns:lead="http://www.extreme.indiana.edu/lead">
<documentInfo>
<documentName targetNamespace="http://www.extreme.indiana.edu/lead">BigRed</documentName>
</documentInfo>
<hostName>bigred.iu.teragrid.org</hostName>
<hostConfiguration>
<tmpDir>/N/u/tg-archi/BigRed/gfactmp</tmpDir>
<globusGatekeeper wsGram="true">
<endPointReference>
https://gatekeeper.bigred.iu.teragrid.org:8443/wsrf/services/ManagedJobFactoryService
</endPointReference>
<jobmanagertype>Loadleveler</jobmanagertype>
</globusGatekeeper>

<gridFTP>
<endPointReference>gridftp.bigred.iu.teragrid.org:2811</endPointReference>
</gridFTP>
</hostConfiguration>
</HostDescription>

Create the service properties file:
Finally, create the GFac service property file: helloworld.properties. This will tell GFac where to get the service, application, and host description files you just made.
serviceMapFile = helloworld/helloworldsvc.xml
appDescFile = helloworld/helloworldapp.xml
hostDescFile = helloworld/bigred.xml
port = 12346
#Comment this to make Factory and Services starts up without https
#transportSecurity=true
#private key used by the GFac to connect to other services, and start https socket, if it is enabled
ssl.hostcertsKeyFile=/home/ogce/ogce-xsul-services-1.0-RC1/sgfac/target/dist-bin/conf/ogce_services_key.pem
ssl.trustedCertsFile=/home/ogce/ogce-xsul-services-1.0-RC1/sgfac/target/dist-bin/conf/trusted_cas.pem
registryURL=https://<yourip>:6666/xregistry?wsdl
#Ask Gfac to go to local mode, so it can work without credentails. Comment if you need a remote installation
installation=local
myproxyServer=myproxy.teragrid.org
myproxyUserName=XXXXX
myproxyLifetime=86400
myproxyPasswd=XXXXXX
#gfac.prefferedProvider=Local|Gram|WSGram|SSH|ResourceBroker
wsgramPrefered=true

Now with your JAVA_HOME and GFAC_HOME set start the persisitent HelloWorldService using the run.sh provided with GFac as foolows:

./run.sh helloworld/helloworld.properties

This will start the service on port 12345 and you are ready to submit HelloWorld jobs.