Using the OGCE Gadget Container

The OGCE Gadget Container allows you to deploy and display Google and Open Social gadgets. To get the gadget container, check it out from SourceForge with SVN:

svn co https://ogce.svn.sourceforge.net/svnroot/ogce/ShindigOGCE/ishindig-trunk/ ogce-gadget-container

You can also download the tar through SF's source code browser: http://ogce.svn.sourceforge.net/viewvc/ogce/ShindigOGCE/ishindig-trunk.tar.gz?view=tar

Next, cd into the source directory and do the following:

* Unpack Apache Maven (included in the download):
tar -zxf apache-maven-2.2.1.tar.gz; export PATH=/path/to/ogce-gadget-container/apache-maven-2.2.1/bin:$PATH

This will make sure you have the correct version of Maven.

* Edit config/ishindig.properties. Edit site.host to be your server's IP address. You can use localhost if you want. For your first build, don't change the H2 database settings, but you can use MySQL in the future.

* Build and deploy everything by running mvn clean install from the top level directory (ogce-gadget container if you used the svn checkout command above).

* Start Tomcat with startup-tomcat.sh.

You should now point your browser to http://your.server.name:7070/ishindig-webapphttps://your.server.name:7443/ishindig-webap. You should see the screen below. Create an account and log in.

After logging in, you should see a tabbed layout like the one below:

You can move gadgets around by drag and drop. Click the skins layout (drop down on the upper right) to change colors. Click the layout drop down (also upper right) to switch between table and tree layouts. Add tabs by clicking the "add a tab" button. You can create a layout like so:


To add a gadget, click the "Add a gadget" link in the upper left tool bar area. Fill out the form by providing a nickname for the Gadget and the URL to its XML definition.

Gadgets can be loaded from anywhere, but we provide a selection that can run locally in the gadgets-repo directory. We render gadgets using Apache Shindig, which is included in the download and deployed by Maven. You can use Shindig to render most gadgets available from iGoogle.

To administer users, click the Admin interface link on the login page and log in as administrator (admin/admin are the defaults).


Finally, you can clean up your installation with the command mvn clean. This will return you to the original SVN check out state, EXCEPT it will not delete the database. If using the default H2 database, the directory for the database files is set in conf/ishindig.properties. By default the directory is located in $HOME/ogce-h2-data/. You must delete this manually.

Job Submission and JavaScript Gadgets

This post continues our series on using the Cyberaide JavaScript API for common Grid operations. For more information on downloading and installing the current code, see
http://collab-ogce.blogspot.com/2009/10/using-ogces-javascript-api-to-myproxy.html. If you already have to code, just use svn update from your working directory to get the latest updates.

In previous posts, we took a tour of the APIs for getting a proxy and submitting simple jobs. These have been assembled into a single gadget, cyberaideJobman.xml, located in the client sub-directory. To build , run mvn clean install from the top level cyberaide directory and then start Tomcat with the startup-tomcat.sh script. The gadget will be available from http://your.server.name:8080/grid/cyberaideJobman.xml. This is suitable for testing purposes, but you should modify Tomcat to run SSL. Near-future versions of the build system will do this by default.

You can now point your favorite gadget container (such as iGoogle) to the above URL. Steps for adding a gadget depend on the container. For iGoogle (assuming you have an account), click "Add stuff" in the upper right corner and then "Add feed or gadget" in the lower left corner. Cut and paste your gadget's URL.

OGCE Web Site Up Again

The server upgrade is complete and the OGCE web site is available again.

OGCE Website Down for Upgrade

We are upgrading the OGCE website (www.collab-ogce.org). The site should be up again soon.

Making a MyProxy Gadget

In an earlier post we looked at a simple example that described how to build a Web interface to MyProxy with the Cyberaide JavaScript API. Now we will turn it into a gadget. This is straightforward, and you can easily test it out with iGoogle, Apache Shindig, or Google Code's wiki. We can reuse most of the previous example, so we will only discuss changes below.

Step 0: Wrap the HTML and JavaScript in the CDATA section of a new file, MyProxyGadget.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Module>
<ModulePrefs title="MyProxy Gadget"
height="500">
</ModulePrefs>

<Content type="html">
<![CDATA[

Actual code goes here

]]>
</Content>
</Module>

It is also possible to use an iFrame for simplicity. The iframe can be embedded in the CDATA section and point to the HTML file's URL, or you can use a "url" gadget. In this case, change Content's type attribute value from "html" to "url" and provide the URL for the HTML code you want to embed.

Step 1: You will need to use full paths to all imported JavaScript libraries, images, and so on. Relative paths will not work. So for example, you import the CogKit2.js library like so:

<script type="text/javascript" src="https://my.host.com/grid/CogKit2.js"></script>

where my.host.com should be replaced with your Cyberaide deployment server. Note this also applies to the Cyberaide agent server URL. This must be changed from

var url="../agent/services/agent";

to

var url="https://my.host.com/agent/services/agent";

Step 2: You will need to use a modified SOAP JavaScript library, soapclientGadget.js. This replaces soapclient24NSMod.js. This is our modification. The library is available from our SVN repository.

Step 3: Load your gadget in your favorite gadget container. To use iGoogle, create an account and log in if necessary and then click the "Add Stuff" link in the upper right. Assuming you haven't published your gadget yet, click the "Add feed or gadget" link in the lower left side of the page. Type in the full URL to your gadget and submit. That's it.

More on JavaScript and Globus GRAM

In the previous post, we looked at how to submit a job using the JavaScript CoG API (a.k.a Cyberaide JavaScript). We'll now look at the remaining parts of the job management API: listing all jobs, checking status of a particular job, listing outputs of a job, and viewing output. We'll just show the JavaScript codes and not the full HTML. The complete example is JobSubGadget.html.

As we saw last time, after a job is submitted, we are returned a job id. We actually treat all jobs as workflows, so this will be colloquially referred to as a wfid in the text. Also as we saw in the last post, all our function calls are accompanied by a callback function. JavaScript is a functional language, so you can pass functions as arguments to other functions. The recipient function then invokes your function.

Listing All Your Jobs

All of your jobs are stored persistently (using a lightweight database and the file system), so you can list all of your jobs' IDs with the function below. The callback function receives a JSON-formatted WFID list, which you can cast directly into a JavaScript variable if you choose.

/*--------------------------------------------------*/
// List jobs
/*--------------------------------------------------*/
function showJobs() {
jscog.listMyWf(listMyWfResponse);
}
//The call back.
function listMyWfResponse(jsonRet){
alert(jsonRet);
}

Checking the Status of a Particular Job

This method returns the status of a particular job as a JSON object. It takes a WFID as input. In the example, we obtain the WFID from the value of an HTML element called "wfid-status". This line of the example is not required.

/*--------------------------------------------------*/
// Check job status
/*--------------------------------------------------*/
function showStatus() {
var wfid=document.getElementById("wfid-status").value;
jscog.statusQuery(wfid,statusQueryResponse,"");
}
//The callback function.
function statusQueryResponse(jsonRet,loc) {
alert(jsonRet);
}

Listing Job Output Files

This function lists the output files associated with a particular WFID. In the previous example, this is the standard output file. Once again the output is JSON-encoded, and the input is obtained from a document element's value (wfid-list in this case). Again, this method for determining the input WFID value is only illustrative.

/*--------------------------------------------------*/
// List job output files
/*--------------------------------------------------*/
function listOutputFiles() {
var wfid=document.getElementById("wfid-list").value;
jscog.listOutput(wfid,listOutputResponse,"");
}
function listOutputResponse(jsonRet) {
alert(jsonRet);
}

Show the Output of a Job

Finally, we can view one of the output files as shown below. The return value is JSON-encoded as before.

/*--------------------------------------------------*/
// Show job output
/*--------------------------------------------------*/
function displayResult() {
var wfid=document.getElementById("wfid-display").value;
var outputfile=document.getElementById("wfid-outputfile").value;
jscog.displayResult(wfid,outputfile,fetchOutputResponse,"");
}
function fetchOutputResponse(jsonRet) {
alert(jsonRet);
}
function fetchOutputResponse(jsonRet,loc) {
alert(jsonRet);
}

Submitting a Grid Job with OGCE JavaScript API

In the previous post, we showed how to get a MyProxy credential with the OGCE's Cyberaide JavaScript API. We'll now look at how to submit a simple job. We'll build up a very simple HTML page to do this, but you should note that we have much more appealing clients than this if you want to use something out of the box.

Let's start by building upon our earlier MyProxyGaget.html example. This file is called JobSubGadget.html and is in the client subdirectory of the source code. We will combine the MyProxy form with a Job Submission form to simplify security issues; that is, the user will be presented with a form to get a proxy credential (shown last time) and a form to launch a job. The job launching form only appears after the user successfully authenticates.

To do this, we will start with a little jQuery trick. We'll wrap the two tables in <div> classes called proxyFormClass and jobSubClass and define the following CSS entries:

.proxyFormClass {
}

.jobSubClass{
display: none;
}

Then in the authentication callback function (see previous post), we put the code snippet below:

$('div.jobSubClass').show();
$('div.proxyFormClass').hide();

This will turn off the proxy form fields and turn on the job submission fields.

Now we're ready for the job submission form itself. Here's a very bare bones one:
<div class="jobSubClass" id="jobSub">
Fill in the form below to launch a job.
<table>
<tr>
<td>Command</td>
<td><input name="cmd" type="text" id="cmd" size="50" value="/bin/ls"/></td>
</tr>

<tr>
<td>Arguments</td>
<td><input name="arg" type="text" id="arg" size="50" value="-l"/></td>
</tr>
<tr>
<td>GRAM Host</td>
<td><input name="rHost" type="text" id="rHost" size="50" value="grid-co.ncsa.teragrid.org"/>
</td>
</tr>

<tr>
<td>GridFTP Host</td>
<td><input name="ftpHost" type="text" id="ftpHost" size="50" value="gridftp-co.ncsa.teragrid.org"/>
</td>
</tr>

<tr>
<td>Standard Output</td>
<td><input name="stdout" type="text" id="stdout" size="50" value="junk-ls.out"/>
</td>
</tr>

<tr>
<td>Provider</td>
<td><input name="provider" type="radio" id="gt2" checked>GT2
<input name="provider" type="radio" id="gt4">GT4</td>
</tr>
<tr>
<div name="submitStatus" id="submitStatus"/>
</tr>

</table>
<input name="button1" type="button" class="runButton" id="button1" onclick="submitJob();return false" value="Submit job"/>
</div>

This should look familiar. One important thing to note (besides the little div wrapper) is that we need to provide the hostname for both the GRAM and GridFTP hosts. We need this to both run the job and pull back the output. Not all grid installations (notably the TeraGrid) run both the GRAM and GridFTP server on the same machine (network file systems), so we have to allow for this.

Now let's look at some minimal JavaScript code to invoke the service. The main thing we need to do is collect the submission form's parameters and invoke the service.

function submitJob() {
/* extract job specification */
var cmd = document.getElementById("cmd").value;
var arg = document.getElementById("arg").value;
var rHost = document.getElementById("rHost").value;
var ftpHost=document.getElementById("ftpHost").value;
var stdout = document.getElementById("stdout").value;
var provider = document.getElementsByName("provider");
var prov = null;
if(provider[0].checked){
prov = "GT2";
}
else {
prov = "GT4";
}

/* construct karajan workflow */
var strProj = jscog.constructRemoteJob(cmd, arg, rHost, stdout, ftpHost, stdout, prov);
document.getElementById("submitStatus").innerHTML = "Job Submitting";
loc="";
jscog.submitWf(strProj,submitResponse,loc);

}

Job submission is actually a two step process in the API: first we construct the job's workflow (we use the CoG's Karajan on the backend), and then we submit the constructed workflow script (jscog.submitWf()). We also need to write a callback handler (submitResponse in the submitWf() command) to see if the job was submitted successfully. Here is a really simple one:

/*
* Callback to handle the submission response.
*/
function submitResponse(ret) {
document.getElementById("submitStatus").innerHTML="Your job has been submitted with workflow ticket "+ret;

}

The callback function gets an integer ret back from the server. If submission was successful, then this integer will be the job ticket, which you can use to query status and get back results or error messages.

Using OGCE's JavaScript API to MyProxy

We are in the process of releasing a JavaScript COG API, Cyberaide, that works with common Grid services like MyProxy, GRAM, and GridFTP. Our goal is to provide a method for embedding these calls into many different frameworks, including non-Java languages and tools such as PHP and Ruby on Rails. We are want to provide tools that can be used to build Open Social gadgets that perform Grid operations.

For a fuller description of the project, see http://cyberaide.org/projects/cyberaide/cyberaide-javascript. To browse or download the working version of the code, see http://code.google.com/p/cyberaide/. The examples (very actively developed) can be built using one command,

mvn clean install

from the cyberaide source's top level directory. Edit the top level pom.xml file's properties first. Use the version of Maven provided in the download. You can also rebuild portions of the system with the command

mvn clean install -f client/pom.xml

where client is one of the project modules. After you build everything, start up Tomcat (use the startup-tomcat.sh convenience script). Point your browser to http://localhost:8080/jsportal.html to see some examples.

To shut down everything, you can use the shutdown scripts (shutdown-tomcat.sh and shutdown-mediator.sh). To see debugging information, check out the log files: agent.log and mediator/mediator.log.

MyProxy API Example
The full example is called "MyProxyGadget.html" and is located in the subdirectory client of the source code. To start, you need a set of HTML input fields as usual:

<table>
<tr>
<div id="authStatus" name="authStatus"/>
</tr>
<tr>
<td>Myproxy Server Host:</td>
<td><input name="host" type="text" id="myproxy.host" size="50" value="myproxy.teragrid.org"/></td>

</tr>
<tr>
<td>Myproxy Server Port:</td>
<td><input name="port" type="text" id="myproxy.port" size="50" value="7512"/></td>
</tr>
<tr>
<td>Myproxy Username:</td>
<td><input name="user" type="text" id="myproxy.user" size="50"/></td>
</tr>
<tr>
<td>Myproxy Passphrase:</td>
<td><input name="password" type="password" id="myproxy.password" size="50"/></td>
</tr>
</table>
<input name="button0" type="button" class="runButton" id="button0" onclick="auth();return false" value="Authenticate"/>

Next, you need to import the COG JavaScript library:

<script type="text/javascript" src="CogKit2.js"/>

Now implement the auth() method from your button's onclick attribute. This should of course be located in a section <script language="javascript" > section.

/* Initialize the JavaScript COG */
var url = "../agent/services/agent";
var jscog = new CogKit2(url);
var authStatus = false;

//Define your auth function here.
function auth(){

//These point to the elements in the table above.
var host = document.getElementById("myproxy.host").value;
var port = document.getElementById("myproxy.port").value;
var user = document.getElementById("myproxy.user").value;
var password = document.getElementById("myproxy.password").value;

//A little window dressing
document.getElementById("authStatus").innerHTML=" Authenticating";

//Now invoke the JS CoG auth method from the imported library.
jscog.auth(host, port, user, password, authResponse);
}

The last argument, authResponse, is a callback function that you can use to determine if authentication was successful or not. A sample implementation is shown below.

/*
* the authenticate callback function
*/
function authResponse(ret) {
if(ret){
authStatus = true;
document.getElementById("authStatus").innerHTML="Authenticated successfully!";
document.getElementsByName("password")[0].value="";
} else{
authStatus = false;
document.getElementById("authStatus").innerHTML="Failed to Authenticate!";
document.getElementsByName("password")[0].value="";
}
}

We'll look at more functions in upcoming posts.

Maven Out of Memory Errors

We have seen some Maven errors compiling the OGCE portal on Mac OS X 10.5 with Java 1.6. To remedy this, set the MAVEN_OPTS environment variable:

(bash) export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=128m"
(csh) setenv MAVEN_OPTS "-Xmx1024m -XX:MaxPermSize=128m"

The actual values (1024m and 128m) can be varied if necessary.

The error message output from running "mvn clean install" is shown below.

...
/target/classes
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure
Failure executing javac, but could not parse the error:


The system is out of resources.
Consult the following stack trace for details.
java.lang.OutOfMemoryError: Java heap space
at com.sun.tools.javac.code.Scope$ImportScope.makeEntry(Scope.java:385)
...

Velocity Portlet Bridge Updates

Thanks go to Michel David da Costa for updates to the Velocity Portlet Bridge code. This is used to build various older portlets in OGCE (see for example JobSubmission) and will work in various versions of GridSphere and uPortal.

The code is here:

http://ogce.svn.sourceforge.net/viewvc/ogce/OGCE-Util/VelocityPortlet/trunk/.


You can check this out anonymously using SVN:

svn co https://ogce.svn.sourceforge.net/svnroot/ogce/OGCE-Util/VelocityPortlet/trunk VelocityPortlet

If you prefer, you can also get the tar from http://ogce.svn.sourceforge.net/viewvc/ogce/OGCE-Util/VelocityPortlet/trunk.tar.gz.


To build, use Apache Maven 2.0.x and the command

mvn clean install

OGCE TG09 Tech Track Presentation

SlideShare Presentation:

OGCE TG09 Tech Track Presentation

View more OpenOffice presentations from marpierc.

Cloud and Gateway Developer Positions at the Pervasive Technology Institute

We are looking to fill two positions. Please see the full ad below.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The following positions based at IUB and reporting to Marlon Pierce, Assistant Director, Community Grid Labs, is being posted externally. If interested, you must apply online at http://www.jobs.indiana.edu/. Refer to positions number #00034420 and #00034425.

Principal Software Research Engineers - PAE4IT (Two positions available)

Description and Responsibilities:

Indiana University is looking for experienced individuals to fill multiple positions in Cloud computing , Grid computing, and message-oriented middleware development. The positions call for expertise in Web service software development, experience developing Java Messaging Service (JMS) applications and related Enterprise event bus technologies, experience with distributed systems, sensor webs, and/or audio-video systems research and advanced development.
Working independently, the incumbent provides expertise in the evaluation, development, implementation and deployment of specific Cloud Computing, Message-Oriented Middleware, and Grid-related technologies in conjunction with Pervasive Technology Institute and Research Technology staff and collaborators. Investigate research problems in distributed computing systems, develop software using best software engineering practices, publish peer-reviewed research articles in workshops and journals, evaluate existing technologies, supervise staff and graduate students, and oversee all development aspects of Cloud and Grid scientific data and application management. Work with scientific users to develop Cloud and Grid systems. Collaborate with key academic researchers and open source user community including faculty, scientific researchers, and grant stakeholders.
Indiana University's Pervasive Technology was recently awarded a 5 year extension to its core funding from the Lilly Endowment, following its highly successful initial 7 years (as the Pervasive Technologies Laboratories). PTI will be located in Indiana University's new state of the art Research Incubator facility. See http://www.pervasive.iu.edu/ for more information. To date, the PTI laboratories have published more than 880 peer-reviewed papers, have won over $180,000,000 in grants over and above their core funding, and have released and maintain numerous open source software packages. PTI researchers work closely with Indiana University's Research Technology group, who maintain IU's world class cyberinfrastructure, http://racinfo.indiana.edu/cyberinfrastructure/.
Qualifications:

Master's degree in Computer Science or relate technical discipline (such as physics, informatics, engineering) is required. PhD preferred. Ph. D. incumbents must have demonstrated (through peer-reviewed publications and software products) a primary research focus on one or more of the following: a) distributed, parallel, Grid and/or Cloud computing; b) high performance, distributed messaging systems; c) audio/video collaboration systems; and d) security in distributed systems. Incumbents with Master's degree must have five to seven years experience in software development and deployment in the appropriate field or fields.

Requirements include a proven ability to undertake and succeed in innovative research projects, and excellent communication skill (covering scientific article writing, presentation skills, and the ability to work with and lead teams). Experience developing software systems using standard software engineering practices (such as source code management software, build and test systems, etc). For Cloud and Grid incumbents, experience developing Web Services, REST services, scientific workflows, and distributed Web applications on Unix/Linux/Mac OSX and/or Windows. Experience developing applications on Grids using software such as Globus or Condor. Experience developing scientific and research Cloud applications using Amazon Web Services, Microsoft Azure, Google AppEngine, and related systems. Experience developing data-parallel scientific, machine learning, and/or information retrieval algorithms using systems such as Apache Hadoop and Microsoft Dryad. For audio/video stream incumbents, demonstrated knowledge and implementation of standards as well as innovative systems. For message-oriented middleware incumbents, demonstrated sophisticated experience with standard MOM software. MOM incumbents should have experience both with core MOM development (efficient routing, security, reliable delivery) as well as applications.
Ability to foster and maintain collaborations with internal and external academic researchers through a balanced, service-based approach required. Demonstrated ability to learn new technologies and tools. Demonstrated initiative in evaluating and adopting new technologies. Organizational and leadership skills required. Able to operate effectively in a complex and dynamic environment with the capability of functioning as a project leader. Excellent oral and written communication skills. Demonstrated conceptual, analytical, and logical abilities. Ability to interact successfully with staff and faculty in extended consultations; demonstrate enthusiasm for new technologies; and deliver excellent outcomes under high pressure and short deadlines. Excellent communication skills required to interact with people of varying levels of knowledge.

Indiana University is an Affirmative Action/Equal Employment institution.

OGCE Portal 2.5

OGCE Portal 2.5 is available for download. The release includes updates to the XBaya portlet, the addition of the GFAC portlet into the main build, and bug fixes to the GPIR portlet.

You can get the code by either SVN anonymous checkout or by downloading the tar.gz.

SVN Anonymous Checkout: svn co https://ogce.svn.sourceforge.net/svnroot/ogce/tags/ogce-portal-only-2.5-release

Download the Tar: http://ogce.svn.sourceforge.net/viewvc/ogce/tags/ogce-portal-only-2.5-release.tar.gz?view=tar

OGCE Workflow Suite: XBaya, XRegistry, and GFAC

Ogce Workflow Suite Tg09

View more OpenOffice presentations from smarru.

Open Life Science Gateway: Gadgets, Open Social, and JSON-RPC

The Open Life Science Gateway code (developed by Wenjun Wu) demonstrates how to develop science gateways with Google Gadgets, Open Social, JSON-RPC, and other Web 2.0 techniques. Browse code here: http://ogce.svn.sourceforge.net/viewvc/ogce/ogce-olsg/.

You can get the code by anonymous SVN checkout or download the tar.gz.

SciDAC Tutorial: GTLAB Tag Libraries Overview

GTLAB Installation Tutorial for SciDAC 2009

View more OpenOffice presentations from marpierc.

SciDAC Tutorial: OGCE Overview

OGCE Overview for SciDAC 2009

View more OpenOffice presentations from marpierc.

OGCE Axis Services 1.0 Tagged

Version 1.0 of the OGCE Axis Services have been tagged. See http://ogce.svn.sourceforge.net/viewvc/ogce/tags/ogce-axis-services-1.0/ This includes the Resource Discovery Service and the Resource Prediction Service. See http://www.collab-ogce.org/ogce/index.php/Main_Page#Key_to_Release_Status for an explanation of our release tagging conventions.



Get the tar: http://ogce.svn.sourceforge.net/viewvc/ogce/tags/ogce-axis-services-1.0.tar.gz?view=tar


Check out from SVN: svn checkout https://ogce.svn.sourceforge.net/svnroot/ogce/tags/ogce-axis-services-1.0

GTLAB TG09 Release Tagged

GTLAB has been tagged. This release includes integration with Tomcat's login, new tags and examples for displaying INCA status information on the TeraGrid, simple support for GridShib invocations, and various bug fixes. See http://ogce.svn.sourceforge.net/viewvc/ogce/tags/GTLAB-tg09/.

Get the tar: http://ogce.svn.sourceforge.net/viewvc/ogce/tags/GTLAB-tg09.tar.gz?view=tar


Check out from SVN: svn co https://ogce.svn.sourceforge.net/svnroot/ogce/tags/GTLAB-tg09

Tagged Releases for GFAC and XRegistry Services

Latest stable release versions of the GFAC and XRegistry services are available for download from our SourceForge repository. This is part of the "TeraGrid 09" release of OGCE software, one of our two major updates this year. OGCE portal software tag releases are in preparation.

Tar Bundles:

XRegistry: http://ogce.svn.sourceforge.net/viewvc/ogce/tags/xregistry-tg09.tar.gz?view=tar

GFAC: http://ogce.svn.sourceforge.net/viewvc/ogce/tags/sgfac-tg09.tar.gz?view=tar

SVN Anonymous Checkout
XRegistry: svn co https://ogce.svn.sourceforge.net/svnroot/ogce/tags/xregistry-tg09

GFAC: svn co https://ogce.svn.sourceforge.net/svnroot/ogce/tags/sgfac-tg09

OGCE Overview Presentation from E-Science 2008

OGCE Project Overview

View more OpenOffice presentations from marpierc.

GTLAB Slides

Gtlab Install

View more OpenOffice presentations from marpierc.

OGCE-Utils and Small SVN Reorganization

The Proxy Manager API and Velocity Portlet Bridge code have been moved to the new OGCE-Util directory in SVN (see http://ogce.svn.sourceforge.net/viewvc/ogce/OGCE-Util). The util directory in general is for standalone applications with jar (rather than war) targets that may get used by other projects.

XRegistry Added to NMI Build and Test

The OGCE XRegistry service (http://www.collab-ogce.org/ogce/index.php/XRegistry) has been added to the NMI nightly build system. See http://nmi-s003.cs.wisc.edu/nmi/index.php?page=results%2Foverview&opt_project=OGCE for all builds.

OGCE Axis Services Added to Nightly Build

Nightly builds of the OGCE Axis Services (http://www.collab-ogce.org/ogce/index.php/OGCE-Axis-Services) are now running on the NMI Testbed at the University of Wisconsin. We perform automated builds for Mac OSX and several flavors of Linux. See http://nmi-s003.cs.wisc.edu/nmi/index.php?page=results%2Foverview&opt_project=OGCE for additional information.

OGCE Tutorial at SciDAC

We will participate in the "TeraGrid Gateways" portion of the SciDAC tutorials, June 19th. See https://outreach.scidac.gov/scidac09/tutorials/ for details.

OGCE Presentation at TG09

OGCE has an accepted presentation for TG09, Wednesday June 24th, 2:30-3:00 pm. See full TG09 schedule at

http://www.teragrid.org/tg09/index.php?option=com_content&task=view&id=62

TG09 Tutorial on Science Gateways

OGCE team members will give a Science Gateway tutorial at TeraGrid 09 on Monday, July 22nd. The full abstract is here: http://www.teragrid.org/tg09/index.php?option=com_content&task=view&id=56

E-Science Demos

We'll be demonstrating OGCE workflow management tools and XMC-Cat (a new cataloging service) at E-Science Conference on Wednesday night.

Abstract: http://escience2008.iu.edu/exhibits/open_grid.shtml

Coordinates: http://escience2008.iu.edu/exhibits/index.shtml

EScience Tutorial Agenda

This is Sunday, Dec 7 from 8:00 to 5:00 PM (US Eastern Standard Time). You can also view remotely the live broadcast. For instructions, see http://escience2008.iu.edu/multimedia/index.shtml. An archived broadcast will be available from the same URL.

Presentation material is available from http://www.collab-ogce.org/ogce/index.php/Tutorials.

Talks will be given by Marlon Pierce, Suresh Marru, and very special guest star Stephen Simms.

• Introduction and attendee feedback
  
• (Talk) Gateways overview (Marlon)
• (Talk) TeraGrid: Resources Overview and Getting Started (Simms)

[BREAK]

• (Talk/Demo) LEAD portal and workflow (Suresh)
• (Demo/Talk) GridChem workflows (Suresh)
• (Demo/HO) OGCE and TGUP Portals (Marlon)

[LUNCH]

• (Demo/HO) OGCE hands on and simple workflows with XBaya (Marlon, Suresh)
• (Talk/HO) Building OGCE portal and services (Marlon)
• (Talk/Demo/HO) Building GTLAB gadgets (Marlon)

[BREAK]

• (Optional Talk) Web 2.0 for science gateways (Marlon)
• (Optional HO) Building the portal, GTLAB, and services
  
• (Optional HO) Using various portals (OGCE, TeraGrid User Portal, LEAD)
  
• (Optional HO) Workflows in depth
  

Javadocs for OGCE Components

If you want to make Javadocs for OGCE components (the portal, GTLAB, Axis services, etc), use the command

mvn javadoc:javadoc -f path/to/component/pom.xml

For example, the GTLAB javadoc can be made with the command

mvn javadoc:javadoc -f jsf_standalone/pom.xml

The docs will be located in jsf_standalone/target/site. Load with your browser.

OGCE Swarm Service Talk at E-Science

Sangmi Pallickara will present the Swarm Job Submission Service at Friday, Dec 12 at the E-Science conference: http://escience2008.iu.edu/sessions/SWARM.shtml.

This talk will be web-cast. See http://escience2008.iu.edu/multimedia/index.shtml for details.

E-Science Gateways Tutorial Web Cast

Our OGCE and Gateways tutorial at E-Science is Sunday, December 7 from 8:00 am to 5:00 pm. See http://escience2008.iu.edu/tutorials/gateways.shtml. The entire conference (including our tutorial) is being web-cast; see http://escience2008.iu.edu/multimedia/index.shtml for more information.

We will post a more detailed agenda on this blog.

SC08 Demo Time Correction

The IU booth demo is 1:00-3:00 on Wednesday, November 19th. These are roughly 2 30 minute demos at 1:00 and 2:00, with 30 minute breaks.

OGCE Portal 2.4 Posted

Version 2.4 is ready. This features the File Manager applet from the TeraGrid User Portal, as well as numerous bug fixes and miscellaneous enhancements. See http://collab-ogce.blogspot.com/2008/09/tagged-preview-of-ogce-portal-24.html for more details.

Tagged SC08 releases of OGCE Axis Services and GTLAB are also available.

OGCE Web Site and Download Reorganization

The OGCE Web Site is being revised to better reflect the reorganization of our downloadable components. Here are the major download suites:

• The OGCE portlet-based Grid portal: this download combines Tomcat, Gridsphere portlet container, and many OGCE portlets developed with OGCE libraries.
• The OGCE Axis Web Services Suite: this includes several OGCE services (currently GPIR, the Resource Prediction Service, and the Resource Discovery Service) combined into an integrated, one-step build.
  
• The OGCE Workflow Suite: this suite contains workflow composer and engine (XBaya), the GFAC application factory, XRegistry, and related components.
  
• GTLAB: This is a collection of Java Server Faces tags and supporting code for building Grid portlets and gadgets. The current version supports simple tag pipelining (so you can associate multiple tasks with a single HTML button click). Tags are no longer limited to Grid clients--any thing can be used.
• OGCE Gadget Container: this is a preview of our work to build an Open Social and iGoogle-compatible container. We view this as the successor to the portlet standard we have used in the past.

See the web site for more information.

SC08 Demos: 1:00-3:00

OGCE team will be giving demos in the Indiana University booth at SC08 on Tuesday from 1:00 pm to 3:00 pm. Demos will include

• OGCE workflow tools
• JavaScript Grid clients
• Tag libraries for building iGoogle style Grid gadgets.
• Gateway information services
• Application portals using Pylons
• Demo portals for the life sciences

Please feel free to come by the IU booth for informal demos.

Updated GCE08 Schedule

The GCE08 Workshop presentation schedule at Supercomputing 2008 in Austin has been updated: http://www.collab-ogce.org/gce08/index.php/Program

Tentative GCE08 Schedule Available

A draft GCE08 Workshop presentation schedule at Supercomputing 2008 in Austin is now available: http://www.collab-ogce.org/gce08/index.php/Program

The program will be updated as we confirm speakers' travel schedules, but the program is set. Slides and papers will be made available from this URL as they become available.

The Dreaded "GRAM Job submission failed: The job manager failed to open stderr" Error

The COG will produce the following infamous error

GRAM Job submission failed: The job manager failed to open stderr

if you don't have values in $HOME/.globus/cog.properties set correctly. The "ip" parameter must be set to the real hostname or static IP address of your portal server. That is, it should be something like

ip=123.456.789

You can edit this file by hand, but note it will be overwritten if your rebuild the portal. You should make sure that the <portal.server.ip> value in ogce-portal-only/pom.xml is set correctly.

Nightly Build Update

The "portal only" release was upgraded to Tomcat 5.5.27, but the build and test scripts on the NMI testbed were still set to use the older Tomcat 5.5.12. This has been corrected.

For more info on the nightly build, see http://www.collab-ogce.org/ogce/index.php/Bugs_email_svn

Nightly Builds OK

There is a bug in the NMI nightly build scripts that prevents Tomcat from starting. This causes the build to report a failure, but the compilation steps complete successfully.

Tagged Update to GTLAB

Numerous improvements and bug fixes to the GTLAB tag libraries have been tagged as "GTLAB-2008". These supersede the TG08 tagged version. To get the updates, use

svn checkout https://svn.sourceforge.net/svnroot/ogce/tags/GTLAB-Sep2008

Major changes include

• Bug the prevented task inter-dependencies from working correctly has been fixed.
• Each GTLAB tag can now be associated with its own resource property file. Previously all tags used the same property file.
  

Tagged Preview of OGCE portal 2.4

The OGCE "portal only" release has been tagged as a preview. This version includes numerous bug fixes and updates. To get it, use

svn checkout https://svn.sourceforge.net/svnroot/ogce/tags/ogce-portal-only-2.4-preview

Most updates are courtesy of Lukas Hoffman.

---

gp-common

---

* Edited files: portlets/gp-common/src/main/webapp/css/fileBrowser.css
portlets/gp-common/src/main/webapp/javascript/fileBrowser.js
portlets/gp-common/src/main/webapp/jsp/fileBrowser.jsp
* This isn't really a portlet, it just contains files that are used by the GridPort portlets

---

Interactive Job Submission / jobsubmit-portlet

---

* Edited files: portlets/jobsubmit-portlet/src/main/webapp/templates/xportlets-jobsubmit.vm
portlets/jobsubmit-portlet/src/main/webapp/templates/xportlets-jobsubmit-help.vm
portlets/jobsubmit-portlet/src/main/java/xportlets/jobSubmitAction.java
* Added *'s for certain fields in the job submission form & a note saying those fields were required.
* Added a way to link to standard output and standard error files
* Added help instructions in "?" viewing window
* If user left stdout/stderr field blank, job output is displayed in portlet when job completes. see 'lukas code' where stdout is mentioned in JobSubmitAction.java and xportlets-jobsubmit.vm
* If user enters path from root to stdout/stderr, there's a link to download the file. This is glitchy though, it only works if filetransfer portlet is connected to correct host, and it doesn't work for relative paths.

---

File Manager/file-manager and Comprehensive File Management/comp-file-management

---

* Edited files: portlets/comp-file-management/src/main/webapp/jsp/fileBrowser.jsp
portlets/comp-file-management/src/main/webapp/jsp/view.jsp
portlets/comp-file-management/src/main/webapp/css/fileManagement.css
portlets/comp-file-management/src/main/java/edu/tacc/gridport/portlets/interactive/FileManagementConstants.java
portlets/comp-file-management/src/main/java/edu/tacc/gridport/portlets/interactive/FileManagementPortlet.java
portlets/gp-common/src/main/webapp/jsp/fileBrowser.jsp - identical to fileBrowser.jsp above
portlets/gp-common/src/main/webapp/javascript/fileBrowser.js:
* Added an image href to the file/directory listing
* Files are now sortable by name, filetype, and size. Default is sort by filetype. User can click on a heading to select a sort.
If user clicks on a heading twice in a row, the sort is in reverse order.
* Refresh button went to home directory, now it's refresh current directory.
* Fixed something in FileManagementPortlet.java but I forgot what the problem was.
destListing.setRemoteHost(destResources[0]); //lukas code, previous array size = 1 --> this portlet crashes.
//don't know why this fix works, but it works...

---

File Transfer Portlet / GridFTP Portlet / filetransfer (Deprecated)

---

Edited files: portlets/filetransfer/src/main/webapp/gridftp-browser.vm
portlets/filetransfer/src/main/webapp/gridftp-help.vm
portlets/filetransfer/src/main/java/xportlets/gridftp/FileAttributes.java
portlets/filetransfer/src/main/java/xportlets/gridftp/GridFTPClientAction.java
ogce-portal/portlets/filetransfer/src/main/java/xportlets/gridftp/FileListSorter.java
* FileListSorter.java is all my own code, so the directory contains a javadoc html page for this class
* fixed bug where file download & file transfer don't work if filenames contain spaces.
* fixed bug where clicking on a link twice really fast made portlet crash
* fixed two or three "annoying but not fatal" bugs that I don't remember now, although I probably wrote that it was a fix in the source code comments
* Added a "Type" (Filetype field) so the file list matches the information that any Windows directory listing gives you
* Made the "Date Modified" dates look better
* Files/Directories are now sortable by Name, Size, Type, and Date Modified. They stay sorted that way until the user clicks on a heading.
* Sorting is in reverse order when user clicks a heading twice.
* Made icons and file/directory names clickable, for easier access with the mouse. Before, the link to download a file was on one side & the
file name was on the other, which is annoying for the user. Now the user can click directly on the file name or icon to download it, or
directory name/icon to go there.
* Moved things around to make portlet look better. Mostly, moved all buttons and links toward the top and put file listing under them.
* User can now
make a new directory
rename a file/directory
delete 1 or more files at a time
delete a directory
go to any directory in 1 step by typing in the full path name, or path name relative to current directory, or ".." to go up a directory
(typing "dir1/dir2/dir3" and pressing submit is faster than clicking 3 links)
* Added error messages when the commands above fail & display to the user.
* Added help instructions in "?" viewing window
* Luke Hammill fixed problem where if you accidentally transfer a file to itself (same action as: mv fileA fileA) file gets deleted. (fix not in version i sent you)
* back.gif button can now go to '/', which it couldn't before.
* 'Home' link now always goes to directory that showed up when the host was first loaded, which it didn't before.
* You can delete several files at a time using checkboxes.

GCE08 Submission Deadline Extension

The deadline for submitting papers to GCE 08 has been extended to September 22. See http://www.collab-ogce.org/gce08/index.php/Main_Page for more information.

Condor Installation Example

The following is an example installation for Condor 6.8.8. This is required to use the Condor and Condor-G portlets. Additional configuration is required to enable Birdbath and Condor-G, so see the portlet notes at www.collab-ogce.org.

Note also that this version of Condor will not work on newer RHEL and Fedora releases, so you will need to get the libstdc++.so.5 library. Do the following as root:

[shell> yum install libstdc++.so.5

Now do the following in the condor-6.8.8 release directory.

[shell> condor_install

Press enter to begin Condor installation


***************************************************************************
STEP 1: What type of Condor installation do you want?
***************************************************************************

Would you like to do a full installation of Condor? [yes]

Press enter to continue.

***************************************************************************
STEP 2: How many machines are you setting up for Condor?
***************************************************************************

Are you planning to setup Condor on multiple machines? [yes] no

Press enter to continue.

***************************************************************************
STEP 3: Install the Condor "release directory", which holds
various binaries, libraries, scripts and files used by Condor.
***************************************************************************

It looks like you've installed a Condor release directory in:
/usr/local/condor
Do you want to use this release directory? [yes] no

Have you installed a release directory already? [no] no

Where would you like to install the Condor release directory?
[/home/mpierce/condor]
That directory doesn't exist, should I create it now? [yes]
Installing a release directory into /home/mpierce/condor ...

[------------------------------Stuff deleted------------------------------]

done.

Using /home/mpierce/condor as the Condor release directory.

Press enter to continue.

***************************************************************************
STEP 4: How and where should Condor send email if things go wrong?
***************************************************************************

If something goes wrong with Condor, who should get email about it?
[mpierce@gw12.quarry.iu.teragrid.org]

What is the full path to a mail program that understands "-s" means
you want to specify a subject? [/bin/mail]

Using /bin/mail to send email to mpierce@gw12.quarry.iu.teragrid.org

Press enter to continue.

***************************************************************************
STEP 5: Filesystem and UID domains.
***************************************************************************

To correctly run all jobs in your pool, including ones that aren't relinked
for Condor, you must tell Condor if you have a shared filesystem, and if
so, what machines share it.

Please read the "Configuring Condor" section of the Administrator's manual
(in particular, the section "Shared Filesystem Config File Entries")
for a complete explaination of these (and other, related) settings.

Do all of the machines in your pool from your domain ("quarry.iu.teragrid.org")
share a common filesystem? [no]

Configuring each machine to be in its own filesystem domain.

Do all of the users across all the machines in your domain have a unique
UID (in other words, do they all share a common passwd file)? [no]

Configuring each machine to be in its own uid domain.

Press enter to continue.

***************************************************************************
STEP 6: Java Universe support in Condor.
***************************************************************************

Enable Java Universe support? [yes] no
OK, Java Universe will be left unconfigured.

Press enter to continue.

***************************************************************************
STEP 7: Where should public programs be installed?
***************************************************************************

The Condor binaries and scripts are already installed in:
/home/mpierce/condor/bin
If you want, I can create some soft links from a directory that is already
in the default PATH to point to these binaries, so that Condor users do not
have to change their PATH. Alternatively, I can leave them where they are
and Condor users will have to add /home/mpierce/condor/bin
to their PATH or explicitly use a full pathname to access the Condor tools.

Shall I create links in some other directory? [yes] no

Press enter to continue.

***************************************************************************
STEP 8: What machine will be your central manager?
***************************************************************************

What is the full hostname of the central manager?
[gw12.quarry.iu.teragrid.org]

Your central manager will be on the local machine.

Press enter to continue.

***************************************************************************
STEP 9: Where will the "local directory" go?
***************************************************************************

Condor will need to create a few directories for its own use

You have a "condor" user on this machine. Do you want to put all the
Condor directories in /home/condor? [yes] no

Do you want to put all the Condor directories in
/home/mpierce/condor/home? [yes]

Creating all necessary Condor directories ... done.

Press enter to continue.

***************************************************************************
STEP 10: Where will the local (machine-specific) config files go?
***************************************************************************

Condor allows you to have a machine-specific config file that overrides
settings in the global config file.

You must specify a machine-specific config file.

Should I put a "condor_config.local" file in /home/mpierce/condor/home?
[yes]
Creating config files in "/home/mpierce/condor/home" ... done.

Configuring global condor config file ... done.
Created /home/mpierce/condor/etc/condor_config.

Press enter to continue.

Setting up gw12.quarry.iu.teragrid.org as your central manager

What name would you like to use for this pool? This should be a
short description (20 characters or so) that describes your site.
For example, the name for the UW-Madison Computer Science Condor
Pool is: "UW-Madison CS". This value is stored in your central
manager's local config file as "COLLECTOR_NAME", if you decide to
change it later. (This shouldn't include any " marks).
gw12condorthing

Setting up central manager config file /home/mpierce/condor/home/condor_config.local ... done.

Press enter to continue.

***************************************************************************
STEP 11: How do you want Condor to find its config file?
***************************************************************************

Condor searches a few locations to find it main config file. The first place
is the envionment variable CONDOR_CONFIG. The second place it searches is
/etc/condor/condor_config, and the third place is ~condor/condor_config.
/home/condor/condor_config exists.
Renaming to: /home/condor/condor_config.old.

Should I put in a soft link from /home/condor/condor_config to
/home/mpierce/condor/etc/condor_config [yes]

Press enter to continue.

***************************************************************************
Condor has been fully installed on this machine.
***************************************************************************

/home/mpierce/condor/sbin contains various administrative tools.
If you are going to administer Condor, you should probably place that
directory in your PATH.

To start Condor on any machine, just execute:
/home/mpierce/condor/sbin/condor_master

Since this is your central manager, you should start Condor here first.

Press enter to continue.

You should probably setup your machines to start Condor automatically at
boot time. If your machine uses System-V style init scripts, look in
/home/mpierce/condor/etc/examples/condor.boot
for a script that you can use to start and stop Condor.

Please read the "Condor is installed... now what?" section of the INSTALL
file for things you should do before and after starting the Condor daemons.
In particular, you might want to set up host/ip access security. See the
Adminstrator's Manual for details.

Using OGCE portlets in Gridshere 3.1 (preliminary)

We had a request to get OGCE portlets running in Gridsphere 3.1. Below is the way to do this manually. We'll try to make an automated version of this with Maven next.

0. Shutdown tomcat.

1. In the ogce-portal-only directory, run the command mvn clean install -f global-config/pom.xml

2. Remove our versions of hibernate, hsqldb, and ehcache (probably these crept in from GS 2.1--I don't think we need them). Specifically,

rm global-config/common/target/cog-common-1.0/lib/hibernate2-OGCE.jar

rm global-config/common/target/cog-common-1.0/lib/hsqldb-1.7.1.jar

rm global-config/common/target/cog-common-1.0/lib/ehcache-0.9.jar

3. Copy the remaining jars into your Tomcat's shared/lib. Something like
cp global-config/common/target/cog-common-1.0/lib/* /Users/mpierce/GridSphere31/apache-tomcat-5.5.20/shared/lib/

4. Copy the OGCE portlet as is from the ogce-portal-only dir. That is, from ogce-portal-only, run "mvn clean install -f portlets/proxymanager-portlet" and then
cp -r portal_deploy/apache-tomcat-5.5.12/webapps/proxymanager-portlet/ /Users/mpierce/GridSphere31/apache-tomcat-5.5.20/webapps/proxymanager-portlet

5. Create empty files named after the portlets for Gridsphere. This used to be done in $CATALINA_HOME/webapps/gridsphere/WEB-INF/CustomPortal/portlets

Now these apparently go in $HOME/.gridsphere, so do this:

touch ~/.gridsphere/portlets/proxymanager-portlet.2

6. Edit the proxymanager's web.xml file to use the correct namespace for the PortletServlet (i.e., remove the "gridlab" section from the full name):

<servlet>
<servlet-name>PortletServlet</servlet-name>
<servlet-class>org.gridsphere.provider.portlet.jsr.PortletServlet</servlet-class>
</servlet>

7. In proxymanager-portlet/WEB-INF/lib, delete gridsphere-ui-tags-2.1.jar

8. Start tomcat and add the portlet using the Gridsphere Layout Manager.

GCE08 Workshop at Supercomputing 2008

We are hosting GCE08, the fourth workshop in this series. Papers on portals, gateways, and other scientific user environments are welcome. For more information, see

http://www.collab-ogce.org/gce08/index.php/Main_Page

New GPIR Service URL for GPIR Portlet

The old TeraGrid GPIR service is being retired and replaced. To point to the new server, edit GPIR's portlet.xml file and replace

http://alpine.tacc.utexas.edu:8080/gpir/webservices

with

http://laredo.tacc.utexas.edu:8080/gpir/webservices

To rebuild this portlet, use the command "mvn clean install -f portlets/gpir-portlet/pom.xml" from the "ogce-portal-only" directory.

Correction to OGCE 2.3 tar download

The default value for the property <project.home> in ogce-portal-only/pom.xml was incorrectly set. While this can be edited manually, it prevents the portal from building out of the box if one follows the documentation.

Also, the obsolete filetransfer module, which causes the errors described at http://collab-ogce.blogspot.com/2007/12/filetransfer-portlet-build-error.html on some machines, has been removed.

These have been corrected in the download link at http://www.collab-ogce.org/. The version number has been incremented to 2.3.1 to reflect this minor update.

GTLAB TG08 Tutorial Release

The TG08 tutorial release of GTLAB is now available from http://www.collab-ogce.org/ogce/index.php/GTLAB.

GTLAB provides Java Server Faces components and backing beans that simplify the process of Grid Web application development. You can use GTLAB to develop standalone Web applications, portlets, and Google Gadgets.

OGCE Portal Release 2.3 Available

OGCE portal release 2.3 is available from http://www.collab-ogce.org/ogce/index.php/Portal_download. This release features major revisions to the XBaya workflow composer as well as minor bug fixes to GPIR and the build system.

The XBaya updates are part of the comprehensive enhancements to the OGCE workflow tools, http://www.collab-ogce.org/ogce/index.php/Workflow.

TG08 Gateways Summary

Courtesy of Nancy Wilkins-Diehr:

For those attending TeraGrid 08, June 9-13 in Las Vegas we have a
terrific representation of Science Gateways at the conference.

Gateway-related topics are featured in:
14 papers
7 demonstrations
5 posters
5 birds-of-a-feather sessions
3 student competition entries
2 tutorials
1 vis gallery entry
And a face-to-face gateway meeting in a pear tree on Thursday June 12,
3pm.

Full information on the conference can be found at
http://www.tacc.utexas.edu/tg08/ and a listing of gateway-specific
activities at
<http://www.teragridforum.org/mediawiki/index.php?title=Science_Gateways
#Gateways_at_TeraGrid_08>.

Thank you to the many of you on this list who are contributing to making
the conference a success.  I hope to see many of you in Las Vegas next
week.

TeraGrid 08 OGCE Tutorial Outline

Tutorial is Monday, June 9. We will make slides and material available for download.

8:30-9:30: Intro, building and using the portal, writing simple grid portlets (Marlon Pierce)

9:30-10:00: Workflow suite part 1 (Suresh Marru, Gopi Kandaswamy)

10:00-10:30: break

10:30-11:15: Worflow suite part 2 (Suresh, Gopi)

11:15-11:45: Web 2.0 for gateways (Gregor von Laszewski)

11:45-12:00: Teragrid gateway authorization requirements and GridShib (Tom Scavo)

OGCE Tutorial at TeraGrid 2008

We will give a 1/2 day tutorial on June 9th at TeraGrid 2008 in Las Vegas. For more information, see http://www.tacc.utexas.edu/tg08/index.php?m_b_c=scienceGateways.

Topics will include building and using the basic portal, using GTLAB to make Grid Web clients, using the OGCE workflow suite of tools to wrap science applications, and Web 2.0 activities for science gateways.

GPIR Service Download Available

An updated version of the Grid Portal Information Repository (GPIR) Web Service is available for download. This is a standalone repackaging of the GPIR Web Service in older OGCE portal releases.

The GPIR service powers the GPIR portlet in the main portal download. Get the GPIR service if you want to provide your own status information server for your grid.

http://www.collab-ogce.org/ogce/index.php/GPIR

Mac OSX Maven Build Errors

We have reports of the following build error on Macs:

*******************************
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Error building POM (may not be this project's POM).


Project ID: org.apache.maven:maven-parent:pom:5

Reason: Cannot find parent: org.apache:apache for project: org.apache.maven:maven-parent:pom:5 for project org.apache.maven:maven-parent:pom:5

*******************************

Workaround: Download and install the Maven third party jars as described in the "Download" section.

The error does not appear in the Mac nightly build dashboard, so we are trying to discover the cause.

Example OGCE Portal

A running, persistent version of the portal software is available from

https://ogceportal.iu.teragrid.org:8443/gridsphere/gridsphere

To log in, you can use the account guest/guest123. If you have a TeraGrid account and credentials in the TeraGrid MyProxy server, you can try out the user interfaces.

To get a personal account or to use this portal with other Grid deployments, contact us for assistance.

Bug Fix in GPIR Portlet

There is a minor bug fix in the GPIR portlet. Clicking machine name links in the portal served from Linux hosts produced an error because of a missing XML parsing library. The fix is available in SVN.

To update existing portals (versions 2.2.0 and 2.2.1), simply replace ogce-portal-only/portlets/gpir-portlet/pom.xml with the update from

http://ogce.svn.sourceforge.net/viewvc/ogce/ogce-portal-only/portlets/gpir-portlet/pom.xml?view=log

Spurious Tomcat Startup Errors

When starting the portal server for the first time, you will notice several errors (see below for a sample) if you are using our startup-tomcat.sh script. These are the standard errors produced by Tomcat's shutdown.sh if no server is running and can be ignored.

Our startup-tomcat.sh script first calls Tomcat's shutdown.sh before calling startup.sh. We do this to prevent accidentally starting two servers that try to use the same ports. This can be a little tricky to clean up (Tomcat's shutdown.sh may not work and you may have to use the kill command).

-bash-3.00$ ./startup-tomcat.sh
####################################
# Starting up Portal Tomcat... #
####################################
Using CATALINA_BASE: /usr/local/dev/maytal/ogce-portal-only/portal_deploy/apache-tomcat-5.5.12
Using CATALINA_HOME: /usr/local/dev/maytal/ogce-portal-only/portal_deploy/apache-tomcat-5.5.12
Using CATALINA_TMPDIR: /usr/local/dev/maytal/ogce-portal-only/portal_deploy/apache-tomcat-5.5.12/temp
Using JRE_HOME: /usr/local/dev/maytal/jdk1.5.0_14
Jan 28, 2008 12:48:29 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:520)
at java.net.Socket.connect(Socket.java:470)
at java.net.Socket.(Socket.java:367)
at java.net.Socket.(Socket.java:180)
at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:394)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:324)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:415)
viUsing CATALINA_BASE: /usr/local/dev/maytal/ogce-portal-only/portal_deploy/apache-tomcat-5.5.12
Using CATALINA_HOME: /usr/local/dev/maytal/ogce-portal-only/portal_deploy/apache-tomcat-5.5.12
Using CATALINA_TMPDIR: /usr/local/dev/maytal/ogce-portal-only/portal_deploy/apache-tomcat-5.5.12/temp
Using JRE_HOME: /usr/local/dev/maytal/jdk1.5.0_14

FileTransfer Portlet Build Error

We have gotten some reports that the filetransfer portlet does not compile correctly on some machines. You will get an error like the one below.


[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure
/opt/ogce-portal-only/portlets/filetransfer/src/main/java/xportlets/gridftp/GridFTPClientAction.java:[1046,10] putFile(java.lang.String,java.lang.String) in org.globus.cog.abstraction.interfaces.FileResource cannot be applied to (java.lang.String,java.lang.String,)


This portlet is obsolete and uses an obsolete Java COG API, so it should be disabled. As a workaround, simply delete or comment out the line

<module>portlets/filetransfer</module>

from ogce-portal-only/pom.xml. We will provide a minor update release to fix this bug.

Expeditious Job Submission with Condor Birdbath

You need to call the rescheduler to force jobs to run right away when using Birdbath. Code snippet is

....
xact.submit(clusterId, jobId, userName, universeType,
 executable,arguments,"(TRUE)",
 extraAttributes, null);
xact.commit();

//Magic line is below.
schedd.requestReschedule();

This assumes you have otherwise edited condor_config to run jobs right away (changed UWCS to TESTINGMODE). See the OGCE condor-job-submission portlet for the full code. Thanks to Sangmi for pointing this out.

Link to CIMA Production Portal

This is the production version of the portal: http://cimaportal.indiana.edu:8080/gridsphere/gridsphere

Google finds a few other versions floating around, and so I'm trying to increase this URL's page rank.

OGCE Portal 2.2 Released

Version 2.2 of the portal is now available. We have upgraded the RC5 release candidate to a full release. See http://www.collab-ogce.org/ogce/index.php/Portal_download.

Updated COG GT4 Provider Jar

The GT4 provider in the COG kit needs some configuration and property changes to work with the portal. This is now in Maven:

http://www.collab-ogce.org/maven/cog-gt4-OGCE/jars/cog-provider-gt4_0_0-2.4.jar

To see the differences, unpack this jar and compare the files

• cog-provider.properties
• classloader-gt4_0_0.properties

with those in the Java COG download.

Since this is now working with WS-GRAM services on the TeraGrid, I'll now roll the final Release Candidate for version 2.2 of the portal.

GCE07 and SciNET SC07 Google Capture

GCE07 made the SciNET Google search capture list. I'm not sure if this is good or bad.

http://scinet.sc07.org/security/googlesearches07.html

See also the link's parent for general information--check to see if your password was detected.

OGCE Web Site Updated

The revised OGCE web site (using MediaWiki) is now available. The Web server should redirect from http://www.collab-ogce.org/ogce. If you do not see the new site, trying clearing your browser cache.

NMI Testbed Dashboard

Here's an updated link to the nightly build dashboard on the NMI testbed. We still have some problems finding java stuff on several machines, but this should be corrected soon.

Indiana University's SC07 Booth Schedule

Portals and gateways are Tuesday: http://supercomputing.iu.edu

GCE07 Workshop Program Available

The preliminary program for GCE07 at Supercomputing 2007 is available from http://www.collab-ogce.org/gce07/index.php/Program.

File Agent Service and Community File Manager

We have developed a File Manager portlet and agent service that helps manage file access in TeraGrid community accounts. This will be part of the OGCE SC07 release and is available in SVN now. See http://sangpall.blogspot.com/ for more information.

More on New Build Process

Here are the Unix commands to use the revised build:

svn co https://ogce.svn.sourceforge.net/svnroot/ogce/ogce-portal-only

cd ogce-portal-only

(Edit properties section of ogce-portal-only/pom.xml as necessary).

tar -zxf maven-2.0.7-bin.tar.gz

export PATH=`pwd`/maven-2.0.7/bin/:$PATH

mvn clean install

The portal stuff is installed under portal_deploy/apache-5.5.12. Use the little convenient scripts tomcat-startup.sh and tomcat-shutdown.sh to start and stop the server.

This needs to be tested on Windows.

New Build Process

The old shell script+Ant+Maven1+Maven2 build process is being replaced with a clean Maven 2 build. This is currently in SVN and will be included in the next release.

The base directory structure is now

• portal_deploy: this contains the apache server and is location of the portal after deployment.
  
• global-config: this contains Maven poms for deploying the Java COG jars into tomcat's shared/lib.
  
• portlets: this contains the portlets. Each portlet is in a separate directory.
  
• certificates: this installs the certificates into $HOME/.globus/certificates
  
• containers: this includes various JSR 168 containers. For now, this only contains Gridsphere 2.1.5 but alternative versions of Gridsphere, Sakai, et al will be placed here.
  
• lnf_portal: this directory includes container specific goals for customizing the look and feel of the logo. This includes logos, style sheets, etc.

The entire portal is installed with the command

mvn install

The portal can be cleaned completely with

mvn clean

To update an existing deployment (for example, reinstalling the filetransfer portlet), use the following command:

mvn clean install -f portlets/filetransfer/pom.xml

Re-compiling individual portlets

0. Set your $MAVEN_HOME variable, if necessary:

export MAVEN_HOME=$HOME/ogce2-x.y/third_party_tools/maven-1.0.2

1. cd to ogce2-x.y/ogce-portal

2. Then run this command below to recompile the :

$HOME/ogce2-x.y/third_party_tools/maven-1.0.2/bin/maven gp:deploy-portlet -Dportlet.name=filetransfer -Denv.CATALINA_HOME=../portal_deploy/apache-tomcat-5.5.12

The value for -Dportlet-name=... should be the specific subdirectory name of ogce-portal/portlets (ie use -Dportlet-name=proxymanager-portlet to compile the proxy manager.

GTLAB 1.0 Beta Available

Grid Tag Libraries and Beans (GTLAB) version 1.0-Beta is available for portlet development. These are extensions to JSF to simplify the development of complicated Grid portlets for invoking science applications.

http://grids.ucs.indiana.edu/users/manacar/GTLAB-website/

OGCE2 2.1 Available for Download

We are pleased to announce the latest update to the OGCE Grid Portal release. Release 2.1 updates version 2.0.4 with support for Condor-G portlets and various minor bug and build system fixes. Downloads are available from http://www.collab-ogce.org/ogce2/

OGCE grid portlets include support for Globus GRAM and WS-GRAM, GridFTP, SRB, MyProxy, PURSe, Condor, GPIR information services, and Sakai collaboration services.

Using Condor-G Portlets

The Condor-G portlet in the next OGCE release requires a few external steps.

1. You must install condor (version 6.8), globus, and the OGCE portal server on the same host computer.
2. You must enable Condor birdbath, as documented on the OGCE web site, http://www.collab-ogce.org/ogce2/condor-job-submission/configuration.html.
3. You must ensure that Condor-G is enabled in your condor installation. This is probably the case, but see instructions at http://communitygrids.blogspot.com/2006/08/about-condor.html.
4. You must construct and publish classads for the Grid resources you want to use. The remainder of this post provides basic instructions.
   

Condor-G uses Condor to act as a client to Globus services. Normally, Condor daemons collect information from the entire condor flock and automatically construct Classads that are used to matchmake your job request to a particular machine. Since Globus services are not part of the flock, you must manually add these to condor on your portal server.

Here is a sample classad for condor-g:

MyType = "Machine"
TargetType = "Job"
Name = "condorTest05"
Machine = "condorTest05"
gatekeeper_url = "login-co.ncsa.teragrid.org/jobmanager"
UpdatesSequenced = 9
CurMatches = 0
Requirements = TARGET.JobUniverse == 9
Rank = 0.000000
CurrentRank = 0.000000
OpSys = "LINUX"
Arch = "INTEL"
State = "Unclaimed"
Activity = "Idle"
LoadAvg = 0.000000
Memory = 2048
WantAdRevaluate = True
StartdIpAddr = "129.79.216.5"

Replace parameters above as appropriate. The StartdIpAddr should be the IP of your portal+condor-g host machine. All other parameters describe the Globus host machine, login-co.ncsa.teragrid.org. Place these values in a file, myCondorClassAd.txt, and then publish this using the command line

[shell-prompt> condor_advertise UPDATE_STARTD_AD myCondorClassAd.txt

This classad will expire in a few minutes, so you will need to execute the above command using a cron script. Rumor has it that better tools for supporting Globus-derived classads exist.

GFac: Wrapping Science Applications as Grid Services

The OGCE is pleased to announce the public release of the Generic Application Service Factory (a.k.a., GFac). For documentation, see

http://www.extreme.indiana.edu/gfac/userguide.html

The code can be downloaded from

http://www.extreme.indiana.edu/gfac/dist/

WS-GRAM Job Submission Configuration

You must provide the port number (typically 8443) for the WS-GRAM service location in your grid.installation.properties. A valid entry will be formatted like so:

firsthost.testinggrid.com:8443


If you omit the port number for a WS-GRAM service, the service invocation will work the first time only and fail on all subsequent invocations. The error message will be or will contain

Error: org.globus.cog.abstraction.impl.common.task.TaskSubmissionException:
Cannot submit job: ; nested exception is: java.net.ConnectException:
Connection refused

Pom.xml Problem in Condor Portlet

The condor job submission portlet contains a call to an obsolete plugin repository that will cause compilation errors of this portlet. To solve in release version 2.0.4, comment out the plugin section near the top of the ogce2-2.0/ogce-portal/portlets/condor-job-submission2/pom.xml as shown.



<build>
<finalName>condor-job-submission2</finalName>
<plugins>
<!--
<plugin>
<groupId>org.mortbay.jetty</groupId>
<artifactId>maven-jetty6-plugin</artifactId>
<configuration>
<scanIntervalSeconds>10</scanIntervalSeconds>
</configuration>
</plugin>
-->
</plugins>
</build>

Sakai JSR 168 Portlet Support

Sakai support for JSR 168 is in the Sakai SVN trunk. For notes on checking it out, compiling, and using it, see http://communitygrids.blogspot.com/2007/03/sakai-jsr-168-tests.html. This will be officially part of the Sakai 2.4 release.

Sakai uses Pluto 1.1 for portlet support. OGCE grid portlet support in Sakai is currently being tested.

See http://www.flickr.com/photos/dr-chuck/420092745/ for a Google Map portlet photo (built with JSF and using the Apache JSF portlet bridge).

OGF 19 Forum Slides

OGCE presentation slides from the Open Grid Forum 19's Software Forum are available below.

• Introduction and Overview Slides by Marlon Pierce (PPT).
  


• Wrapping Scientific Applications as Web Services by Gopi Kandaswamy (PPTX). A PDF version of this is available here.
  

Correction to tar download of version 2.0.4

The file

ogce2-2.0/ogce-portal/portlets/condor-job-submission/project.properties

has a typo that will cause the install script to fail if you do not also install the Maven jar repository. This has been corrected, but we will not immediately increment the version to 2.0.5.

OGCE Maven 1 and 2 Repository URLs

Here are the URLs for OGCE's Maven 1 and Maven 2 repositories:

• Maven 1: http://www.collab-ogce.org/maven/.
• Maven 2: http://www.collab-ogce.org/maven2/.