Condor week summary - Revision history

Tanthony@uab.edu: Completed page

2012-05-08T20:27:31Z

Completed page

← Older revision		Revision as of 20:27, 8 May 2012
Line 204:		Line 204:
	Day 2 talk were focused on use cases of Condor at different institutions, research labs, and companies and their specific implementations for the same.		Day 2 talk were focused on use cases of Condor at different institutions, research labs, and companies and their specific implementations for the same.

	===Session 1:===		===Session 1===
	* Brookhaven National Lab: Virtualization		* Brookhaven National Lab: Virtualization
	* UAB: Pilot		* UAB: Pilot
Line 210:		Line 210:
	* RENCI: Condor in networked clouds (ORCA-Open Resource Control Architecture)		* RENCI: Condor in networked clouds (ORCA-Open Resource Control Architecture)

	===Session 2:===		===Session 2===
	* Redhat: Redhat and Condor Developer community (MRG- Messaging, realtime, grid)		* Redhat: Redhat and Condor Developer community (MRG- Messaging, realtime, grid)
	* The Hartford: GPU computing with Condor (250 M2070, two pools, Windows, ~7000 cores, actuarial, financial modelling, everything written in CUDA in-house, 40x-60x improvement)		* The Hartford: GPU computing with Condor (250 M2070, two pools, Windows, ~7000 cores, actuarial, financial modelling, everything written in CUDA in-house, 40x-60x improvement)
Line 216:		Line 216:
	* Aptina: Condor in a 24x7 manufacturing environment		* Aptina: Condor in a 24x7 manufacturing environment

	===Session 3:===		===Session 3===
	* UW-Madison: Deep Linguistic Processing with Condor (ad-hoc Map reduce cluster, daily crawl onto millions of webpages using condor, making logical relations)		* UW-Madison: Deep Linguistic Processing with Condor (ad-hoc Map reduce cluster, daily crawl onto millions of webpages using condor, making logical relations)
	* U.S. Geologic Survey: Hydrological Modelling		* U.S. Geologic Survey: Hydrological Modelling
Line 223:		Line 223:
	* OSMOSIX: Condor integrated Cloud bursting (brain imaging applications)		* OSMOSIX: Condor integrated Cloud bursting (brain imaging applications)

	===Session 4:===		===Session 4===
	* MIT: MW for mixed integer non -linear problems (Master worker, branch and bound, Coupe-COUenne Parallel Extensions)		* MIT: MW for mixed integer non -linear problems (Master worker, branch and bound, Coupe-COUenne Parallel Extensions)
	* Notre Dame: Compiling and linking workflows (Student of Dough Thain, working off Makeflow, Weaver,Abstraction: simple programming interface, hides details of distributed system, Workflows: exploit natural parallelism, large applications, abstractions DAG)		* Notre Dame: Compiling and linking workflows (Student of Dough Thain, working off Makeflow, Weaver,Abstraction: simple programming interface, hides details of distributed system, Workflows: exploit natural parallelism, large applications, abstractions DAG)
Line 232:		Line 232:

	==Day 3: Talks==		==Day 3: Talks==
	===Session 1:===		===Session 1===
	* Fermilab: FermiCloud--Dynamic Resource Provisioning (High availability, Open Nebula-cloud 2.0 command line launch		* Fermilab: FermiCloud--Dynamic Resource Provisioning (High availability, Open Nebula-cloud 2.0 command line launch
	Virtual machines distributed via SCP		Virtual machines distributed via SCP
Line 241:		Line 241:
	* Fermilab: OSG (background, resources, provisioning, opportunistic use, HTC in parallel)		* Fermilab: OSG (background, resources, provisioning, opportunistic use, HTC in parallel)

	===Session 2: ===		===Session 2 ===
	* Cycle Computing: Cycle computing (Condor or the cloud, 50000 core cluster, commercial use)		* Cycle Computing: Cycle computing (Condor or the cloud, 50000 core cluster, commercial use)
	* Morgridge Institute for Research: NGS (genome sequencing, interest in Galaxy)		* Morgridge Institute for Research: NGS (genome sequencing, interest in Galaxy)
Line 247:		Line 247:
	* Embraer: Condor at Embraer (use care, muultiphysics, additional scripts)		* Embraer: Condor at Embraer (use care, muultiphysics, additional scripts)

	===Session 3:===		===Session 3===
	* UCSD: CMS - move requirements out of users hands		* UCSD: CMS - move requirements out of users hands
	* Dreamworks: Condor at Dreamworks (Condor render farms, each film -5yrs, 65 Million CPU hours, 200TB data, 500+ million files)		* Dreamworks: Condor at Dreamworks (Condor render farms, each film -5yrs, 65 Million CPU hours, 200TB data, 500+ million files)
Line 254:		Line 254:
	* Condor Project: Love Preemption		* Condor Project: Love Preemption

	===Session 4:===		===Session 4===
	* Purdue: BLAST and Bioinformatics on DiaGrid (use case)		* Purdue: BLAST and Bioinformatics on DiaGrid (use case)
	* U. Nebraska Lincoln: Putting Condor in a container (Cgroups, namespaces)		* U. Nebraska Lincoln: Putting Condor in a container (Cgroups, namespaces)
Line 263:		Line 263:

	==Day 4: Discussion Panels ==		==Day 4: Discussion Panels ==

			=== Panel 1: Trust but verify===
			Monitoring the pool, scheduling policy, health, botnets?
			Reboots each day and match the software image they are supposed to run, and reimage it automatically if it is not exact..

			Hash of the operating system.

			hold after too many restarts…

			Cloud execute nodes is a fresh nodes..

			Make job not tun on the host it failed on last.. (Requirements)

			Chirp command: to write back to the logs..

			Exploratory activity into jobs.. to see if everything is running fine with the pool…

			Botnets: shared port daemon

			Condor security mechanism: slot user, privilege separation, limit number of services on the machine, lot of watching…

			insulate the users and only run blessed applications.. windows without shared file systems..

			password authentication between nodes, encryption between nodes,

			monitor the thing you provision


			=== Panel 2: Feedback Forum===

			How-to recipies on the Wiki

			Release cycle: 1 developer release 1 stable release


			=== Panel 3: Growing your Condor Pool===
			Strategies and challenges for growing the size of your Condor installation

			Growing condor pool: flocking is best bet, BOSCO, CMS

			advertise pool utilization, and work done hours used…

			Carrot and stick approach for soliciting participation at your organization

Tanthony@uab.edu at 20:13, 8 May 2012

2012-05-08T20:13:50Z

← Older revision		Revision as of 20:13, 8 May 2012
Line 204:		Line 204:
	Day 2 talk were focused on use cases of Condor at different institutions, research labs, and companies and their specific implementations for the same.		Day 2 talk were focused on use cases of Condor at different institutions, research labs, and companies and their specific implementations for the same.

	Session 1:		===Session 1:===
	* Brookhaven National Lab: Virtualization		* Brookhaven National Lab: Virtualization
	* UAB: Pilot		* UAB: Pilot
Line 210:		Line 210:
	* RENCI: Condor in networked clouds (ORCA-Open Resource Control Architecture)		* RENCI: Condor in networked clouds (ORCA-Open Resource Control Architecture)

	Session 2:		===Session 2:===
	* Redhat: Redhat and Condor Developer community (MRG- Messaging, realtime, grid)		* Redhat: Redhat and Condor Developer community (MRG- Messaging, realtime, grid)
	* The Hartford: GPU computing with Condor (250 M2070, two pools, Windows, ~7000 cores, actuarial, financial modelling, everything written in CUDA in-house, 40x-60x improvement)		* The Hartford: GPU computing with Condor (250 M2070, two pools, Windows, ~7000 cores, actuarial, financial modelling, everything written in CUDA in-house, 40x-60x improvement)
Line 216:		Line 216:
	* Aptina: Condor in a 24x7 manufacturing environment		* Aptina: Condor in a 24x7 manufacturing environment

	Session 3:		===Session 3:===
	* UW-Madison: Deep Linguistic Processing with Condor (ad-hoc Map reduce cluster, daily crawl onto millions of webpages using condor, making logical relations)		* UW-Madison: Deep Linguistic Processing with Condor (ad-hoc Map reduce cluster, daily crawl onto millions of webpages using condor, making logical relations)
	* U.S. Geologic Survey: Hydrological Modelling		* U.S. Geologic Survey: Hydrological Modelling
Line 223:		Line 223:
	* OSMOSIX: Condor integrated Cloud bursting (brain imaging applications)		* OSMOSIX: Condor integrated Cloud bursting (brain imaging applications)

			===Session 4:===
	Session 4:
	* MIT: MW for mixed integer non -linear problems (Master worker, branch and bound, Coupe-COUenne Parallel Extensions)		* MIT: MW for mixed integer non -linear problems (Master worker, branch and bound, Coupe-COUenne Parallel Extensions)
	* Notre Dame: Compiling and linking workflows (Student of Dough Thain, working off Makeflow, Weaver,Abstraction: simple programming interface, hides details of distributed system, Workflows: exploit natural parallelism, large applications, abstractions DAG)		* Notre Dame: Compiling and linking workflows (Student of Dough Thain, working off Makeflow, Weaver,Abstraction: simple programming interface, hides details of distributed system, Workflows: exploit natural parallelism, large applications, abstractions DAG)
Line 233:		Line 232:

	==Day 3: Talks==		==Day 3: Talks==
			===Session 1:===
	Session 1:
	* Fermilab: FermiCloud--Dynamic Resource Provisioning (High availability, Open Nebula-cloud 2.0 command line launch		* Fermilab: FermiCloud--Dynamic Resource Provisioning (High availability, Open Nebula-cloud 2.0 command line launch
	Virtual machines distributed via SCP		Virtual machines distributed via SCP
Line 243:		Line 241:
	* Fermilab: OSG (background, resources, provisioning, opportunistic use, HTC in parallel)		* Fermilab: OSG (background, resources, provisioning, opportunistic use, HTC in parallel)

	Session 2:		===Session 2: ===
	* Cycle Computing: Cycle computing (Condor or the cloud, 50000 core cluster, commercial use)		* Cycle Computing: Cycle computing (Condor or the cloud, 50000 core cluster, commercial use)
	* Morgridge Institute for Research: NGS (genome sequencing, interest in Galaxy)		* Morgridge Institute for Research: NGS (genome sequencing, interest in Galaxy)
Line 249:		Line 247:
	* Embraer: Condor at Embraer (use care, muultiphysics, additional scripts)		* Embraer: Condor at Embraer (use care, muultiphysics, additional scripts)

	Session 3:		===Session 3:===
	* UCSD: CMS - move requirements out of users hands		* UCSD: CMS - move requirements out of users hands
	* Dreamworks: Condor at Dreamworks (Condor render farms, each film -5yrs, 65 Million CPU hours, 200TB data, 500+ million files)		* Dreamworks: Condor at Dreamworks (Condor render farms, each film -5yrs, 65 Million CPU hours, 200TB data, 500+ million files)
Line 256:		Line 254:
	* Condor Project: Love Preemption		* Condor Project: Love Preemption

	Session 4:		===Session 4:===
	* Purdue: BLAST and Bioinformatics on DiaGrid (use case)		* Purdue: BLAST and Bioinformatics on DiaGrid (use case)
	* U. Nebraska Lincoln: Putting Condor in a container (Cgroups, namespaces)		* U. Nebraska Lincoln: Putting Condor in a container (Cgroups, namespaces)
Line 262:		Line 260:
	* Condor Project: Use of Cisco UCS		* Condor Project: Use of Cisco UCS
	* Condor Project: What's new and Whats coming (v7.8, dynamic partition-able slots, statistics, IPv6, support SGE, Globus, resource management)		* Condor Project: What's new and Whats coming (v7.8, dynamic partition-able slots, statistics, IPv6, support SGE, Globus, resource management)








	==Day 4: Discussion Panels ==		==Day 4: Discussion Panels ==

Tanthony@uab.edu: /* Day 2: Talks */

2012-05-08T20:11:15Z

Day 2: Talks

← Older revision		Revision as of 20:11, 8 May 2012
Line 226:		Line 226:
	Session 4:		Session 4:
	* MIT: MW for mixed integer non -linear problems (Master worker, branch and bound, Coupe-COUenne Parallel Extensions)		* MIT: MW for mixed integer non -linear problems (Master worker, branch and bound, Coupe-COUenne Parallel Extensions)
	* Notre Dame: Compiling and linking workflows (Student of Dough Thain, working off Makeflow, Weaver,Abstraction: simple programming interface, hides details of distributed system		* Notre Dame: Compiling and linking workflows (Student of Dough Thain, working off Makeflow, Weaver,Abstraction: simple programming interface, hides details of distributed system, Workflows: exploit natural parallelism, large applications, abstractions DAG)
	Workflows: exploit natural parallelism, large applications, abstractions DAG)
	* Information Sciences Institute: Pegasus for large workflows		* Information Sciences Institute: Pegasus for large workflows
	* Fermilab: GlideinWMS in the cloud		* Fermilab: GlideinWMS in the cloud
	* U.ChicagoL Cloud based services		* U.ChicagoL:Cloud based services
	* U.Of Buffalo: Stork-Cloud hosted data transfer optimization		* U.Of Buffalo: Stork-Cloud hosted data transfer optimization


	==Day 3: Talks==		==Day 3: Talks==

Tanthony@uab.edu at 20:10, 8 May 2012

2012-05-08T20:10:38Z

@@ Line 198: / Line 198: @@
 Ganglia Plugin
@@ Line 221: / Line 218: @@
 Session 3:
 * UW-Madison: Deep Linguistic Processing with Condor (ad-hoc Map reduce cluster, daily crawl onto millions of webpages using condor, making logical relations)
-* U.S. Geologic Survey: Hydrologic Modelling
+* U.S. Geologic Survey: Hydrological Modelling
 * UW-Madison: Machine design optimization
 * UW-Madison: JMAG and Condor(multiphysics)
@@ Line 229: / Line 226: @@
 Session 4:
 * MIT: MW for mixed integer non -linear problems (Master worker, branch and bound, Coupe-COUenne Parallel Extensions)
-* Notre Dame: Compiling and linking workflows (Student of Dough Thain, workign off Makeflow, Weaver,Abstraction: simple programming interface, hides details of distributed system
+* Notre Dame: Compiling and linking workflows (Student of Dough Thain, working off Makeflow, Weaver,Abstraction: simple programming interface, hides details of distributed system
 Workflows: exploit natural parallelism, large applications, abstractions DAG)
 * Information Sciences Institute: Pegasus for large workflows
@@ Line 238: / Line 235: @@
 ==Day 3: Talks==
 ==Day 4: Discussion Panels ==

Tanthony@uab.edu at 19:49, 8 May 2012

2012-05-08T19:49:58Z

@@ Line 212: / Line 212: @@
 * Syracuse university: Virtualized desktop grid (Condor VM coordinator, runs as non privileged user, distrust of 3rd party application i.e. Condor, uses a MS task scheduler)
 * RENCI: Condor in networked clouds (ORCA-Open Resource Control Architecture)
 Session 2:
@@ Line 219: / Line 218: @@
 * Pacific Life: MoSes on Condor (actuarial, insurance statistics modelling, no GPU's yet, excess computing moved to Amazon EC2)
 * Aptina: Condor in a 24x7 manufacturing environment
 ==Day 3: Talks==

Tanthony@uab.edu: /* Day 2: Talks */ first part

2012-05-08T19:37:55Z

Day 2: Talks: first part

← Older revision		Revision as of 19:37, 8 May 2012
Line 205:		Line 205:
	==Day 2: Talks==		==Day 2: Talks==

			Day 2 talk were focused on use cases of Condor at different institutions, research labs, and companies and their specific implementations for the same.

			Session 1:
			* Brookhaven National Lab: Virtualization
			* UAB: Pilot
			* Syracuse university: Virtualized desktop grid (Condor VM coordinator, runs as non privileged user, distrust of 3rd party application i.e. Condor, uses a MS task scheduler)
			* RENCI: Condor in networked clouds (ORCA-Open Resource Control Architecture)


			Session 2:
			* Redhat: Redhat and Condor Developer community (MRG- Messaging, realtime, grid)
			* The Hartford: GPU computing with Condor (250 M2070, two pools, Windows, ~7000 cores, actuarial, financial modelling, everything written in CUDA in-house, 40x-60x improvement)
			* Pacific Life: MoSes on Condor (actuarial, insurance statistics modelling, no GPU's yet, excess computing moved to Amazon EC2)
			* Aptina: Condor in a 24x7 manufacturing environment

	==Day 3: Talks==		==Day 3: Talks==

Tanthony@uab.edu: /* Configuration of Partition-able slots: Greg Thain */

2012-05-08T15:31:09Z

Configuration of Partition-able slots: Greg Thain

← Older revision		Revision as of 15:31, 8 May 2012
Line 180:		Line 180:

	===Configuration of Partition-able slots: Greg Thain===		===Configuration of Partition-able slots: Greg Thain===
			Out of box 8core 8GB (1GB/core) default does not work because of different memory requirements

			Static requirements work but still not better (wait for entire machine to be free and reinitialize the entire machine as single slot)
			https://condor-wiki.cs.wisc.edu/index.cgi/wiki?p=WholeMachineSlots

			partitionable slot: introduced v7.2

			8cores 8GB, can be partitioned as
			1core + 4GB and the remaining 4 GB distributed among the other 7cores (585.14 MB) or any way the user wants to partition the slot..

			The slots are De-fragmented occasionally so that jobs with different requirements can run.

	===Condor Statistics on your submit Node: TJ Knoeller, Ken Hanh, Becky Gietzel===		===Condor Statistics on your submit Node: TJ Knoeller, Ken Hanh, Becky Gietzel===

Tanthony@uab.edu: completed tutorial talks page

2012-05-08T15:22:57Z

completed tutorial talks page

@@ Line 146: / Line 146: @@
 Debug level: D_FULLDEBUG D_COMMAND
 </pre>
 ==Day 2: Talks==

Tanthony@uab.edu: added talk 2,3 and 4

2012-05-08T14:56:45Z

added talk 2,3 and 4

@@ Line 55: / Line 55: @@
 use macro subs: %this gives the process id..
 InitialDir=run_$(Process)
 ==Day 2: Talks==

Tanthony@uab.edu: /* 1: Basic Introduction to using Condor: Karen Miller */

2012-05-08T14:50:15Z

1: Basic Introduction to using Condor: Karen Miller

← Older revision		Revision as of 14:50, 8 May 2012
Line 15:		Line 15:
	==Day 1: Tutorials ==		==Day 1: Tutorials ==

	===1: Basic Introduction to using Condor: Karen Miller===		=== Basic Introduction to using Condor: Karen Miller===
	Background HTC		Background HTC
	Definitions: Job, Class Ads, Match Making, Central Manager, Submit host, Execute Host		Definitions: Job, Class Ads, Match Making, Central Manager, Submit host, Execute Host
Line 54:		Line 54:

	use macro subs: %this gives the process id..		use macro subs: %this gives the process id..
	InitialDir=run_$(Process)		InitialDir=run_$(Process)




	==Day 2: Talks==		==Day 2: Talks==