Introduction to LAVA Workload Scheduler

1. Introduction to LAVA workload scheduler High Performance Computing and Networking Center (HPCNC) Kasetsart University In collaboration with Innovative Extremist (INOX) Co.,Ltd. and Platform Computing Inc.

2. Outline Introduction to HPC, cluster and workload scheduler

3. LAVA workload scheduler

4. Installing and configuring LAVA Cluster

5. Workshop : Using LAVA

6. Introduction to HPC, Cluster and Workload Scheduler

7. Cluster Computing Cluster computing is a technology related to the building of high performance scalable computing system from a collection of small computing system and high speed interconnection network

8. Why now? Maturity of many enabling technologies Low to medium cost high speed network Gigabit Ethernet, Myrinet, InfiniBand Powerful operating systems such as Windows, Linux, UNIX

9. Parallel Programming Systems which is portable and efficient

10. MPI (LAM, MPICH)

11. Software library that ease the application development e.g. Scalapack, Plapack, PetSc

12. PC also rule the world Impact of PC technology Intel Pentium can deliver supercomputing performance at low cost

13. PC mass market nature drive the price down while performance increase rapidly

14. Cluster nature make it easy to capitalize on PC new technology right away

15. Why? Price Performance!

16. Goal of Clustering High-performance clustering Link many computers together to team up and finish problem fasters by having multiple computer working on the same problem independently

17. Goal of Clustering High-availability clustering make more reliable computer system by having many computers working together and takeover when any of them fail

18. Applications Scientific computing CAD/CAM

19. Bioinformatics

20. Large scale financial analysis

21. Simulation

22. Drug Design

23. Automobile Design ( Crash Simulation) IT infrastructure Scalable web server, Search engine

24. (Google use more than 10000 node servers) Entertainment Rendering - On-line Gaming

25. Molecular Dynamic Simulation Drug Discovery using molecular docking Avian Flu

26. HIV Analyzing property of Chemical compound

27. Graphics Rendering and Special Effect Rendering Generating 3D image from model Problem Rendering is a time consuming process especially for complex and realistic scene

28. Massive number of rendering job needed to be done to create a movie

29. Cluster Software Architecture HTC HPC HPTC

30. High Throughput Computing High throughout, not high performance Complete most number of jobs in shortest amount of time Serial, parametric (usually), non-parallelized code Solve them on multiple processors at the same time, varying input parameters Example BLAST, Monte Carlo simulation Use of Load Schedulers Condor, Codine, LSF, Sun Grid Engine, SQMS

31. High Throughput Computing (con) Pros and Cons Easy to get started. Use the sequential code in C or Fortran.

32. Excellence for many type of applications such as Parametric computing: Running the same computation with multiple data set

33. Distributed application such as massive rendering in animation industry Excellence when model can fit well in memory of a single computer No communication at all

34. High Performance Computing Maximum performance, not maximum throughput

35. Use of specialised codes, libraries MPI (Message Passing Interface)

36. Parallel Maths Libraries (ScaLapack) Solve large problem by breaking it in to a number of small problems (data or task partitioning), then solve them on distributed, multiple processors at the same time.

37. Pros and Cons Difficult since a parallel program must be developed

38. Good when Problem is larger than memory size of a single machines

39. Speedup for a single instance of problem is needed

40. Advantages and Challenges Advantages Highly scalable, light weight, easy setup

41. Plenty of free software Challenges Require a very highly trained, skill personal to maintain the system

42. No powerful software development environment

43. Low compatibility with many enterprise computing environment

44. Parallel Application Development Shared memory – data is exchanged using memory reference

45. Message passing – data is exchanged by sending/receiving messages between processors

46. Workload Scheduler Or “Job scheduler” or “Load scheduler”

47. Main role of distributed computing

48. Allow users to share computing resources and time sharing Unify resources in the cluster in to a shared resource pool

49. Control shared resource usage for multiple users Job queue

50. Scheduling Policies Utilize resources efficiently

51. Hide the complexity of using cluster's computing resources by submitting job to the scheduler

52. Key Features Resources Control Where are the resources?

53. How many we can use?

54. By whom? Job queue Classify users sharing, waiting queue.

55. Apply scheduling policy / resource pool User Interface Job control Submit / Suspend / Resume / Delete Monitoring job status

56. Implementations SGE / N1 Grid Engine

57. Platform LSF / Platform LAVA

58. PBS / PBS Pro

59. Torque

60. Maui

61. MS Job scheduler

62. LAVA Workload Scheduler

63. LAVA Workload Scheduler An open source entry-level workload scheduler

64. Designed to meet a wide range of workload scheduling needs for clusters up to 512-nodes

65. Features Scalability

66. Reliability

67. Parallel Job Scheduling

68. Complete Job History

69. Interactive Jobs

70. Job Arrays

71. Job Dependency

72. Job Migration

73. Components (mbatchd) (sbatchd) (sbatchd) (sbatchd) LAVA Base LIM (Load Information Manager) RES (Remote Execution Service) Computer LSBATCH

74. Installing and Configuring

75. Installing LAVA First, We need a cluster (or servers)

76. Installing LAVA Manual installation Setup a cluster

77. Install LAVA from source Download source code from HPCCommunity Website

78. Extract, compile and install tar zxvf lava-tarball.tar.gz

79. ./configure

80. make && make install

81. Installing LAVA Using cluster distribution LAVA Kit for KUSU

82. LAVA Roll for ROCKS Cluster Advantages Auto configuration tools

83. Can scale nodes without editing config files

84. Configuring LAVA Environments

85. LAVA Base (lsf)

86. Batch scheduler (lsbatch)

87. Configuration Environments

88. Setting in /etc/profiles.d/lava.sh LSF_VERSION=1.0

89. LSF_TOP=/usr

90. LSF_BINDIR=/usr/bin

91. LSF_SERVERDIR=/usr/sbin

92. LSF_LIBDIR=/usr/lib

93. LSF_ENVDIR=/etc/lava/conf

94. Configuration LAVA Base (LSF) Local information and execution process

95. Config file usually located at $LSF_ENVDIR = /etc/lava/conf/

96. Files lsf.conf * (Installation and operation of Lava)

97. lsf.cluster.lava * (general configuration, nodes, parameters)

98. lsf.task (type of tasks)

99. lsf.shared (default parameters)

100. hosts * (list of known hosts and IPs) Mark (*) is configuration needed on master and slave nodes

101. Configuration files Batch scheduler (LSBATCH) Central batch scheduler of the cluster

102. Config file usually located at $LSF_ENVDIR/lsbatch/lava/configdir/

103. Files lsb.hosts (list of nodes and parameters)

104. lsb.modules (list of plugin modules)

105. lsb.params (batch scheduler parameters)

106. lsb.queues (job queue name & properties)

107. lsb.users (list of allowed users)

108. Start the LAVA Start/Stop the LAVA service on every nodes /etc/init.d/lava start

109. /etc/init.d/lava stop For individual service

110. Controlling Queues Adding Job Queue Edit lsb.queues to add the new queue definition.

111. Copy another queue definition from this file as a starting point and change the QUEUE_NAME of the copied queue.

112. Save the changes to lsb.queues.

113. Run badmin reconfig to reconfigure mbatchd.

114. Adding a queue does not affect pending or running jobs.

115. Controlling Queues Removing Queue Close the queue to prevent any new jobs from being submitted using command badmin qclose QUEUE_NAME

116. Move all pending and running jobs into another queue using command bswitch -q Q_FROM Q_TO 0

117. Edit lsb.queues and remove or comment out the definition for the queue you want to remove.

118. Save the changes to lsb.queues.

119. Run badmin reconfig to reconfigure mbatchd.

120. Using LAVA

121. LAVA Commands LSBATCH (starts with 'b') badmin*

122. bbot

123. bchkpnt

124. bhist

125. bhosts

126. bjobs

127. bkill

128. bmgroup

129. bmig

130. bmod

131. bparams LAVA base (starts with 'ls') lsadmin*

132. lsacct

133. lseligible

134. lshosts

135. lsid

136. lsinfo

137. lsload

138. lsloadadj

139. lsmon

140. lsplace

141. lsrcp bpeek

142. bqueues

143. brequeue

144. brestart

145. bresume

146. brun

147. bstop

148. bsub

149. bswitch

150. btop

151. bugroup

152. busers

153. Administrative Commands lsadmin

154. badmin

155. System information bhosts / bqueues -l

156. bqueues / bqueues -l

157. bparams / bparams -l

158. bmgroup

159. bugroup

160. busers [all] lsid

161. lsinfo

162. lshosts

163. lsload

164. lsmon

165. lsacct

166. Monitoring Jobs bjobs

167. bjobs <job id>

168. bjobs -a show all state (include EXIT, DONE) bjobs -r / -p / -s show only running / pending / suspended jobs bjobs -u user1 / bjobs -u all show only user1 / show all users bjobs -l show more detail

169. View Job History bhist

170. bhist -l long detail

171. Submitting a Job Use command bsub [-option]/path/to/command args If you do not specify any options, the job is submitted to the default queue configured by the Lava administrator (usually thenormal queue)

172. Example $ bsub my_job

173. Job <1234> is submitted to default queue <normal> In the above example, 1234 is the job ID assigned to this job, and normal is the nameof the default job queue.

174. Submitting a Script Any command or script you can execute from a shell prompt can be submitted to Lava for batch execution.

175. Create file myscript

176. chmod u+x myscript

177. bsub < myscript #!bin/sh #BSUB -q test #BSUB -o outfile -R "mem>10" myjob arg1 arg2 #BSUB -J myjob

178. Submitting a job to specific hosts To indicate that a job must run on one of the specified hosts, use the to a single host bsub -m "hostA hostB ..." option.

179. By specifying a single host, your job will wait until that host is available and then run on that host. $ bsub -q idle -m "hostA hostD hostB" myjob Or select by specific resources $ bsub -R "hname!=hostb && type==LINUX86" myjob

180. Running Parallel Jobs To submit a parallel job, use bsub -n and specify multiple processors. $ bsub -n 4 myjob This command submits myjob as a parallel job. The job is started when 4 job slots are available.

181. Job slot limits for parallel jobs A job slot is the basic unit of processor allocation in Lava. A sequential job uses one job slot. A parallel job that has N components (tasks) uses N job slots, which can span multiple hosts.

182. Modify Jobs Use bmod command to modify job submission parameters on pending job bmod -b 2:00 101

183. change the start time of job 101 to 2:00 a.m To reset an option to its default submitted value (undo a bmod), append the n character to the option name bmod -bn 101

184. Killing a job Killing a job bkill command cancels pending batch jobs and sends signals to running jobs.

185. By default, bkill sends SIGINT, SIGTERM and then SIGKILL signal to running jobs

186. Example bkill 3421

187. Job <3421> is being terminated bkill -r command removes a job from the system without waiting for the job to terminate in the operating system and mark as EXIT status

188. Suspend and Resume a job Suspend Using command bstop job_ID

189. Your job goes into USUSP state if the job is already started, or into PSUSP state if it is pending. bstop 3421

190. Job <3421> is being stopped Resume Run bresume job_ID bresume 3421

191. Job <3421> is being resumed

192. Requeuing and Rerunning Jobs You can kill and requeue a job while it is running or when it is suspended. Use the brequeue command to requeue the job brequeue -u user5 45 67 90 To enable automatic job rerun, submit the job with the re-runnable option bsub -r. If the execution host fails, Lava wil dispatch the job to another host. Youlll receive an email informing you of the host failure and the requeuing of the job. bsub -r my_job

193. Moving Jobs Moving a job to the bottom of a queue bbot job_id Moving a job to the top of a queue btop job_id

194. Switching jobs from one queue to another Switch a single job Use bswitch to move pending and running jobs from queue to queue. bswitch priority 5309

195. Job <5309> is switched to queue <priority> Switch all jobs Use bswitch -q from_queue to_queue 0

196. The job ID number 0 specifies that all jobs will be moved bswitch -q night idle 0

197. END OF SESSION Thank you

198. Reference Lava User Guide http://www.hpccommunity.org/exdata/lava/docs/lava_using.pdf

Introduction to LAVA Workload Scheduler

More Related Content

What's hot (20)

Similar to Introduction to LAVA Workload Scheduler (20)

Recently uploaded (20)

Introduction to LAVA Workload Scheduler