run-class.sh concurrency issues when on samza-yarn

Several possible issues were identified in run-class.sh, including:

Race condition in pathing jar manifest creation

A race condition exists when setting up the classpath during container launch.

During container launch using samza-yarn, run-class.sh creates a pathing jar file (which holds the classpath for the container launch). However, during the creation of this pathing jar, temporary files, as well as the pathing jar itself is not placed in a location unique to the container. This results in multiple containers writing to the same pathing jar location and temporary file location, which results in a race condition.

This race condition may show up in several ways, such as when Yarn removes jars from a finished container (other containers will point to a classpath which no longer exists) or when multiple run-class.sh scripts attempt to write the manifest.txt or pathing jar at the same time.

Note that host affinity being enabled will make this problem worse. The pathing.jar is written to the usercache, so when the container which created the pathing.jar is finished and removed, any new container which launches on that host will point to jar files which do not exist anymore. When host affinity is enabled, it will not move to a new host and just keep failing.

Typical errors for this include sporadic exceptions (typically it works on retry) like the following:

java.io.IOException: line too long (line 511)
	at java.base/java.util.jar.Attributes.read(Attributes.java:380)
	at java.base/java.util.jar.Manifest.read(Manifest.java:290)
	at java.base/java.util.jar.Manifest.<init>(Manifest.java:100)
	at java.base/java.util.jar.Manifest.<init>(Manifest.java:76)
	at jdk.jartool/sun.tools.jar.Main.run(Main.java:277)
	at jdk.jartool/sun.tools.jar.Main.main(Main.java:1683)

and

Error: Could not find or load main class <snip job task process class name>
Caused by: java.lang.ClassNotFoundException: <snip job task process class name>

Container logging directory fallback is not unique for each container

The fallback log directory is the same among all containers running on the same host. It should be unique per-container.

Container tmp dir is not unique per-container

The JAVA_TMP_DIR directory is the same for all containers. We should make sure that it's safe to use the same directory for all containers.