Spark 并行开发

最新推荐文章于 2022-04-02 17:10:49 发布

qq_40375298

最新推荐文章于 2022-04-02 17:10:49 发布

阅读量312

点赞数

CC 4.0 BY-SA版权

文章标签： spark

本文链接：https://blog.csdn.net/qq_40375298/article/details/112372900

开发前提

这里,需要对全杭州经纬度进行xgboost模型的训练预测,需要采用CountDownLatch并行开发.

CountDownLatch

CountDownLatch介绍

代码模板

package com.nokia.zjbigdata.open.spark.impora

import java.util.concurrent.{
   
   CountDownLatch, Executors}

import ml.dmlc.xgboost4j.scala.spark.{
   
   XGBoostRegressionModel, XGBoostRegressor}
import org.apache.spark.{
   
   SparkConf, SparkContext}
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.sql.{
   
   DataFrame, SparkSession}
import org.slf4j.LoggerFactory

/**
  *
  * program: 
  * description:上传训练模型
  * Author:
  * Date:2021/1/8 14:52
  *
  */
object demoXgBoost24 {
   
   
  def main(args: Array[String]): Unit = {
   
   
    if (args.length != 2) {
   
   
      println("Usage: SparkMLlibPipeline input_path native_model_path pipeline_model_path")
      //      sys.exit(1)
    }


    val LOG = LoggerFactory.getLogger("")
    val conf = new SparkConf()
    conf.set("spark.yarn.am.waitTime", "3600")
    conf.set("spark.sql.broadcastTimeout", "3600")
    conf.set("spark.kryo.registrationRequired", "false")

    val spark = SparkSession.builder().appName("XG