pyspark udf in get_return_value py4j.protocol.Py4JJavaError:

在使用PySpark时,定义UDF(用户定义函数)必须明确指定输入数据类型,否则会导致Py4JJavaError。不进行类型转换,特别是在处理数值时,将引发错误。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

1、在用pyspark的时候,像这种udf函数一定要加上需要转换的数据类型,,不然会报错~~

def parse_label(val):
	return 1/(float(val)+0.0001)
TypeError: can only concatenate str (not "float") to str

2、pyspark 如图,如果不加float,会报下面这个错误
在这里插入图片描述

File "/mnt/yarn/usercache/hadoop/appcache/application_1614052931500_111643/container_1614052931500_111643_01_00
Traceback (most recent call last): File "/home/hadoop/PycharmProjects/SparkProject/src/test1.py", line 16, in <module> t_env.from_path("kafka_user_meal").select(lit(1).count).execute().print() File "/home/hadoop/桌面/pyflink/lib/python3.8/site-packages/pyflink/table/table_environment.py", line 615, in from_path return Table(get_method(self._j_tenv, "from")(path), self) File "/home/hadoop/桌面/pyflink/lib/python3.8/site-packages/py4j/java_gateway.py", line 1322, in __call__ return_value = get_return_value( File "/home/hadoop/桌面/pyflink/lib/python3.8/site-packages/pyflink/util/exceptions.py", line 146, in deco return f(*a, **kw) File "/home/hadoop/桌面/pyflink/lib/python3.8/site-packages/py4j/protocol.py", line 326, in get_return_value raise Py4JJavaError( py4j.protocol.Py4JJavaError: An error occurred while calling o18.from. : org.apache.flink.table.api.ValidationException: Table `kafka_user_meal` was not found. at org.apache.flink.table.api.internal.TableEnvironmentImpl.lambda$from$6(TableEnvironmentImpl.java:583) at java.util.Optional.orElseThrow(Optional.java:290) at org.apache.flink.table.api.internal.TableEnvironmentImpl.from(TableEnvironmentImpl.java:580) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282) at org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79) at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) hadoop@master:~$ hdfs dfs -ls /user/hive/warehouse/user_meal.db/hive_user_meal_cdc ls: `/user/hive/warehouse/user_meal.db/hive_user_meal_cdc': No such file or directory hadoop@master:~$ hadoop@master:~$
最新发布
06-19
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值