技术标签: linux运行pyspark
在pycharm上配置pyspark
在pycharm上配置pyspark
在windows上下面的错误,linux上应该正常
C:\ProgramData\Anaconda3\envs\tensorflow\python.exe E:/github/data-analysis/tf/SparkTest.py
2018-07-19 10:35:41 ERROR Shell:397 - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394)
at org.apache.hadoop.util.Shell.(Shell.java:387)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:80)
at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
2018-07-19 10:35:41 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Binarizer output with Threshold = 5.100000
[Stage 0:> (0 + 1) / 1]2018-07-19 10:35:51 ERROR Executor:91 - Exception in task 0.0 in stage 0.0 (TID 0)
java.io.IOException: Cannot run program "python": CreateProcess error=2, ϵͳ�Ҳ���ָ�����ļ���
at java.lang.ProcessBuilder.start(Unknown Source)
at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:133)
at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:76)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:86)
Caused by: java.io.IOException: CreateProcess error=2, ϵͳ�Ҳ���ָ�����ļ���
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 35 more
2018-07-19 10:35:51 WARN TaskSetManager:66 - Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.io.IOException: Cannot run program "python": CreateProcess error=2, ϵͳ�Ҳ���ָ�����ļ���
at java.lang.ProcessBuilder.start(Unknown Source)
at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:133)
at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:76)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:86)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:64)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
Caused by: java.io.IOException: CreateProcess error=2, ϵͳ�Ҳ���ָ�����ļ���
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 35 more
2018-07-19 10:35:51 ERROR TaskSetManager:70 - Task 0 in stage 0.0 failed 1 times; aborting job
Traceback (most recent call last):
File "E:/github/data-analysis/tf/SparkTest.py", line 21, in
binarizedDataFrame.show()
File "E:\hadoop-common\spark-2.3.1-bin-hadoop2.7\python\pyspark\sql\dataframe.py", line 350, in show
print(self._jdf.showString(n, 20, vertical))
File "E:\hadoop-common\spark-2.3.1-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\java_gateway.py", line 1257, in __call__
File "E:\hadoop-common\spark-2.3.1-bin-hadoop2.7\python\pyspark\sql\utils.py", line 63, in deco
return f(*a, **kw)
File "E:\hadoop-common\spark-2.3.1-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o49.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.io.IOException: Cannot run program "python": CreateProcess error=2, 系统找不到指定的文件。
at java.lang.ProcessBuilder.start(Unknown Source)
at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:133)
at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:76)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:86)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:64)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
Caused by: java.io.IOException: CreateProcess error=2, 系统找不到指定的文件。
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 35 more
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1602)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1590)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1589)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1589)
at
Caused by: java.io.IOException: Cannot run program "python": CreateProcess error=2, 系统找不到指定的文件。
at java.lang.ProcessBuilder.start(Unknown Source)
at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:133)
at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:76)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:86)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:64)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
Caused by: java.io.IOException: CreateProcess error=2, 系统找不到指定的文件。
Process finished with exit code
windows操作系统的原因
Anaconda上jupyter notebook使用pyspark
打开anaconda navigator,可以安装pyspark。可以运行,不用配置hadoop
image.png
另外参考
文章浏览阅读820次。跟请求对象/响应对象一样,也是在容器初始化时被创建好的对象;该对象可用于容器初始化时 servlet 辅助信息的设置;例如:编码格式的设置、servlet 在执行时需要的一些数据。在 web.xml 中 servlet 标签下可以进行 ServletConfig 辅助信息的设置格式:ServletConfig 对象和 Servlet 类是一对一的关系;只有 web.xml 配置文件中配置了。_servlet注解开发
文章浏览阅读661次,点赞24次,收藏13次。例如,我们定义一个学生类和一个老师类,我们会发现他们有大量的重复代码,而这些重复代码都是老师和学生作为人的公共属性和行为,所以在这种情况下,我们应该先定义人类,再通过继承,从人类派生出老师类和学生类,代码如下所示。子类继承父类的方法后,还可以对方法进行重写(重新实现该方法),不同的子类可以对父类的同一个方法给出不同的实现版本,这样的方法在程序运行时就会表现出多态行为(调用相同的方法,做了不同的事情)。需要提醒大家的是,对象的方法其实本质上也是对象的属性,如果给对象发送一个无法接收的消息,引发的异常仍然是。
文章浏览阅读658次。1.SequoiaDB 简介2.整体架构3.数据库存储引擎4.核心特性。_sequoiadb
文章浏览阅读1.7k次,点赞6次,收藏3次。作者 | 程序员客栈来源 | ID:proginnwx根据中国互联网络信息中心(CNNIC)近日发布第 44 次《中国互联网络发展状况统计报告》。截至 2019 年 06 月,中国网民规模为 8.54 亿,较 2018 年底增加 2598 万。网上外卖用户规模达 4.21 亿,较 2018 年底增长 1516 万;网络视频用户规模达 7.59 亿,较 2018 年底增长 3391 万;我..._2020年薪酬数据报告
文章浏览阅读1.3w次,点赞78次,收藏73次。轻舟低代码平台,NASL是其中的关键,也是最大的差异化。回头看Gartner的定义,也有LCAP(低代码平台)和CADP(无代码平台)之分,前者对开发完整性、应用独立性、逻辑完备性、可接入可集成等都有要求,所以说,NASL的实现使得轻舟低代码更符合这一理念。说到框架,大家首先想到肯定是是react,springboot这种被广泛使用的前后端框架,低代码框架本质上跟他们类似,差别在于,低代码框架通常支持从前后端到数据库的全栈模型实现,而且封装程度更高。_网易nasl语言
文章浏览阅读1.2k次。A2B系统的EEPROM自启动如何做,本文来告诉你_a2b技术学习
文章浏览阅读392次。对于二叉排序树,下面的说法 ___是正确的。(华南理工大学 2006年)设备驱动程序在读写磁盘数据时一般釆用下列哪种I/O方式?(中国科学院大学 2018)下列___交换技术是独占信道工作方式。(中南大学 2006年)命中率高且电路实现简单的Cache 与内存映射方式是___映射方式。(中国科学院大学 2015)_命中率高且电路实现简单的cache内存映射方式
文章浏览阅读3.2k次。可能是没有异常处理,只需要加上throws Exception就解决问题了。_unhandled exceptions: java.lang.instantiationexception, java.lang.illegalacc
文章浏览阅读1.5k次。webView.setWebViewClient(new WebViewClient(){ @Override public boolean shouldOverrideUrlLoading(WebView view, String url) { try{ if(url.startsWith("baidumap://")){ Intent _error_unsupported_scheme
文章浏览阅读66次。Embed (一)、基本语法: embed src=url 说明:embed可以用来插入各种多媒体,格式可以是 Midi、Wav、AIFF、AU、MP3等等, Netscape及新版的IE 都支持。url为音频或视频文件及其路径,可以是相对路径或绝对路径。 示例:<embed src="your.mid"> (二)、属性设置: 1、自动播放...
文章浏览阅读330次。PHP -- 文件包含、文件上传漏洞PHP -- 文件包含、文件上传漏洞文件包含文件引入漏洞,是由一个动态页面编译时引入另一个文件的操作。文件引入本身是没有问题,它是用于统一页面风格,减少代码冗余的一种技术。但是在特定的场景下就会形成漏洞jsp:include指令和include动作,其中include动作用于引入动态文件php:include(),include_once(),require()..._php文件上传4漏洞
文章浏览阅读406次。SSL 是需要申请证书的,key和PEM文件要放到服务器路径。然后NGINX下要进行443端口和80端口的绑定。server { listen 80; server_name ietaiji.com www.ietaiji.com; root "D:/aaa/WWW/ietaiji"; index index.html_nginx 和 http无法同时启动