개발 영어 공부

 

https://spark.apache.org/docs/latest/index.html

 

-  pandas API on Spark for pandas workloads

 

- Downloads are pre-packaged for a handful of popular Hadoop versions

 

- Spark runs on both Windows and UNIX-like systems, and it should run on any platform that runs a supported version of Java

 

- it is necessary for applications to use the same version of Scala that Spark was compiled for

For example, when using Scala 2.13, use Spark compiled for 2.13

 

- use this class in the top-level Spark directory.

 

- with this approach, each appliction is given a maximum amount of resources it can use

and holds onto them for its whole duration.

 

- Resource allocation can be configured as follows, based on the cluster type.

 

- At a high level, Spark should relinquish executors when they are no longer used and acquire when  they are needed.

 

- We need a set of heuristics to determine when to remove and request executors.

 

- By default, Spark's scheduler runs jobs in FIFO fashion.

 

- If the jobs at the head of the queue don't need to use the whole cluster, 

later jobs can start torun right away, but if the jobs at the head of the queue are large,

then ddlater jobs may be delayed significantly.

 

- Under fair sharing, Spark assigns tasks between jobs in a "round robin" fashion,

so that all jobs get a roughly equal share of cluster resources.

 

- This feature is disabled by default and available on all coarse-grained cluster managers.

 

- Without any intervention, newly submitted jobs go into a default pool

 

- This is done as follows

 

- This setting is per-thread to make it easy to have a  thread run multiple jobs on behalf of the same user.

 

- If you would like to clear the pool that a thread is associated with, simply call this.

 

- jobs run in FIFO order.

 

- each user's queries will run in order instead of later queries taking resources from that user's earlier ones.

 

- At a high level, every Spark application consists of a driver program that runs the user's main function and executes various parallel opperations on a cluster.

 

- ...the cluster that can be operated on in parallel.

 

- This guide shows each of these features in each of Spark's supported languages.

 

- it's easiest to follow along with if you launch Spark's interactive shell.

 

- 

 

 

 

 

 

 

 

 

 

 

- It is not only Value but also Pointer, both of these together make up the node.

 

- We do it by just having the next value of A node be the B node.

 

- the same is true of the C node.

 

- if you look at how we're going to have to traverse this, we are going to have to start at head.

 

- that's what we are going to do down here with this print statement.

 

- the syntac is a little bit different than if you are going to use dictionaries.

 

-