눈가락★

https://leetcode.com/problems/palindrome-number/

주어진 숫자가 팰린드롬을 충족하는지 확인한다.

직관적으로, 두 개의 포인터를 생성하여 비교하는 방법을 이용할 수 있겠다.

첫번째 포인터는 왼쪽에서부터 오른쪽으로,

두번째 포인터는 오른쪽에서 왼쪽으로 순회한다.

그리고 그 포인터가 가리키는 값이 다르면 곧바로 False 를 출력한다.

음수는 minus 부호 때문에 팰린드롬을 충족하지 않는다.

따라서 곧바로 False 출력한다.

class Solution:
    def isPalindrome(self, x: int) -> bool:
        if x<0 : return False
        x = str(x)
        mid = int(len(x)/2)
        for i in range(0,mid):
            l = x[i]
            r = x[-i-1]
            if l!=r: return False
        return True

이 방법 외에, 숫자값을 뒤집은 새로운 숫자를 만들어서 비교하는 방법도 있겠다.

class Solution:
    def isPalindrome(self, x: int) -> bool:
        if x<0 : return False
        a = str(x)
        b = a[::-1]
        return a==b

< English (feedback by ChatGPT) >

Task is to check if the given number is satisfied Palindrome or not.

(The task is to check whether the given number is a palindrome.)

Intuitionally, we can use two pointers to compare.

(Intuitively, we can use two pointers to compare the digits.)

First pointer iterates from left to right

(The first pointer iterates from left to right,)

Second pointer iterates from right to left.

(and the second pointer iterates from right to left.)

And return False when the two pointer's values are not matched.

(If the values pointed to by the two pointers do not match, we immediately return False.)

Minus number has '-' mark so it can't be Palindrome.

(Negative numbers are not palindromes because of the minus sign (-),)

So return False.

(so we can return False right away.)

Another solution is to compare a new reverse number of the given number.

(Alternatively, we can reverse the number and compare it with the original.)

https://leetcode.com/problems/two-sum/

list와 숫자 하나(target)가 주어진다.

list 내에는 숫자 두 개를 더해서 target 이 만들어지는 쌍이 무조건 존재한다.

이 쌍의 index 를 찾아내어 출력하면 된다.

간단하게 생각하면, for문을 두 번 돌아 쉽게 해결할 수 있다.

class Solution:
    def twoSum(self, nums: List[int], target: int) -> List[int]:
        for l in range(0,len(nums)-1):
            for r in range(l+1, len(nums)):
                if(nums[l]+nums[r]==target):
                    return [l,r]

이렇게 제출 할 경우, time complexity (time big O) 는 O(n2)

이보다 더 효율적인 방법으로 문제를 해결할 수 있다.

list 는 정렬된 상태로 주어진 게 아니기 때문에,

list 를 순회하는 것은 피할 수 없다.

list 를 순회할 때 나머지 쌍을 이루는 숫자는 정해져있다.

바로, target - list[i]

즉, list 에 list[i] 와 target-list[i] 가 둘 다 존재하는지 확인하고, 존재하면 그 둘의 index 를 출력하면 된다.

(이 때 둘의 index 는 동일하면 안 된다)

list 에서 특정 값을 검색하는 작업도 O(n) 시간이 들기 때문에,

list 값과 index 를 미리 hashing 한 dictionary 를 사용하여 O(1) 으로 빠르게 검색한다.

class Solution:
    def twoSum(self, nums: List[int], target: int) -> List[int]:
        d = {}
        for i in range(0, len(nums)):
            d[nums[i]] = i
        for i in range(0, len(nums)):
            if( (target-nums[i]) in d and d[target-nums[i]]!=i ):
                return [i, d[target-nums[i]]]

< English (feedback by ChatGPT) >

One list and one integer(target) are given.

(A list and a single integer (target) are given.)

There must be a pair of integers in the list which sum to 'target'.

(There is guaranteed to be a pair of numbers in the list whose sum equals the target.)

We need to find the two indexes of the pair.

(Our task is to find and return the indexes of the two numbers.)

Simply we can use double for loops to solve this problem.

(We can simply solve this problem by using two nested for-loops.)

But in this case, the time complexity is O(n^2) and we can do better than this.

(However, in that case, the time complexity would be O(n²), and we can find a more efficient solution.)

List is given not being sorted. So it's inevitable to iterate through the list.

(The list is not sorted when given, so iterating through it is unavoidable.)

During the iteration, a rest integer of the pair is already set, and it is 'target - list[i]'

(During iteration, the other number needed to complete the pair is determined — it is target - list[i].)

It means, we can check if 'list[i]' and 'target-list[i]' exist in the list and return the pair's indexes.

(In other words, we need to check whether both list[i] and target - list[i] exist in the list, and if they do, return their indexes.)

shouln't be correspond with each indexes

(The two indexes must not be the same.)

It takes O(n) time to search an element in list.

(Since searching for an element in a list takes O(n) time,)

Instead, we can use dictionary with list values and it's index to search fast by hashing with O(1) time.

(we can instead use a dictionary that maps each list value to its index, allowing for O(1) time lookup via hashing.)

1.

2.

3.

4.

6.

셔플 조인은 가장 느린 조인

맵(Map) 단계에서 각 테이블을 읽고, 조인 컬럼을 키로 설정하여 셔플

리듀서로 데이터가 이동되고 리듀서에서 테이블을 조인

버켓팅 되어있으면 Bucket Map Join 이 빨라짐.

join 의 기준이 되는 key 들이 모두 버케팅 되어있는 상황에서 Join 을 진행하면,

작은 테이블의 버킷들(Table a, Table c)이 큰 테이블의 버킷(Table b)의 메모리에 모두 복사됨

이렇게되면 join 에 필요한 모든 key 가 하나의 Mapper 에서 접근 가능하기 때문에 join 속도 향상

작은 테이블 크기가 메모리에 올라갈 정도로 작아야 함

브로드캐스트 조인임.

Sort Merge Join 은 조인 테이블이 버켓팅 되어 있을 때 사용 가능

버켓팅된 키의 정보를 이용하여 빠르게 조인

다음 절차로 join 이 진행됨

- Table a 와 Table b 에서 join 에 필요한 데이터를 읽음

- 별도의 공간에서 읽은 데이터를 정렬sort 함

- 정렬된 데이터를 기준으로 join 함

참고 : https://coding-factory.tistory.com/757

hive 명령어와 beeline 명령어 차이?

- hive 명령어는 하이브 옵션 설정, 쿼리 실행 등이 가능

- beeline 은 단지 hive 에 thrift 로 접근하는 인터페이스

thrift : 다양한 언어로 개발된 소프트웨어들을 쉽게 결합(통신)하는 프레임워크. 내부에서 RPC 사용

JDBC :

- Java Database Connectivity

- JAVA 언어로 DB 에 접근해 DML 쿼리 하기 위한 인터페이스

- Java와 연동되는 DBMS에 따라 그에 맞는 JDBC를 설치할 필요가 있음

ODBC

- Open Database Connectivity

- 응용프로그램에서 다양한 DB 에 접근해 DML 쿼리하기 위한 인터페이스

- 접속처의 데이터베이스가 어떠한 DBMS에 의해 관리되고 있는지 의식할 필요가 없음

hive 에서 orc 를 사용하는 이유

- 높은 압축률

- 컬럼 기반 포맷이라 처리 속도가 빠름

- 스키마를 가지고 있음

- orc 는 hive 에 최적화되어있고, parquet 은 spark 에 최적화 되어있음

ORDER BY vs DISTRIBUTE BY vs SORT BY vs CLUSTER BY

ORDER BY

- 매퍼에서 나오는 모든 데이터를 하나의 리듀서로 몰아서 정렬 수행

- 리듀서가 하나 뿐이라, 저장되는 파일도 하나

- limit 을 추가하여 리듀서에 부하를 줄이는 게 좋음

- order by COLUMN

SORT BY

- ORDER BY 와 다르게, 리듀서가 하나 이상, 각 리듀서에서 정렬 수행

- 각 리듀서별로 정렬하기 때문에, 모든 리듀서 결과는 전체 정렬 되어있지 않음

- 리듀서 개수를 1로 설정하면 ORDER BY 와 같은 결과

- sort by COLUMN

DISTRIBUTE BY

- distributed by 의 대상이 되는 컬럼의 값 기준으로 group 지어 하나의 리듀서에 들어감

- 정렬 수행하지 않음

- 예)

  정렬 대상 : a a b c c c d

  리듀서 1) a d a

  리듀서 2) c b c c

  (리듀서 개수와 상관 없이) 같은 값은 모두 하나의 리듀서에 몰려 있음. 

CLUSTER BY : 

- distributed by 와 sort by 를 같이 사용

- 즉, distributed by 실행하며 정렬까지 진행

- 예)

  정렬 대상 : a a b c c c d

  리듀서 1) a a d

  리듀서 2) b c c c

참고) https://saurzcode.in/2015/01/hive-sort-order-distribute-cluster/

Hive 정적 파티션 vs 동적 파티션

정적 파티션

- 데이터의 테이블에 파티션 값을 직접 입력

  예)

  INSERT INTO TABLE tbl(yymmdd='20220625')
  SELECT name FROM temp;

  > hdfs://hive/tables/yymmdd=20220625/

동적 파티션

- 데이터의 컬럼값을 기준으로 파티션이 생성됨

- 쿼리 시점에 어떤 데이터가 어떤 파티션에 가는지 모름

- 동적 파티션은 느림

  예)

  INSERT INTO TABLE tbl(yymmdd)
  SELECT name, yymmdd FROM temp;

  > hdfs://hive/tables/yymmdd=20220625/

  > hdfs://hive/tables/yymmdd=20220626/

  > hdfs://hive/tables/yymmdd=__HIVE_DEFAULT_PARTITION__/

HDFS 에서 작은 파일들 합치기

https://gyuhoonk.github.io/hive-merge-query

결론

작은 파티션들이 많으면, HDFS IO 도 많아지고 NN memory 에 부담도 커짐

Hive

- insert into 등의 쿼리는 매퍼만 동작하는데,

  매퍼에서 읽은 데이터를 그대로 HDFS 블럭으로 저장하기 때문에 블럭 개수가 늘어남

  이를 합쳐주기 위해 (sort by 등을 추가하여) 리듀서를 추가

Spark

- 셔플 파티션은 기본값이 200 이라서, 셔플 후에는 200개의 작은 파티션들이 생성되어 HDFS 에 저장됨

  이를 합쳐주기 위해 셔플 파티션 값을 조정하거나, repartition 혹은 coalesce 를 사용하여 파티션 개수 줄임

< Presto >

< Iceberg >

https://spark.apache.org/docs/latest/index.html

-  pandas API on Spark for pandas workloads

- Downloads are pre-packaged for a handful of popular Hadoop versions

- Spark runs on both Windows and UNIX-like systems, and it should run on any platform that runs a supported version of Java

- it is necessary for applications to use the same version of Scala that Spark was compiled for

For example, when using Scala 2.13, use Spark compiled for 2.13

- use this class in the top-level Spark directory.

- with this approach, each appliction is given a maximum amount of resources it can use

and holds onto them for its whole duration.

- Resource allocation can be configured as follows, based on the cluster type.

- At a high level, Spark should relinquish executors when they are no longer used and acquire when  they are needed.

- We need a set of heuristics to determine when to remove and request executors.

- By default, Spark's scheduler runs jobs in FIFO fashion.

- If the jobs at the head of the queue don't need to use the whole cluster, 

later jobs can start torun right away, but if the jobs at the head of the queue are large,

then ddlater jobs may be delayed significantly.

- Under fair sharing, Spark assigns tasks between jobs in a "round robin" fashion,

so that all jobs get a roughly equal share of cluster resources.

- This feature is disabled by default and available on all coarse-grained cluster managers.

- Without any intervention, newly submitted jobs go into a default pool

- This is done as follows

- This setting is per-thread to make it easy to have a  thread run multiple jobs on behalf of the same user.

- If you would like to clear the pool that a thread is associated with, simply call this.

- jobs run in FIFO order.

- each user's queries will run in order instead of later queries taking resources from that user's earlier ones.

- At a high level, every Spark application consists of a driver program that runs the user's main function and executes various parallel opperations on a cluster.

- ...the cluster that can be operated on in parallel.

- This guide shows each of these features in each of Spark's supported languages.

- it's easiest to follow along with if you launch Spark's interactive shell.

- 

- It is not only Value but also Pointer, both of these together make up the node.

- We do it by just having the next value of A node be the B node.

- the same is true of the C node.

- if you look at how we're going to have to traverse this, we are going to have to start at head.

- that's what we are going to do down here with this print statement.

- the syntac is a little bit different than if you are going to use dictionaries.

- 

- Our managers deal with all kinds of clients every day. So I can say that we maintain the highest level of service.

- In my work I follow the best practices to maintain clean and easy to understand python codes.

- In my previous project I carried out the responsibilities of both the Project Manager and the Team Leader.

carry out : to do something, to perform

both A and B

- As a QA specialist I worked with a test environment where I tested many aspects of the platform

  to ensure that it works as desired.

- A cloud architect oversees application architecture

  and deploys it in cloud environments like public cloud, private cloud and hybrid cloud.

to oversee : to watch over and control something to make sure that the work is good or satisfactory, to supervise

- I took a course where I learned how to design and write programs that are easy to maintain.

to design : to create, draw, or construct something

- I will set up all the necessary equipment in my home office to work remotely on this project

- I'm an IT Technician, so I install and configure different software on all computers in the office.

to install : to put a new program or piece of software into a computer

to configure : to chagne setting of software on a computer

- As a Jr Software Engineer, I assist and participate in the research, design, development and testing software and tools.

to assist : to help someone or something

- I am a web designer, so I know hot to provide the best UX for your website visitors.

- Project managers usually estimate new projects by analogy, using previous projecs and past experience.

to estimate : to give a general idea of the cost of work or the time you need to do the work

analogy : a comparison of two things based on their being alike in some way

- Sometimes I need to google my questions, for ex "how to execute the code inside of function in JS"

- Working on my project I improved my time management and organizational skills.

- press the F2 key on your keyboard

- The screen resolution is 1366x768

- I prefer work with desktop

- Workstation PCs have multiple processor cores.

- Some tablets have a long battery life

- The volume on my speakers won't ture up.

- My printer broke down, so I printed out these documentations at work.

- 'ram' is not countable, so only possible to say 'ram is' or 'ram was', not 'rams' or 'rams are'

- with a cable : wired mouse, wired connection(Ethernet)

- without a cable : wireless mouse, wireless connection(WiFi)

- ISP stands for Internet Service Provider.

- so many folers on my desktop

- start or shut down a computer.

- turn on or turn off a computer

- to crash / to freeze up : when a computer suddenly stops working

- to look up a word or address : to find something.

  we can use 'nslookup' command in terminal to query to DNS server.

- It will take about two hours to key in all this data.

to key in : to enter info into computer

- a shortcut key : 단축키.

Use Ctrl + L shortcut to see the last saved version.

- 'perform' is used a lot more than my thought. for ex, the server performs instructions written in code.

- use only the numbers in given array.

- I assigned the number 33 to age variable.

- fraction : 분수

- numerator : 분자

- denominator : 분모

  ex, 1/3 : one thrid, 2/3 : two thirds, 1/2 : a half(second), 1/4 : a quarter, 3/4 : three quarters

- decimal : 소수

- decimal point : 소수점

- floating point : 부동소수점

  ex, 1.23 : one point two three, 15.1 : fifteen point one

'double' also has a point but 'float' and 'double' are different each.

let's look up how they consist differently.

- I created an array of strings.

- To debug is to investigate the program and fix bugs.

- Comment is a text written around code that is ignored by the computer.

It is used for writing extra info about your code to help you undertand it later.

so we can say, 'leave comments in your code.'

- 'Comment out' is to turn a piece of code into a comment with the help of special characters.

like, //, #, -- ... etc

you can comment out some lines to see how it works without them.

- Constant is a variable that never changes its value.

for ex, val a = 1, final int a= 1

we can say "In Java, a constant is assigned using the final keyword"

"the PI constant has the value of 3.14"

- If you try to divide a number by zero, your program will crash.

A program crashes when it stops running because of an error.

- An 'executable' is a program which is ready to be run. 

Short for executable file, executable program

A common filename extension .exe means that it is an executable file.

it sounds '엨즤큐터블'

- To declare(선언) in programming means to say that something exists

usually a variable, a function, or a class.

I've only declared a function, but I haven't written it yet.

- To implement(구현) means to write and complete something in code

for example, to implement a function or a class

I declared a function and implemented it. It works well! 

- To instantiate(인스턴스화) means to create an object from a class.

I instantiated another object of the Student class.

it sounds '인스탠시에이트'

- A loop is a piece of code that runs itself many times.

It can also be used as a verb - to loop or to iterate

I used a "for" loop to run this code for every value in the array.

I iterate throught every element in the list.

- He read some data values from another source over the internet.

- Syntax is the grammatical rules of a programming language.

Syntax determines if code is written correctly or not.

- find any typing mistakes if you got a syntax error.