Upgrade IntelliJ IDEA for Big Data Tool Plug-In & running spark!

2021.07.01 - [BigDATA/spark] - [Spark-Study] Day-3

회사에서 인텔리J를 사줘서 유용하게 사용하고 있다! 그래서 Ultimate version으로 셋팅! 
학생이라면 Community 버전으로! 그런데 역시나 돈을 안내면 제한은 있다..ㅜㅜ 

위에서 IntelliJ IDEA -> Applications로 드래그 해주면 인스톨 된다.
인스톨 이후 Plugins에서 Big Data Tools를 인스톨 해준다.
단, 커뮤니티 버전이면 big data tool 플러그인은 보이지 않는다..

그리고 spark-submit을 위해 local에 spark를 셋팅 해주어야하며
Mac 유저라면 brew를 통해 인스톨 해주자.

스터디에서 사용되는 예제는 spark 3.x 이상이어야 동작한다.
brew install spark를 했는데 버전이 낮다면 brew를 업데이트 먼저 해주자.
brew update

brew update
Error:
  homebrew-core is a shallow clone.
  homebrew-cask is a shallow clone.
To `brew update`, first run:
  git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
  git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask fetch --unshallow
These commands may take a few minutes to run due to the large size of the repositories.

위처럼 update시 오류가 난다면 가이드대로 git -C 명령어 2개를 실행한다.

 git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
remote: Enumerating objects: 637232, done.
remote: Counting objects: 100% (637188/637188), done.
remote: Compressing objects: 100% (223799/223799), done.
remote: Total 627500 (delta 410959), reused 617216 (delta 400839), pack-reused 0
Receiving objects: 100% (627500/627500), 249.55 MiB | 3.78 MiB/s, done.
Resolving deltas: 100% (410959/410959), completed with 7706 local objects.
From https://github.com/Homebrew/homebrew-core
   0de67c266d5..a0c3ee40ff1 master     -> origin/master
   
 git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask fetch --unshallow
remote: Enumerating objects: 32440, done.
remote: Counting objects: 100% (30856/30856), done.
remote: Compressing objects: 100% (11091/11091), done.
remote: Total 26958 (delta 19508), reused 23241 (delta 15851), pack-reused 0
Receiving objects: 100% (26958/26958), 5.51 MiB | 3.56 MiB/s, done.
Resolving deltas: 100% (19508/19508), completed with 2133 local objects.
From https://github.com/Homebrew/homebrew-cask
   9175dab7ad..b1e64d5200  master     -> origin/master

그리고나서 brew update를 해준다.

brew update

Updated 2 taps (homebrew/core and homebrew/cask).
==> New Formulae
.
.
.

You have 46 outdated formulae installed.
You can upgrade them with brew upgrade
or list them with brew outdated.

brew info spark 명령어로 버전을 확인한다. 1.0.1로 확인이 된다.

brew info spark
Warning: Treating spark as a formula. For the cask, use homebrew/cask/spark
spark: stable 1.0.1 (bottled)
Sparklines for the shell
https://zachholman.com/spark/
/usr/local/Cellar/spark/1.0.1 (6 files, 7.2KB) *
  Poured from bottle on 2021-07-29 at 11:09:47
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/spark.rb
License: MIT
==> Analytics
install: 455 (30 days), 1,324 (90 days), 5,745 (365 days)
install-on-request: 443 (30 days), 1,288 (90 days), 5,550 (365 days)
build-error: 0 (30 days)

brew에서 spark를 검색하고 인스톨 해보자.

brew search spark
==> Formulae
apache-spark ✔                 spark ✔                        sparkey                        spack                          sparse                         par
==> Casks
river-sparkle                        spark                                spark-ar-studio                      sparkle                              sparkleshare
 ⚙ terrypark  ~   master
 brew install apache-spark
apache-spark 3.1.1 is installed but outdated
==> Upgrading apache-spark
  3.1.1 -> 3.1.2

==> Downloading https://ghcr.io/v2/homebrew/core/apache-spark/manifests/3.1.2
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/apache-spark/blobs/sha256:5c1a0656373e8ece0a03ea78d4e90895b307ed48b7e8f4eeb2dca88d0bf6c32b
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:5c1a0656373e8ece0a03ea78d4e90895b307ed48b7e8f4eeb2dca88d0bf6c32b?se=2021-07-29T04%3A50%3A00Z&sig=eM6
######################################################################## 100.0%
==> Pouring apache-spark--3.1.2.all.bottle.tar.gz
🍺  /usr/local/Cellar/apache-spark/3.1.2: 1,360 files, 242.6MB
Removing: /usr/local/Cellar/apache-spark/3.1.1... (1,453 files, 246.7MB)
Removing: /Users/terrypark/Library/Caches/Homebrew/apache-spark--3.1.1.tgz... (218.1MB)

마지막으로 정리해보면

1. 인텔리J 업그레이드 및 플러그인 설치 완료
2. 스파크 설치 완료
3. 소스 개발 이후 아래와 같이 Run/Debug Configurations를 설정 한다.(Spark-Submit을 하기 위해!)

 

아래와 같이 Spark home / Main class / Run arguments 등을 설정하고 돌려보면 된다.

실행 후 아래와 같이 잘 동작함을 알수 있다.

 

동작 확인을 했으니 Debugging하는 것도 해보자~

끝~