Skip to main content

Hadoop and Spark Setup

#get centos container
git pull centos

#start centos with port mapping
docker run -d -p 8088:8088 -p 9870:9870 --name "centos" -i centos

#access centos
docker exec -it centos /bin/bash

#download the instructions
wget https://www.clustermonkey.net/download/LiveLessons/Hadoop_Fundamentals/Hadoop_Spark_Fundamentals_Code_Notes-V3.0.tgz
cd /tmp
tar xzvf Hadoop_Spark_Fundamentals_Code_Notes-V3.0.tgz

#follow the instructions in /tmp/Hadoop_Spark_Fundamentals_Code_Notes-V3.0/Lesson-2/Lesson-2.2/NOTES.txt to setup hadoop, pig, hive

a few catches
* adjust the JAVA_HOME and HADOOP_HOME accordingly based on your installation
* modify HADOOP_PATH in the script hadoop-setup-script.sh under /tmp/Hadoop_Spark_Fundamentals_Code_Notes-V3.0/Lesson-2/Lesson-2.2/scripts
* check if logs under /opt/hadoop-3.2.0/ is created
* after hadoop 3.0.0, the namenode port is 9870 instead of 50070
* before running yarn, set the YARN_CONF_LIB
export YARN_CONF_LIB=/opt/hadoop-3.2.0/etc/hadoop


##follow the instructions in /tmp/Hadoop_Spark_Fundamentals_Code_Notes-V3.0/Lesson-2/Lesson-2.3/NOTES.txt to setup spark


Popular posts from this blog

spring 2.0 bean scope

singleton Scopes the bean definition to a single instance per Spring container (default). prototype Allows a bean to be instantiated any number of times (once per use). request Scopes a bean definition to an HTTP request. Only valid when used with a web capable Spring context (such as with Spring MVC). session Scopes a bean definition to an HTTP session. Only valid when used with a webcapableSpring context (such as with Spring MVC). global-session Scopes a bean definition to a global HTTP session. Only valid when used in a portlet context.
Post a Comment
Read more

Crocs sandals

Suddenly one special looking sandals get popular. The brand is Crocs. It even opens a brand store at Marina Square. The design idea is from Dutch wooden shoes, I guess. A pair of Crocs sandals is sold at around SGD 50. The price is justified for what it is made of - Croslite. Based on Crocs website, "Croslite™, a proprietary Closed Cell Resin (PCCR) which is NOT plastic NOR rubber. Croslite™ is closed-cell in nature and anti-microbial, which virtually eliminates odor. it is an extraordinary impact absorbing resin material developed for maximum cushioning. its closed cell properties resist odor, inhibits bacterial and fungal growth and are non toxic. this versatile material can be worn next to skin and be cleaned with just soap and water." However, it really looks like made of plastic or rubber, and the design is unique. Replica comes. they are sold at SGD 20, SGD 10, SGD 5 depending on quality.
Post a Comment
Read more

Singapore Girl Sex Clip Posted Online

A couple days ago, a sex video clip about a female Chinese graduate student Wang Ting Ting (王婷婷) was posted on the internet. Just as it is about to cool down, another sex video clip pops. It is a sex clip of a Singapore Nanyang Polytechnic student . The video clip was stored on her cell phone. Someone stole it and posted the video clip on the Internet. This is a breaking news. It is even reported on major Singapore and Malaysia newspapers. Now we have so many cool gadgets. It is a breeze to shoot photos/videos, and share them on the internet. Everyone, even dog, is on the internet waiting for breaking news. Be careful when you do something secret or stupid. It may appear all over the world, live!
Post a Comment
Read more

No smoking sign

Watch out this sign before you light the cigarette up. SGD 1000 fine! However, I wonder if someone had really paid so much for violation.
Post a Comment
Read more

Prostitutes in Singapore

Singapore is very realistic about this issue. Prostitution is legal. The famous red light zone is Geylang area. I heard Hong Kong officials are considering to legalize this business in HK also. Singaporeans are not allow to work as prostitute. Maybe also SPR. The prostitutes are mainly foreign workers from poor countries in SEA area, such as Indonesia, Thailand, India, Malaysia, not from China. They come under special 2-year working pass, and must pass the health exam, yes, to prevent STD such as HIV, AIDS and so on. The brothels are mixed with normal resident houses. The brothel's house number is red lighted. The price is ranged from SGD 50 to SGD 200. Illegal sex workers are also around, and in some massage clinics. A very recent fatal case and newly effective government policy make a special social group - Chinese accompanying mom for studying kids a hot media buzz again.
Post a Comment
Read more