Sunday, January 26, 2020




Thursday, January 31, 2019

Hadoop and Spark Setup

#get centos container
git pull centos

#start centos with port mapping
docker run -d -p 8088:8088 -p 9870:9870 --name "centos" -i centos

#access centos
docker exec -it centos /bin/bash

#download the instructions
cd /tmp
tar xzvf Hadoop_Spark_Fundamentals_Code_Notes-V3.0.tgz

#follow the instructions in /tmp/Hadoop_Spark_Fundamentals_Code_Notes-V3.0/Lesson-2/Lesson-2.2/NOTES.txt to setup hadoop, pig, hive

a few catches
* adjust the JAVA_HOME and HADOOP_HOME accordingly based on your installation
* modify HADOOP_PATH in the script under /tmp/Hadoop_Spark_Fundamentals_Code_Notes-V3.0/Lesson-2/Lesson-2.2/scripts
* check if logs under /opt/hadoop-3.2.0/ is created
* after hadoop 3.0.0, the namenode port is 9870 instead of 50070
* before running yarn, set the YARN_CONF_LIB
export YARN_CONF_LIB=/opt/hadoop-3.2.0/etc/hadoop

##follow the instructions in /tmp/Hadoop_Spark_Fundamentals_Code_Notes-V3.0/Lesson-2/Lesson-2.3/NOTES.txt to setup spark

Sunday, May 27, 2018

More Principles - Clean Architecture: A Craftsman's Guide to Software Structure and Design, First Edition

REP: Reuse/Release Equivalence Principle, the granule of reuse is the granule of release
CCP: Common Closure Principle
CRP: Common Reuse Principle, don't force to depend on things they don't need
ADP: Acyclic Dependencies Principle, no cycles in the dependency graph
SDP: Stable Dependencies Principle, depend in the direction of stability
SAP: Stable Abstractions Principle, as abstract as stable


SRP: Single Responsibility Principle, a module should be responsible to one and only one actor
OCP: Open-Closed Principle, a software artifact should be open for extension but closed for modification
LSP: Liskov Substitution Principle, interchangeable parts adhere to a contract
ISP: Interface Segregation Principle, avoid depending on things not use
DIP: Dependency Inversion Principle, details should depend on policies

Thursday, March 01, 2018

generate pem from keypair, and sign message

openssl rsa -in id_rsa -outform pem > id_rsa.pem
openssl rsa -in id_rsa -pubout -out
openssl dgst -sha256 -sign ~/.ssh/id_rsa.pem -out message
openssl dgst -sha256 -verify ~/.ssh/ -signature message

start the ethereum private net on windows

download/install geth from

setup private net jason file under ~/.privether, an example of privategenesis.json

  "nonce": "0x00006d6f7264656e",
  "difficulty": "0x000002",
  "mixhash": "0x00000000000000000000000000000000000000647572616c65787365646c6578",
  "coinbase": "0x0000000000000000000000000000000000000000",
  "timestamp": "0x00",
  "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "extraData": "0x",
  "gasLimit": "0x2FEFD8",
  "config": {
        "chainId": 15,
        "homesteadBlock": 0,
        "eip155Block": 0,
        "eip158Block": 0
  "alloc": {
    "cd2a3d9f938e13cd947ec05abc7fe734df8dd826": { "balance": "10000000000000000" }

initialize the private net and specify a network id

geth --datadir ~/.ethereum/privatenet init .privether/privategenesis.json
geth --datadir ~/.ethereum/privatenet --networkid=346

in another console, attach to this private net

geth attach ipc:\\.\pipe\geth.ipc

and happy mining!

> personal.newAccount("Password123")
> miner.setEtherbase(personal.listAccounts[0])
> eth.getBalance(eth.coinbase).toNumber()
> miner.start(2)
> eth.getBalance(eth.coinbase).toNumber()

Sunday, October 30, 2016

install dc/os on centos 7

first install and start docker
sudo yum -y install docker docker-registry
[osboxes@osboxes ~]$ sudo systemctl enable docker.service
[osboxes@osboxes ~]$ sudo systemctl start docker.service
[osboxes@osboxes ~]$ sudo systemctl status docker.service

install based on

if you meet permission issue, you need disable selinux

[osboxes@osboxes ~]$ cat /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.