Thursday, January 31, 2019

Hadoop and Spark Setup

#get centos container
git pull centos

#start centos with port mapping
docker run -d -p 8088:8088 -p 9870:9870 --name "centos" -i centos

#access centos
docker exec -it centos /bin/bash

#download the instructions
wget https://www.clustermonkey.net/download/LiveLessons/Hadoop_Fundamentals/Hadoop_Spark_Fundamentals_Code_Notes-V3.0.tgz
cd /tmp
tar xzvf Hadoop_Spark_Fundamentals_Code_Notes-V3.0.tgz

#follow the instructions in /tmp/Hadoop_Spark_Fundamentals_Code_Notes-V3.0/Lesson-2/Lesson-2.2/NOTES.txt to setup hadoop, pig, hive

a few catches
* adjust the JAVA_HOME and HADOOP_HOME accordingly based on your installation
* modify HADOOP_PATH in the script hadoop-setup-script.sh under /tmp/Hadoop_Spark_Fundamentals_Code_Notes-V3.0/Lesson-2/Lesson-2.2/scripts
* check if logs under /opt/hadoop-3.2.0/ is created
* after hadoop 3.0.0, the namenode port is 9870 instead of 50070
* before running yarn, set the YARN_CONF_LIB
export YARN_CONF_LIB=/opt/hadoop-3.2.0/etc/hadoop


##follow the instructions in /tmp/Hadoop_Spark_Fundamentals_Code_Notes-V3.0/Lesson-2/Lesson-2.3/NOTES.txt to setup spark


Sunday, May 27, 2018

More Principles - Clean Architecture: A Craftsman's Guide to Software Structure and Design, First Edition

REP: Reuse/Release Equivalence Principle, the granule of reuse is the granule of release
CCP: Common Closure Principle
CRP: Common Reuse Principle, don't force to depend on things they don't need
ADP: Acyclic Dependencies Principle, no cycles in the dependency graph
SDP: Stable Dependencies Principle, depend in the direction of stability
SAP: Stable Abstractions Principle, as abstract as stable

S.O.L.I.D

SRP: Single Responsibility Principle, a module should be responsible to one and only one actor
OCP: Open-Closed Principle, a software artifact should be open for extension but closed for modification
LSP: Liskov Substitution Principle, interchangeable parts adhere to a contract
ISP: Interface Segregation Principle, avoid depending on things not use
DIP: Dependency Inversion Principle, details should depend on policies

Thursday, March 01, 2018

generate pem from keypair, and sign message

openssl rsa -in id_rsa -outform pem > id_rsa.pem
openssl rsa -in id_rsa -pubout -out id_rsa.pub.pem
openssl dgst -sha256 -sign ~/.ssh/id_rsa.pem -out signature.data message
openssl dgst -sha256 -verify ~/.ssh/id_rsa.pub.pem -signature signature.data message

start the ethereum private net on windows



download/install geth from https://github.com/ethereum/go-ethereum/wiki/Installation-instructions-for-Windows

setup private net jason file under ~/.privether, an example of privategenesis.json

{
  "nonce": "0x00006d6f7264656e",
  "difficulty": "0x000002",
  "mixhash": "0x00000000000000000000000000000000000000647572616c65787365646c6578",
  "coinbase": "0x0000000000000000000000000000000000000000",
  "timestamp": "0x00",
  "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "extraData": "0x",
  "gasLimit": "0x2FEFD8",
  "config": {
        "chainId": 15,
        "homesteadBlock": 0,
        "eip155Block": 0,
        "eip158Block": 0
  },
  "alloc": {
    "cd2a3d9f938e13cd947ec05abc7fe734df8dd826": { "balance": "10000000000000000" }
  }
}

initialize the private net and specify a network id

geth --datadir ~/.ethereum/privatenet init .privether/privategenesis.json
geth --datadir ~/.ethereum/privatenet --networkid=346

in another console, attach to this private net

geth attach ipc:\\.\pipe\geth.ipc

and happy mining!


> personal.newAccount("Password123")
"0x01e08a06618d5a96956080ead0c8fc2452cf3311"
> miner.setEtherbase(personal.listAccounts[0])
true
> eth.getBalance(eth.coinbase).toNumber()
0
> miner.start(2)
null
> eth.getBalance(eth.coinbase).toNumber()
1.415e+21


Sunday, October 30, 2016

install dc/os on centos 7

first install and start docker
sudo yum -y install docker docker-registry
[osboxes@osboxes ~]$ sudo systemctl enable docker.service
[osboxes@osboxes ~]$ sudo systemctl start docker.service
[osboxes@osboxes ~]$ sudo systemctl status docker.service


install based on https://dcos.io/docs/1.8/administration/installing/custom/gui/

if you meet permission issue, you need disable selinux

[osboxes@osboxes ~]$ cat /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted







Thursday, October 27, 2016

setup mesos 1.0.1 on centos 7

following the guide at https://mesos.apache.org/gettingstarted/

if you met the following issue, need hack it to change the Version: 2012d to 2012 in file pytz-2012d-py2.7.egg-info


pkg_resources.VersionConflict: (pytz 2012d (/usr/lib/python2.7/site-packages), Requirement.parse('pytz>=2010'))

python examples runs ok with mesos 1.0.1

/home/osboxes/mesos/build/3rdparty/setuptools-20.9.0/pkg_resources/__init__.py:1268: UserWarning: /home/osboxes/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).
  warnings.warn(msg, UserWarning)
I1031 02:04:19.299875 13734 sched.cpp:226] Version: 1.0.1
I1031 02:04:19.312309 13773 sched.cpp:330] New master detected at master@10.88.48.57:5050
I1031 02:04:19.313021 13773 sched.cpp:341] No credentials provided. Attempting to register without authentication
I1031 02:04:19.322294 13773 sched.cpp:743] Framework registered with 6a67bc4f-1339-42e0-97bd-dd3243887c2a-0006
Registered with framework ID 6a67bc4f-1339-42e0-97bd-dd3243887c2a-0006
Received offer 6a67bc4f-1339-42e0-97bd-dd3243887c2a-O15 with cpus: 1.0 and mem: 6609.0
Launching task 0 using offer 6a67bc4f-1339-42e0-97bd-dd3243887c2a-O15
Task 0 is in state TASK_RUNNING
Task 0 is in state TASK_FINISHED
Received message: 'data with a \x00 byte'
Received offer 6a67bc4f-1339-42e0-97bd-dd3243887c2a-O16 with cpus: 1.0 and mem: 6609.0
Launching task 1 using offer 6a67bc4f-1339-42e0-97bd-dd3243887c2a-O16
Task 1 is in state TASK_RUNNING
Task 1 is in state TASK_FINISHED
Received message: 'data with a \x00 byte'
Received offer 6a67bc4f-1339-42e0-97bd-dd3243887c2a-O17 with cpus: 1.0 and mem: 6609.0
Launching task 2 using offer 6a67bc4f-1339-42e0-97bd-dd3243887c2a-O17
Task 2 is in state TASK_RUNNING
Task 2 is in state TASK_FINISHED
Received message: 'data with a \x00 byte'
Received offer 6a67bc4f-1339-42e0-97bd-dd3243887c2a-O18 with cpus: 1.0 and mem: 6609.0
Launching task 3 using offer 6a67bc4f-1339-42e0-97bd-dd3243887c2a-O18
Task 3 is in state TASK_RUNNING
Task 3 is in state TASK_FINISHED
Received message: 'data with a \x00 byte'
Received offer 6a67bc4f-1339-42e0-97bd-dd3243887c2a-O19 with cpus: 1.0 and mem: 6609.0
Launching task 4 using offer 6a67bc4f-1339-42e0-97bd-dd3243887c2a-O19
Task 4 is in state TASK_RUNNING
Task 4 is in state TASK_FINISHED
All tasks done, waiting for final framework message
Received message: 'data with a \x00 byte'
All tasks done, and all messages received, exiting
I1031 02:04:26.495678 13773 sched.cpp:1987] Asked to stop the driver
I1031 02:04:26.495754 13773 sched.cpp:1187] Stopping framework '6a67bc4f-1339-42e0-97bd-dd3243887c2a-0006'

I1031 02:04:26.497079 13734 sched.cpp:1987] Asked to stop the driver


Hadoop and Spark Setup

#get centos container git pull centos #start centos with port mapping docker run -d -p 8088:8088 -p 9870:9870 --name "centos" ...