Monday, 10 April 2017

Event Sourcing and CQRS

The real-world is all about collaboration, intentions and eventually dealing with issues by compensating use-cases for exception handling.

Events:
  • Atomic
  • Immutable
  • Happened in the past
  • Reflect use-cases
  • Single source of truth
Event-driven:

  • Eventually consistent
  • Split-up use-cases & transactions
  • Communication via events
  • Events published reliably

Event Sourcing

Enables building a forward-compatible application architecture - the ability to add more application in the future that need to process the same event but create a different materialized view.
The essence of event sourcing: rather than performing destructive state mutation on a database when writing to it, we should record every write as a "command", as an immutable event.

All changes to application state are stored as a sequence of events. This sequence of events can be used to query the current state of an object but also the history of an object. Event sourcing allows one to create different kinds of views, even views which one did not even think of when creating the application. Event store only directly supports PK-based lookup => use CQRS to handle.


Command Query Responsibility Segregation(CQRS)

An application architecture pattern most commonly used with event sourcing. CQRS involves splitting an application into two parts internally - the command side ordering the system to update state and the query side that gets information without changing state. The command or write side is all about the business; it doesn't care about the queries. On the other hand, the query or read side is all about the read access; its main purpose is making queries fast and efficient.

The way event sourcing works with CQRS is to have part of the application models updates as writes to an event log or Kafka topic. This is paired with an event handler that subscribes to the Kafka topic, transforms the event and writes the materialized view to a read store. Finally, the read part of the application issues queries against the read store.

CQRS decouples the load from writes and reads allowing each to be scaled and optimized independently.The input events are quite simple. They are immutable facts, we can simply store them all. The database we read from is just a cached view of the event log.
The beautiful thing about this separation between source of truth and caches is that in your caches, you can denormalize data to your content.

To perform an action in the domain, you have to send a command to it. The command will be validated and processed. This will lead to one or more events that are stored in the journal. This is the Command part of CQRS.
To create a view of the domain model (which is eventsourced) you have to collect the events and process them into a view. This is Query part of CQRS.

Advantages:

If you write data to the database in the same schema as you use for reading, you have tight coupling between the part of the application doing the writing and the part doing the reading. Otherwise, you can have fast writes and fast reads. If data is immutable, you can always replay events in the same order when something is wrong.

Why Scala and Akka?

The Akka actor model works on the basis of processing a command and outputting one or many events that represent the result of processing that command.

Scala is a great language for implementing ES-based domain models:
  1. Case classes
  2. Pattern matching
  3. Recreating state = functional fold over events
Implementation: 

Partition the domain model into Aggregates. For each aggregate (aka. business entity):
  • Identify (state-changing) domain events
  • Define Event classes,e.g. OrderCreated, OrderCancelled.
Aggregate granularity is important

  • Transaction = processing one command by one aggregate
  • No opportunity to update multiple aggregates within a transaction
  • If an update must be atomic (i.e. no compensating
  • transaction) then it must be handled by a single aggregate
  • Preserved history ⇒ More easily implement future requirements
Designing commands
  • Created by a service from incoming request
  • Processed by an aggregate
  • Immutable
  • Contains value objects for:
    Validating request
    Creating event
    Auditing user activity
Normalization vs Denormalization
In regular databases, it is often considered best parctice to normalize data, because if something changes, you then only have to change it one place. Normalization makes writes fast and simple, but means you have to do joins at read time. In order to speed up reads, you can denormalize data, duplicate information in various places so that it can be read faster.

Tuesday, 21 March 2017

AWS Network and Security

VPC:

Virtual Private Cloud enables you to launch AWS resources into a virtual network that you've defined. An internet Gateway allows your VPC to connect to the Internet.
An Internet gateway is used to enable outbound access to the Internet from VPC the clusters.


Subnet:

A range of IP addresses in your VPC. Use a public subnet for resources that must be connected to the Internet and a private subnet for resources that won't be connected to the Internet.


Route Table:

It determines where network traffic is directed. Every subnet has to be associated with a route table.
A route table is used to connect the subnet to the Internet gateway.

IP Addresses:

Private IP is not reachable over the Internet.
Public IP is reachable from the Internet.
Elastic IP is a static or public persistent IP, even after the instance is rebuilt.


NAT device:

It enables instances in a private subnet to connect to the Internet or other AWS service.


Security Groups:

A security group acts as a virtual firewall that controls the traffic for one or more instances.
you add rules to each security group that allow traffic to or from its associated instances.


Network ACL:

A network access control list is an optional layer of security for your VPC.


Friday, 10 March 2017

DB2 Alter Column

1. If a table is not empty, and we want to alter a column as NOT NULL.

ALTER TABLE my_table ALTER COLUMN name SET NOT NULL WITH DEFAULT 'None';


2. If we want the column to be nullable,

ALTER TABLE my_table DATA CAPTURE NONE;

ALTER TABLE my_table ALTER region DROP NOT NULL;


ALTER TABLE my_table DATA CAPTURE CHANGES; 

Since Drop column, and alter nullability is not allowed on any table with data capture on.
The data capture has to be changed.

After the ALTER TABLE statement, The access to the table is restricted.
You need to run the REORG command as follows:

CALL SYSPROC.ADMIN_CMD('REORG TABLE schema.table');

Install NodeJs

1. 
$ sudo node install
Errors:
> node-gyp configure buildgyp: Call to 'node -e "require('nan')"' returned exit status 127 while in binding.gyp. while trying to load binding.gypgyp ERR! configure errorgyp ERR! stack Error: `gyp` failed with exit code: 1
The Reason: Some npm plugins need node-gyp to be installed.
$ npm install --global node-gyp
2. 
To update Node, you’ll need npm’s handy n module. Run this code to clear npm’s cache, install n, and install the latest stable version of Node:
$ npm cache clean -f
$ npm install -g n

To install the latest release,
$ n latest

Alternatively, you can run
$ n #.#.#
to get a specific Node version(upgrading or downgrading).
Error when run node script: Module version mismatch. Expected 48, got 47

Solution: reinstall node

1. Remove node_modules directory completely (rm -rf node_modules)
2. Please ensure that you don't use sudo
3. npm install

bower install
Error
/usr/bin/env: node: No such file or directory
Solution
sudo ln -s /usr/bin/nodejs /usr/bin/node
grunt install
Error
grunt: command not found

Solution
npm install grunt-cli -g

Reference:
https://docs.npmjs.com/misc/removing-npm

Sunday, 12 February 2017

Scala Java 8 SAM



Java 8 introduces SAM(Single Abstract Method) type to embrace functional programming.
SAM type is enabled by -Xexperimental flag in scala 2.11.x flags. In build.sbt, add below:
scalacOptions := Seq("-unchecked", "-deprecation", "-Xexperimental")

Interoperating with Java requires SAM, which also generates more efficient byte code since SAM has a native byte code counterpart. Using anonymous class for event handler or callback can be more pleasant in Scala just as in Java.

So for the stream example, if compiler has the -Xexperimental flag, scala will automatically change the function to java’s function, which grant scala user a seamless experience with the library.

Reference:
https://herringtondarkholme.github.io/2015/01/24/scala-sam/

Sunday, 29 January 2017

Ethereum Overview

What is Blockchain?

A blockchain is a distributed computing architecture where every network node executes and records the same transactions, which are grouped into blocks. Only one block can be added at a time, and every block contains a mathematical proof that verifies that it follows in sequence from the previous block.

What is Ethereum?

Ethereum is a programmable blockchain, a suite of protocols. Ethereum is suited for applications that automate direct interaction between peers or facilitate coordinated group action across a network.

What is Contract?

Ethereum's basic unit is the account.

  • Externally Owned Accounts(EOAs)
    Has an Ether balance,
    Can send transactions(ether transfer or trigger contract code)
    Is controlled by private keys.
    Has no associated code.
  • Contract Accounts,
    Has an Ether balance
    Has associated code
    Code execution is triggered by transactions or messages received from other contracts.
All action on the Ethereum block chain is set in motion by transactions fired from externally owned accounts.
Every time a contract account receives a transaction, its code is executed as instructed by the input parameters sent as part of the transaction. 
The contract code is executed by the EVM on each node participating in the network as part of their verification of new blocks.


What is Smart Contract?

The smart contracts refer to code in a Contract Account : programs that execute when a transaction is sent to that account. Ethereum requires nodes to be able to agree on the outcome of computation, which requires a guarantee of strictly deterministic execution.

A Contract is a collection of code (its function) and data (its state) that resides at a specific address on the Ethereum blockchain. Contracts live on the blockchain in a Ethereum-specific binary format called EVM bytecode. Contracts are typically written in a higher level language and then compiled using the EVM compiler into bytecode to be deployed to the blockchain.

EVM is completely isolated. The code running inside the EVM has no access to network, filesystem, or other processes. Smart contracts even have limited access to other smart contracts.

Ethereum contracts CANNOT pull data from external information sources.

Data and contracts on the Ethereum network are encoded, but not encrypted. Everyone can audit the behavior of the contracts and the data sent to them.

What is Transaction and Message?

Transaction refers to the signed data package that stores a message to be sent from an externally owned account to another account on the blockchain.

Contracts have the ability to send "messages" to other contracts. Messages are virtual objects that are never serialized and exist only in the Ethereum execution environment. A message is like a transaction, except it is produced by a contract and not an external actor.


What is Transaction Fee and Gas?

The sender of a transaction must pay for each step of the "program" they activated, including computation and memory storage.

These transaction fees are collected by the nodes that validate the network. These "miners" are nodes in the Ethereum network that receive, propagate, verify, and execute transactions. The miners then group the transactions into what are called blocks.

Mining is also the way to secure the network by creating, verifying, publishing and propagating blocks in the blockchain. A block is only valid if it contains proof of work of a given difficulty.
Unlike Bitcoin, Ethereum blocks contain a copy of both the transaction list and the most recent state.
Ethash PoW is memory hard, making it ASIC resistant.

Gas is a central part of every network request and requires the sender to pay for the computing resources consumed. The principle behind Gas is to have a stable value for how much a transaction or computation costs on the Ethereum network. Each operation in the EVM was assigned a number of how much gas it consumes.

Every node participating in the network runs the EVM as part of the block verification protocol. Each and every full node in the network does the same calculations and stores the same values. Its parallel processing is redundantly parallel.
It is to offer an efficient way to reach consensus on the system state without needing trusted third parties. Don't use the blockchain for computation that can be done offchain.

How to connect?

Geth finds peers through discovery protocol. Nodes are gossipping with each other to find out about other nodes on the network. In order to get going initially, geth uses a set of bootstrap nodes whose endpoints are recorded in the source code.



Friday, 27 January 2017

Kafka Operation Notes

Hardware Requirements:

Unlike some systems, Kafka itself doesn't require a lot of RAM. Kafka makes use of the operation system's page cache to hold recently-used data. But more memory will improve performance because of a larger pagecache.

Kafka brokers have a relatively small memory footprint. Extra RAM will be used by the operating system for disk caching.
Kafka is heavily multi-threaded, favor more cores over faster cores.

Typical JVM options:

-Xms6g -Xmx6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M

-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80

Avoid clusters that span datacenters.

Zookeeper is sensitive to I/O latency. Make sure that it has its own disk.
Run a Zookeeper quorum of 3 or 5 nodes.

Each broker must have its own unique ID. Set the advertised.listeners property.

To Delete a Topic:
All brokers must have the "delete.topic.enable" set to true. Otherwise, the delete command will be silently ignored.

What does Committed Really Mean?

  • Data is received by all the replicas in the ISR(In Sync Replicas)
  • Not related to the ack setting chosen by the producer
  • Committed state is checkpointed to disk
  • Data can't be seen until it is committed

Leader maintains the latest committed offset.
Replica is added to the ISR when it is fully caught up.
Control the lag between leader and replica: replica.lag.time.max.ms
If too large, replicas will slow down writes
If too small, replicas will drop in and out of ISR.

Controller Broker:

Detects broker failure/restart via Zookeeper.
When the leader fails, controller selects a new leader and updates the ISR.
Persists the new leader and ISR to Zookeeper.
Sends the new leader/ISR change to all brokers

Check Topic Partition and Replica:
$ kafka-topics --zookeeper localhost:2181 --describe --topic mytopic

Important log4j Files:

controller.log: Logs all Broker failures, and actions taken because of them.
state-change.log: Logs every decision it has received from the controller.

Group Coordinator:

  • Each Consumer Group has a GroupCoordinator(elected Broker)
  • Consumers heartbeat to the GroupCoordinator
  • Lack of heartbeats causes a rebalance
  • During rebalance, consumption is paused.
  • Group Coordinator makes one consumer as Group Leader.
    Only the leader gets the list of group members.
    Group leader calls the partition assigner to assign consumers to partitions.

Manual Offset Management:

Set auto.commit.offset = false
commitSync(): Includes retry logic
commitAsync(): No retries, has potential to reorder
Consider combinations of sync and async

Log Compaction: old value for the key are deleted.
Uses: Database change capture; Stateful stream processing; Event sourcing.

Number of Partitions:
More partitions means higher throughput.
However, it requires more open file handles, increase unavailability, end-to-end latency, more memory in the client.

Check Consumer Offsets:

$ kafka-consumer-groups --group my-group --describe --new-consumer

--bootstrap-server=localhost:9090