All Posts

Serialize/Deserialize With Apache Arrow

Apache Arrow Apache Arrow is an in-memory columnar data format across various systems such as Apache Spark, Impala, Apache Drill. Arrow have a columnar data represent format - Value Vectors. There are various types of value vectors depending on the data type. In this post, I serialize NullableIntVector to a file and deserialize from it. Sample Code Getting Started The arrow-vector module is already in maven repos. pom.xml: 1<dependencies> 2 <!

Debugging Apache Drill on Windows

TL;DR I try to remote debug Apache Drill in Embedded Mode with IntelliJ on Windows10. Prerequirements Apache Drill installation Apache Drill source code(same version) IDEA IntelliJ Set DRILL_SHELL_JAVA_OPTS Using sqlline.bat for starting drill as embedded mode. We can set debug options to DRILL_SHELL_JAVA_OPTS. 1set DRILL_SHELL_JAVA_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 Timeout Configuration To avoid timeout for debugging, set drill.exec.rpc.user.timeout to 0 in drill-override.conf. drill-override.conf: 1drill.exec: { 2 cluster-id: "drillbits1", 3 zk.connect: "localhost:2181", 4 rpc: { 5 user: { 6 timeout: 0 7 } 8 } 9}

Migrate to Jekyll3

Blog upgrade once a year 昨年のGW中にOctopressのupgradeをしたのを最後にblogの更新をしなかったので、もう改めて手を入れる必要はないと思

Update Octopress to 3.0


Getting Started with Quasar

Quasar - Fiber, Channel, Actor Quasarは軽量スレッド、GoライクなChannl、ErlangライクなActorや、非同期プログラミングツールを提供するJav

Debugging Maven Plugins

Maven plugin throw some exception eclipseでincubator-parquet-mrのコードをビルドする為に、mvn eclipse:eclipseを実行したと

CouchDB Source Code Reading part11

remainging code of couch_db:update_docs/4 前回まででcouch_db:doc_flush_atts/2の一連の処理を読み終えました。久々にcouch_db:update_do

CouchDB Source Code Reading part10


CouchDB Source Code Reading part9

couch_db:update_doc/4 引き続きcouch_db:update_docs/4を。 couch_db.erl: 1 if (AllOrNothing) and (PreCommitFailures /= []) -> 2 {aborted, 3 lists:foldl(fun({#doc{id=Id,revs=Revs}, Ref},Acc) -> 4 case lists:keyfind(Ref,1,PreCommitFailures) of 5 {Ref, Error} -> 6 case Revs of 7 {Pos, [RevId|_]} -> 8 [{{Id,{Pos, RevId}}, Error} | Acc]; 9 {0, []} ->

CouchDB Source Code Reading part8

couch_db:update_doc/4 前回からの続きで、couch_db:update_docs/4の続きを。 couch_db.erl: 1 case (Db#db.validate_doc_funs /= []) orelse 2 lists:any( 3 fun({#doc{id= <<?DESIGN_DOC_PREFIX, _/binary>>}, _Ref}) -> 4 true; 5 ({#doc{atts=Atts}, _Ref}) -> 6 Atts /= [] 7 end, Docs3) of 8 true -> 9