All Posts

Serialize/Deserialize With Apache Arrow

Apache Arrow Apache Arrow is an in-memory columnar data format across various systems such as Apache Spark, Impala, Apache Drill. Arrow have a columnar data represent format - Value Vectors. There are various types of value vectors depending on the data type. In this post, I serialize NullableIntVector to a file and deserialize from it. Sample Code Getting Started The arrow-vector module is already in maven repos. pom.xml: 1<dependencies> 2 <!

Debugging Apache Drill on Windows

TL;DR I try to remote debug Apache Drill in Embedded Mode with IntelliJ on Windows10. Prerequirements Apache Drill installation Apache Drill source code(same version) IDEA IntelliJ Set DRILL_SHELL_JAVA_OPTS Using sqlline.bat for starting drill as embedded mode. We can set debug options to DRILL_SHELL_JAVA_OPTS. 1set DRILL_SHELL_JAVA_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 Timeout Configuration To avoid timeout for debugging, set drill.exec.rpc.user.timeout to 0 in drill-override.conf. drill-override.conf: 1drill.exec: { 2 cluster-id: "drillbits1", 3 zk.connect: "localhost:2181", 4 rpc: { 5 user: { 6 timeout: 0 7 } 8 } 9}

Migrate to Jekyll3

Blog upgrade once a year 昨年のGW中にOctopressのupgradeをしたのを最後にblogの更新をしなかったので、もう改めて手を入れる必要はないと思

Update Octopress to 3.0

大分blogの更新をサボっていましたが、GWでちょっと時間ができたので、Octopressを3.0にアップデートしました。最初に構築した際の

Getting Started with Quasar

Quasar - Fiber, Channel, Actor Quasarは軽量スレッド、GoライクなChannl、ErlangライクなActorや、非同期プログラミングツールを提供するJav

Debugging Maven Plugins

Maven plugin throw some exception eclipseでincubator-parquet-mrのコードをビルドする為に、mvn eclipse:eclipseを実行したと

CouchDB Source Code Reading part11

remainging code of couch_db:update_docs/4 前回まででcouch_db:doc_flush_atts/2の一連の処理を読み終えました。久々にcouch_db:update_do

CouchDB Source Code Reading part10

前回、アタッチメントをファイルに書き込む箇所を読んだのですが、書き込んだデータがアタッチメントかどうかをどのように判別できるようにしているの

CouchDB Source Code Reading part9

couch_db:update_doc/4 引き続きcouch_db:update_docs/4を。 couch_db.erl: 1 if (AllOrNothing) and (PreCommitFailures /= []) -> 2 {aborted, 3 lists:foldl(fun({#doc{id=Id,revs=Revs}, Ref},Acc) -> 4 case lists:keyfind(Ref,1,PreCommitFailures) of 5 {Ref, Error} -> 6 case Revs of 7 {Pos, [RevId|_]} -> 8 [{{Id,{Pos, RevId}}, Error} | Acc]; 9 {0, []} ->

CouchDB Source Code Reading part8

couch_db:update_doc/4 前回からの続きで、couch_db:update_docs/4の続きを。 couch_db.erl: 1 case (Db#db.validate_doc_funs /= []) orelse 2 lists:any( 3 fun({#doc{id= <<?DESIGN_DOC_PREFIX, _/binary>>}, _Ref}) -> 4 true; 5 ({#doc{atts=Atts}, _Ref}) -> 6 Atts /= [] 7 end, Docs3) of 8 true -> 9