Thursday, January 28, 2016

How to build MongoRocks for MongoDB 3.2

This explains how I built MongoDB 3.2 from source with support for RocksDB thanks to help from Igor Canadi. There are more details here and here. My server uses Fedora.

# Install many of the dependencies for MongoRocks
sudo yum install snappy-devel zlib-devel bzip2-devel lz4-devel
sudo yum install scons gcc-g++ git

# Unpack MongoDB 3.2 source in $MONGOSRC

# Directory in which git repos are created
mkdir ~/git

# Get MongoRocks engine
cd ~/git
git clone https://github.com/mongodb-partners/mongo-rocks.git
cd mongo-rocks
git checkout --track origin/v3.2 -b v32


# get and build RocksDB libraries
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
git checkout --track origin/4.4.fb -b 44fb
make static_lib

# prepare source build with support for RocksDB
cd $MONGOSRC
mkdir -p src/mongo/db/modules/
ln -sf ~/git/mongo-rocks src/mongo/db/modules/rocks

# build mongod & mongo binaries
# if you have zstd installed then use LIBS="lz4 zstd"
# if you need to debug and see command lines then add
#     --debug=presub

scons CPPPATH=/home/mdcallag/git/rocksdb/include \
      LIBPATH=/home/mdcallag/git/rocksdb \

      LIBS=lz4 mongod mongo

# install mongod
mkdir -p ~/b/m321
cd ~/b/m321
mkdir data
mkdir bin
cp $MONGOSRC/build/opt/mongo/mongod bin
cp $MONGOSRC/build/opt/mongo/mongo bin

# create mongo.conf file with the text that follows. You must
# change $HOME and consider changing 
the value for cacheSizeGB
---
processManagement:
  fork: true
systemLog:
  destination: file
  path: $HOME/b/m321/log
  logAppend: true
storage:
  syncPeriodSecs: 600
  dbPath: $HOME/b/m321/data
  journal:
    enabled: true
operationProfiling.slowOpThresholdMs: 2000
replication.oplogSizeMB: 4000
storage.rocksdb.cacheSizeGB: 40
---

# start mongod, consider using numactl --interleave=all
bin/mongod --config mongo.conf --master --storageEngine rocksdb

# confirm RocksDB is there
$ head -1 data/db/LOG

2016/01/28-11:54:15.738641 7f7bd45d5e80 RocksDB version: 4.4.0

7 comments:

  1. Hi,
    I'm getting the error
    scons CPPPATH=/vol/mongo/git/rocksdb/include LIBPATH=/vol/mongo/git/rocksdb LIBS=lz4 mongod mongo

    scons: *** No SConstruct file found.
    File "/usr/lib/scons/SCons/Script/Main.py", line 905, in _main

    And there is no SConstruct file in the mongo-rocks dir
    ls -lh
    total 16K
    -rw-rw-r-- 1 ubuntu ubuntu 239 Feb 24 13:22 build.py
    -rw-rw-r-- 1 ubuntu ubuntu 601 Feb 24 13:22 README.md
    -rw-rw-r-- 1 ubuntu ubuntu 2.7K Feb 24 13:56 SConscript
    drwxrwxr-x 3 ubuntu ubuntu 4.0K Feb 24 13:41 src

    If I rename the SConscript file to SConstuct
    The error is
    scons CPPPATH=/vol/mongo/git/rocksdb/include LIBPATH=/vol/mongo/git/rocksdb LIBS=lz4 mongod mongo
    scons: Reading SConscript files ...

    scons: *** Import of non-existent variable ''env''
    File "/vol/mongo/git/mongo-rocks/SConstruct", line 1, in

    Any points how to resolve it?
    Thank you

    ReplyDelete
    Replies
    1. From which directory do you run scons? I run it from $MONGOSRC which is the top directory in which I unpacked the MongoDB source distribution. I don't run it from mongo-rocks which is a subdirectory of $MONGOSRC

      Delete
    2. The directory where the mongo-rocks is unpacked is /vol/mongo/git
      I'm using ubuntu 14.04
      The error is same, even in the top directory.

      ubuntu@ip-10-4-2-67:/vol/mongo/git$ ls -la
      total 16
      drwxrwxr-x 4 ubuntu ubuntu 4096 Feb 24 14:43 .
      drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 24 12:50 ..
      drwxrwxr-x 4 ubuntu ubuntu 4096 Feb 24 14:43 mongo-rocks
      drwxrwxr-x 19 ubuntu ubuntu 4096 Feb 24 14:44 rocksdb
      ubuntu@ip-10-4-2-67:/vol/mongo/git$ scons CPPPATH=/vol/mongo/git/rocksdb/include LIBPATH=/vol/mongo/git/rocksdb LIBS=lz4 mongod mongo

      scons: *** No SConstruct file found.
      File "/usr/lib/scons/SCons/Script/Main.py", line 905, in _main
      ubuntu@ip-10-4-2-67:/vol/mongo/git$ cd mongo-rocks/
      ubuntu@ip-10-4-2-67:/vol/mongo/git/mongo-rocks$ scons CPPPATH=/vol/mongo/git/rocksdb/include LIBPATH=/vol/mongo/git/rocksdb LIBS=lz4 mongod mongo

      scons: *** No SConstruct file found.
      File "/usr/lib/scons/SCons/Script/Main.py", line 905, in _main

      Delete
    3. You skipped multiple steps from the instructions above.

      Delete
  2. Do you have a similar set of instructions for building MySQL on RocksDB ?

    Would you know what version/branch of https://github.com/facebook/mysql-5.6 will work with rocksdb 4.6 ?

    ReplyDelete
    Replies
    1. https://github.com/facebook/mysql-5.6/wiki/Build-Steps

      FB MySQL in github includes RocksDB as a submodule. Whatever it includes is the version that works for MyRocks.

      Delete
  3. This comment has been removed by the author.

    ReplyDelete

Fixing some of the InnoDB scan perf regressions in a MySQL fork

I recently learned of Advanced MySQL , a MySQL fork, and ran my sysbench benchmarks for it. It fixed some, but not all, of the regressions f...