Overview

Holmes Processing was born out of the need to rapidly process and analyze large volumes data in the computer security community. At its core, Holmes Processing is a catalyst for extracting useful information and generate meaningful intelligence. Furthermore, the robust distributed architecture allows the system to scale while also providing the flexibility needed to evolve.

I quoted the description from its website. The architecture looks like this:

Figure. 1 Architecture of Holmes-Processing

Figure. 1 Architecture of Holmes-Processing

Publication can be downloaded from here.

Install docker

Binaries for OSX is provided on their home page, docker is a great idea of virtualizing applications instead of the whole OS. It simply runs on all the platforms.

docker for OSX comes with docker engine, docker-compose, and also docker-machine.

Install RabbitMQ

It is a message queue help you process messages asynchronously. The easiest way of installing it is via homebrew:

brew update
brew install rabbitmq

This will install rabbitmq in /usr/local/Cellar/rabbitmq/VERSION/

Install Cassandra

Binaries can be downloaded from their home page. Unzip the tar-ball, and optionally add the $CASSANDRA_HOME/bin to your PATH.

Install RiakCS

  • distributed on nodes
  • scalable
  • peer-2-peer network
  • fault-tolorence by replicate
  • high availability
  • key-value data store

Passed for the moment, it worths awhile to take a look at fake-s3.

Installed fake-s3, run the fake-s3 server by:

fake3 -r YOUR_ROOT_FOLDER -p PORT

the fake-s3 server is up and running. You need then configure it in Holmes-Storage.

Install Go

Go-lange can be downloaded and installed without any problem. After installation, you need to do one thing: 

export $GOPATH=$HOME/go/workspace

To define the environment variable GOPATH, run command above as you like. Any binaries by go get xxx will be pulled & compiled in the workspace.

Install Holmes-Storage

It is basically a wrapper for coordinating different databases on clusters, and is responsible to interact with data. To install, run bash:

go get -v -x -u "github.com/HolmesProcessing/Holmes-Storage"

This will install binaries in your GOPATH. One problem occurred is the “magic.h” is missing, it turns out you need to install dev version of magicmime: libmagic, which can be easily installed by homebrew.

brew install libmagic

that’s it.

Holmes-Storage Config

Default config file is shipped, you can copy it and change your data storage settings accordingly.

Install Holmes-Totem

Totem provides a set of static file analysis services. This can be downloaded and built by:

git clone https://github.com/HolmesProcessing/Holmes-Totem.git
cd Holmes-Totem
sbt assembly

Of course you need to install sbt first by brew install sbt.

Run Totem services

To run totem services, checkout the individual services folder in src folder. In each service, you need to copy service.conf.example to service.conf.

Compile and run service in docker:

cd Holmes-Totem/conf/
cp docker-compose.yml.example docker-compose.yml
docker-compose up -d

This will bring up services in docker.

At this point, services are all running in docker. You can run Totem by:

java -jar ./target/scala-2.11/totem-assembly-0.5.0.jar

You will see then:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Using default config file: ./config/totem.conf
Configuring details for Totem Tasking
Configuring details for downloading objects
Configuring details for Rabbit queues
Configuring setting for Services
Completing configuration
Creating Totem Actors
Totem version 0.5.0 is running and ready to receive tasks
[INFO] [12/14/2016 17:10:56.873] [totem-akka.actor.default-dispatcher-4] [akka://totem/user/producer] RabbitProducer: exchange totem should be made

TODO

  • configure local storage
  • configure service confs
  • manual tasking totem
  • install holmes-dynamic
  • install holmes-gateway
  • install holmes-toolbox
  • install RiakCS or S3-like storage
  • …