Import Mordor Data

Catapult

You can simply download the json files available in this repo and start using some grep fu! if you feel like it. However, there are other more efficient ways you can consume the pre-recorded data and even simulate a real data pipeline to ingest the data to your own SIEM or data lake.

Kafkacat Style

You can start using a tool named Kafkacat to act as a Kafka producer and send data to Kafka brokers. In producer mode, Kafkacat reads messages from standard input (stdin) or a file. This means that you can send data back to any other Kafka broker that you are using as part of your pipeline. You can just grab the logs from this repo and re-play them as if they were being ingested in real-time.

Requirements

  • Kafka Broker : A distributed publish-subscribe messaging system that is designed to be fast, scalable, fault-tolerant, and durable (Installed by HELK).
  • Kafkacat : A generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka.
  • HELK (Basic Option) : An elastic ELK (Elasticsearch, Logstash, Kibana) stack.
  • Docker CE : Docker Community Edition (CE) is ideal for developers and small teams looking to get started with Docker and experimenting with container-based apps (Installed by HELK).
  • Docker Compose : a tool for defining and running multi-container Docker applications (Installed by HELK).
Kafkacat Infrastructure

Consume Logs

Install Kafkacat following the instructions from the official Kafkacat repo.

  • If you are using a debian-based system, make sure you install the latest Kafkacat deb package.
  • I recommend at least Ubuntu 18.04. You can check its Kafkacat deb package version and compare it with the latest one in the Kafkacat GitHub repo.
  • You can also install it from source following the Quick Build instructions.

Download and run the HELK. Make sure you have enough memory to run the basic build. You can run it with 5-6GB of RAM now (More information here).

$ git clone https://github.com/Cyb3rWard0g/HELK.git
$ cd HELK/docker
$ sudo ./helk_install

Use the defaults (Option 1 and Basic license)

**********************************************
**          HELK - THE HUNTING ELK          **
**                                          **
** Author: Roberto Rodriguez (@Cyb3rWard0g) **
** HELK build version: v0.1.7-alpha02262019 **
** HELK ELK version: 6.6.1                  **
** License: GPL-3.0                         **
**********************************************

[HELK-INSTALLATION-INFO] HELK being hosted on a Linux box
[HELK-INSTALLATION-INFO] Available Memory: 12541 MBs
[HELK-INSTALLATION-INFO] You're using ubuntu version xenial

*****************************************************
*      HELK - Docker Compose Build Choices          *
*****************************************************

1. KAFKA + KSQL + ELK + NGNIX + ELASTALERT
2. KAFKA + KSQL + ELK + NGNIX + ELASTALERT + SPARK + JUPYTER

Enter build choice [ 1 - 2]: 1
[HELK-INSTALLATION-INFO] HELK build set to 1
[HELK-INSTALLATION-INFO] Set HELK elastic subscription (basic or trial): basic
[HELK-INSTALLATION-INFO] Set HELK IP. Default value is your current IP: 192.168.64.138
[HELK-INSTALLATION-INFO] Set HELK Kibana UI Password: hunting
[HELK-INSTALLATION-INFO] Verify HELK Kibana UI Password: hunting
[HELK-INSTALLATION-INFO] Installing htpasswd..
[HELK-INSTALLATION-INFO] Installing docker via convenience script..
[HELK-INSTALLATION-INFO] Installing docker-compose..
[HELK-INSTALLATION-INFO] Checking local vm.max_map_count variable and setting it to 4120294
[HELK-INSTALLATION-INFO] Building & running HELK from helk-kibana-analysis-basic.yml file..

Download the mordor repo and choose your technique:

$ cd ../../
$ git clone https://github.com/Cyb3rWard0g/mordor.git
$ cd mordor/small_datasets/windows/credential_access/credential_dumping_T1003/credentials_from_ad/

Decompress the specific mordor log file

$ tar -xzvf empire_dcsync.tar.gz
x empire_dcsync_2019-03-01174830.json

Send the data to HELK via Kafcakat with the following flags:

-b

Kafka Broker

-t

Topic in the Kafka Broker to send the data to

-P

Producer mode

-l

Send messages from a file separated by delimiter, as with stdin. (only one file allowed)

$ kafkacat -b <HELK IP>:9092 -t winlogbeat -P -l empire_dcsync_2019-03-01174830.json

Browse to your Kibana Discover view and start going through the data

DCSync

You could look for potential DCSync actvity from a non-Domain-Controller account with the following query in Kibana:

event_id:4662 NOT user_name:*$ AND object_properties:("*1131f6aa-9c07-11d1-f79f-00c04fc2dcd2*" OR "*1131f6ad-9c07-11d1-f79f-00c04fc2dcd2*" OR "*89e95b76-444d-4c62-991a-0facbeda640c*")
DCSync Found

Jupyter Notebook Style

You can consume mordor data directly with a Jupyter notebook and analyze it via python libraries such as Pandas.

Requirements

  • Docker CE : Docker Community Edition (CE) is ideal for developers and small teams looking to get started with Docker and experimenting with container-based apps.
  • Jupyter Notebook : an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

Consume Logs

Install docker by following the official Docker instructions.

Download the mordor repo

$ git clone https://github.com/Cyb3rWard0g/mordor.git

Run a HELK dockerized Jupyter Notebook server and mount your mordor folder to it with the following command:

$ docker run -p 127.0.0.1:8888:8888 --env JUPYTER_TYPE=notebook -v $PWD/mordor:/opt/helk/jupyter/notebooks/mordor -it cyb3rward0g/helk-jupyter:0.1.2
[HELK-JUPYTER-DOCKER-INSTALLATION-INFO] Starting Jupyter..
[HELK-JUPYTER-DOCKER-INSTALLATION-INFO] Running Jupyter Type: notebook..
[HELK-JUPYTER-DOCKER-INSTALLATION-INFO] Running the following parameters --ip=0.0.0.0 --port=8888 --notebook-dir=/opt/helk/jupyter/notebooks --no-browser --NotebookApp.base_url=/
[I 03:27:32.369 NotebookApp] Writing notebook server cookie secret to /home/jupyter/.local/share/jupyter/runtime/notebook_cookie_secret
[I 03:27:32.560 NotebookApp] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
[I 03:27:32.561 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 03:27:32.563 NotebookApp] Serving notebooks from local directory: /opt/helk/jupyter/notebooks
[I 03:27:32.563 NotebookApp] The Jupyter Notebook is running at:
[I 03:27:32.564 NotebookApp] http://(2e83a98485eb or 127.0.0.1):8888/?token=90311c8670ed2bd71f7d9e8378fdc39711ef65a0b3ed6296
[I 03:27:32.564 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 03:27:32.568 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/jupyter/.local/share/jupyter/runtime/nbserver-76-open.html
    Or copy and paste one of these URLs:
        http://(2e83a98485eb or 127.0.0.1):8888/?token=90311c8670ed2bd71f7d9e8378fdc39711ef65a0b3ed6296

Browse to 127.0.0.1:8888 in your favorite browser and enter the token provided in the jupyter output above

Jupyter login

You will be taken to the Jupyter main interface

Jupyter main men

Create a new notebook with kernel Python 3

Jupyter new notebook

You can go through the directory tree of the mordor project and even hit [TAB] for auto-completion to get to a specific technique

Create a new notebook

Jupyter list mordor files

Decompress the mordor file you want to work with. Let’s pick a dcsync example.

Jupyter decompress file

Use pandas to read the file. You are ready to start exploring and analyzing the data

Jupyter Pandas