Recently I’ve been busy with CogSec Collab and CTI League helping set up a counter disinformation community to do something about the dumpster fire known as plague in the golden age of disinformation. It’s been real. Between sipping on bleach cocktails and trying to build a better bat trap I’ve had opportunity to e-meet a lot of very interesting, very quarantined infosec frens.

There are too many to go into detail so I’ll just shout out the latest person I stole source code from. Thank you Emilio Escobar for writing the Slack bot this project is based off. His awesome flask-sigauth library makes it easy to authenticate a Slack bot to a Flask app.

To get started clone the bot code here.

MISP Objects

A MISP object is a JSON file containing some object metadata and an allowed set of attributes. We use objects to describe complex things such as a Twitter post, ELF section, or a Bitcoin transaction. The MISP object GitHub page documents the object properties.

This is a microblog object used to represent a Twitter post.

{
"name": "microblog", # the name of the object
"meta-category": "misc", # category of the object. allowed values in (file, network, financial, misc, internal)
"description": "A microblog post from Twitter, Reddit or similar.", # some info about the purpose of the object
"version": 1, # increment the version each time the object is modified
"uuid": "f47559d7-6c16-40e8-a6b0-eda4a008376f", # UUID must be unique for each object
"attributes" : # list of attributes contained in the object
  {
    "post-id": {  # object attribute name can be anything
            "misp-attribute": "text", # attribute type must be a valid MISP attribute type
            "ui-priority": 1, # frequency usage orders the attributes. 1 is the standard value.
            "multiple": false, # whether to allow multiple instances of this attribute in the object
            "disable_correlation": false # whether to disable correlation on this attribute
    },
    "url": {
            "misp-attribute": "url",
            "ui-priority": 1,
            "categories": ["Network activity","External analysis"] # attribute category
    }
  },
"required": ["post-id","url"]
}

Create a MISP Object Definition

We’re going to use the microblog object to push a Twitter post to MISP. Before doing that it’s worth quickly looking at how you can build your own MISP objects as you’ll need to do this for your bot.

Clone the MISP object source code.

git clone https://github.com/MISP/misp-objects.git

Make a directory in misp-objects/objects in which to place the new object definition. The directory name should be the same as the object name. We’ll create an android-app object so our path should look like misp-objects/objects/android-app/definition.json.

Inside the object definition we list the attributes and any special qualities each should have.

{
"name": "android-app", 
"meta-category": "misc", 
"description": "An object that describes an Android mobile application.",
"version": 1,
"uuid": "c3a9b689-3f0e-44b7-84a5-1e5e71da5fa1",
"attributes" : 
  {
    "company": {
            "description": "The company who produced the app.",
            "misp-attribute": "text",
            "ui-priority": 1, 
            "multiple": true
    },
    "name": {
            "description": "The name of the mobile application.",
            "misp-attribute": "text",
            "ui-priority": 1
    }
  },
"required": ["name"]
}

Once you’re done creating a new object run the validate_all.sh and jq_all_the_things.sh scripts in the misp objects source repo to verify your object is valid and properly formatted.

MISP Object Generator

Let’s continue with the android-app example to see how we can create this object using the MISP’s Python API library PyMISP. We’ll do this so that we can use a Python object rather than appending JSON into a event directly.

PyMISP stores it’s object generators in the /pymisp/tools folder which is where we’ll add our new object.

Clone the PyMISP source code and create new Python file called androidappobject.py in the /pymisp/tools directory.

Add the following boilerplate to get the object generator started. Comments in the code will explain what we need to change.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

# Import the AbstractMISPObjectGenerator from PyMISP.
from .abstractgenerator import AbstractMISPObjectGenerator
import logging

# Setup a logger. Not strictly required.
logger = logging.getLogger("pymisp")

# The class name can be whatever but you should use the name of the object as it appears in misp-object GitHub.
class FooObject(AbstractMISPObjectGenerator):

    def __init__(self, parameters: dict, strict: bool=True, standalone: bool=True, **kwargs):
        # Make sure to change foo to your object name here.
        super(FooObject, self).__init__("foo", strict=strict, standalone=standalone, **kwargs)
        # Parameters is a dictionary used to populate the class attributes.
        self._parameters = parameters
        # Generate the class attrbiutes when the object is instantiated. 
        self.generate_attributes()

    def generate_attributes(self):
        """
        generate_attributes() does the work of reading self._parameters values into class attributes.

        The idea is to pass FooObject() a dict and turn each key in that dict into a class attribute
        so each class attribute must have the same name of it's misp-object JSON counter-part.
        """
        self._parameters["bar"] = self._parameters.pop("bar", None)
        self._parameters["baz"] = self._parameters.pop("baz", None)

Let’s try that again with the previous android-app example.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

from .abstractgenerator import AbstractMISPObjectGenerator
import logging

logger = logging.getLogger("pymisp")

# Fancy new object name.
class AndroidAppObject(AbstractMISPObjectGenerator):

    def __init__(self, parameters: dict, strict: bool=True, standalone: bool=True, **kwargs):
        # Here we copy pasta'd the object name again.
        super(AndroidAppObject, self).__init__("android-app", strict=strict, standalone=standalone, **kwargs)
        self._parameters = parameters
        self.generate_attributes()

    def generate_attributes(self):
        """
        These class attributes are what we expect to see given the definition.json file.
        """
        self._parameters["name"] = self._parameters.pop("name", None)
        self._parameters["company"] = self._parameters.pop("company", None)

That’s it. You’re ready to use the new object in PyMISP. Sharing is caring so submit a PR for your new object.

Use an Object Generator

Now that we can create android-app Python objects with PyMISP we’ll walk through adding the object to an event.

Set up the PyMISP client and create a new event.

from pymisp import ExpandedPyMISP, MISPEvent
from pymisp.tool.androidappobject import AndroidAppObject
import json
import os

# Initialize MISP API client.
misp = ExpandedPyMISP(os.environ["MISP_URL"], os.environ["MISP_SECRET"], ssl=True)

# Create a new event to store our android-app object.
event = MISPEvent()

# Name the event.
# PyMISP pythonify's MISP objects so we can use class methods to modify the object's attributes.
# This is much better than working with JSON or dict's directly which isn't super fun.
event.info = "Test Event: Hello from PyMISP"

PyMISP’s AbstractMISPObjectGenerator verifies the object against it’s definiton.json. New objects may not yet be included in the latest release. We’re using a custom object we’ve just made so we need to explicitly handle the object definition.

Create a new object.

# Path to our android-app definition.
with open("misp-objects/android-app/definition.json") as f:
    definition = json.load(f)
    f.close()

# This is the data we pass to the object. Notice the keys are attribute names in the definition.json
android_app_data = {"name": "Hungry Dragon", "company": "Ubisoft Entertainment"}

# Instantiate the misp object using the object generator we made earlier.
android_app = AndroidAppObject(parameters=android_app_data,
                               # This tells AbstractMISPObjectGenerator where to look for the definition.
                               # It assumes you've created a local folder containing <object name>/definition.json 
                               misp_objects_path_custom="misp-objects",
                               # This is the template version that should be used. We're using a custom object so the 
                               # correct answer is whatever version used in the object we just made.
                               template_version=str(definition["version"]))

Add it to the event and then add the event to MISP.

# We call the event's Object.append method to append the AndroidAppObject we just made.
event.Object.append(android_app)
# Add the new event to MISP.
result = misp.add_event(event)

Slack Bot

Now we need to build the Slack bot. The bot will:

1. do the thing
2. build the object
3. add the object to a MISP event

Our users call the Slack bot like this: /misp_twitter <misp event id> <Twitter status ID or Twitter URL>

We’re going to use Docker for portability and easy deployment, and we’ll also use a Redis queue so our bot doesn’t lock when handling a request.

The bot source code is available here. Clone the repo, make a copy of the misp-twitter directory and re-name it to whatever you’re building.

The directory looks like this. Most files won’t need to be changed.

You will need to change the misp-object directory and it’s definition.json to relfect your fancy new MISP object. Also the misp_objects.py to include your MISP object constructor. main.py needs to handle whatever slash command args are sent from Slack. Finally utils.py contains all the logic for getting whatever data we need, constructing the MISP object and updating the MISP event.

.
├── app
│   ├── config.py
│   ├── __init__.py
│   ├── main.py
│   ├── misp-objects
│   │   └── microblog
│   │       └── definition.json
│   ├── misp_objects.py
│   ├── prestart.sh
│   ├── utils.py
│   ├── worker.py
│   └── wsgi.py
├── docker-compose.yml
├── Dockerfile
└── requirements.txt

misp_objects.py

This one’s easy. Just paste in your MicroblogObject constructor.

misp-objects/

This one’s also easy. Paste in your definition.json file in a directory of the object name.

main.py

In main.py change the app route (line 30) to the endpoint you’ll configure in the slack app. In this example Slack is sending POST requests to example.com/misp_twitter. Change line 31 function name too.

29
30
31
32
33

@slack_sig_auth
@app.route('/misp_twitter', methods=['POST'])
def microblog():
    text=request.form['text']
    response_url=request.form['response_url']

The valid_input (line 35) func validates the user input to the Slack command. This function is located in utils.py and must be modified for your needs. If the check passes we then construct a dict containing the misp_event_id we want to modify, the twitter_post_id we want to read, and a response_url so we can callback to the Slack user.

We push the run() function and args to the Redis queue on line 43.

35
36
37
38
39
40
41
42
43

    if valid_input(text):
        misp_event_id = text.split(" ")[0]
        twitter_post_id = text.split(" ")[1]
        data = {
            'misp_event_id' : misp_event_id,
            'twitter_post_id' : twitter_post_id,
            'response_url': response_url
        }
        q.enqueue(run, data)

utils.py

This is where all the action happens. We’re going to write some functions to do the thing and wrap them in run() that get’s pushed to the Redis queue in the previous step.

The file starts by setting up API clients in lines 13-25.

Line 28 valid_input() in the function we saw in main.py. Modify this for your needs. The function must return True if the input is ok.

Next we use Tweepy in extended mode to get a Tweet. In your own app you’ll create some function like to call an API and return a raw response.

52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

def twitter_get_extended_status(status_id):
    """
    Get a Twitter status using Tweepy in extended mode.  
    Extended mode returns the full status (will not truncate).
    :param status_id: URL or status ID 
    :return: Twitter status
    """
    # We want the user to be able to submit a URL or a status ID.
    # Try a regex to extract the status ID.
    p = re.compile('twitter.com/[^/]+/status/([\d]+)')
    r = p.findall(status_id)
    if len(r) > 0:
        try:
            response = api.get_status(r[0], tweet_mode="extended")
        except tweepy.error.TweepError as e:
            return str(e[0])
    else:
        # If the regex doesn't work the user gave us a status ID and we can pass it to get_status() directly.
        try:
            response = api.get_status(status_id, tweet_mode="extended")
        except tweepy.error.TweepError as e:
            return str(e[0])
    return response

Now we have a Tweet but there’s a lot of stuff in there we don’t care about. We’re going to clean that up and extract the interesting bits while formatting it into a dict our microblog object constructor understands. Line 82 transform_status_response() does this for us.

The last function that needs to be looked at is run() which is calling all the logic we just went through.

246
247
248
249
250
251

# Get the Twitter status.  This is the raw response we'll format for the object constructor.
status = twitter_get_extended_status(twitter_post_id)

# Extract interesting values from the Twitter status.
# microblog_data can now be passed to the object constructor.
microblog_data = transform_status_response(status._json)

Next we build the microblog object using the definition.json. If the MISP object is already included in PyMISP there’s no need to include the definition in your container and you can omit misp_objects_path_custom and template_version args.

253
254
255
256
257
258
259
260
261

# Load the microblog definition from the local definition.json file.
with open("misp-objects/microblog/definition.json") as f:
    microblog_definition = json.load(f)
    f.close()

# Create the MISP mircroblog object.
microblog = MicroblogObject(parameters=microblog_data,
                            misp_objects_path_custom="misp-objects",
                            template_version=str(microblog_definition["version"]))

We want to modify the MISP event our user supplied as an input arg. MISP users can only modify their own org’s events. Our Slack bot belongs to some org but it might be expected to interact with events owned by other orgs as well so we’ll need to account for that.

This code shouldn’t change between bots. If you’ve made it this far and you can build valid MISP objects you’re basically finished.

265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309

# Get the MISP event. Using pythonify=True means we're transforming the returned results into a PyMISP object.
# We use extended=True because we also want to inspect available event extensions.
working_event = misp.get_event(misp_event_id, extended=True, pythonify=True)

# Get the Slack bot's MISP org ID.
user_profile = misp.get_user("me")
bot_org_id = user_profile["User"]["org_id"]

# If the bot org is the same and the MISP Event org we can update it directly.
if str(bot_org_id) == str(working_event["org_id"]):
    # Add the object to the event.
    working_event.Object.append(microblog)
    result = misp.update_event(working_event)
    # Print?? we're logging to Docker stdout
    print(result)
else:
    # If the bot org is not the same as the event we're trying to modify then we need to use an extension.
    # Let's check if the bot already created an extension event for this event.
    new_event = True
    if "extensionEvents" in working_event:
        for k, event_extension in working_event["extensionEvents"].items():
            # Checking if the org IDs match.
            if event_extension["Orgc"]["id"] == bot_org_id:
                # Check if the extension has the event name we expect this bot to create.
                if event_extension["info"] == "Covid Slack: Disinfo Bot":
                    # Get the extension event and add the microblog object we just created.
                    extension_event = misp.get_event(event_extension["id"], pythonify=True)
                    extension_event.Object.append(microblog)
                    result = misp.update_event(extension_event)
                    # send the result to docker stdout 
                    print(result)
                    # Set new event false so we don't create a new event extension in the next step.
                    new_event = False
    # If the event isn't owned by the bot org , and no event extension exists, we need to create a new event.
    # This event will be added to the parent as an event extension.
    if new_event:
        # Create a new empty event.
        extended_event = MISPEvent()
        # Set self.info to row Title
        extended_event.info = "Covid Slack: Disinfo Bot"
        # extends_uuid tells us the event we are extending with this new event.
        extended_event.extends_uuid = working_event["id"]
        extended_event.Object.append(microblog)
        result = misp.add_event(extended_event, pythonify=True)
        print(result)

The last thing we need to do in utils.py is construct the Slack response we send back to the user.

35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

# Build slack response.
# 'response_type': 'in_channel' means the output will be visible to the whole channel.
response = {'blocks': [], 'response_type': 'in_channel'}
response['blocks'].append({
    'type': 'section',
    'text': {
        'type': 'mrkdwn', # We're returning a message as markdown
        'text': 'Twitter Status: {}'.format(twitter_post_id)  # This is the post header
    }
})

# Create an empty message and add whatever fields need to be here.
twitter_message = ""
if microblog_data.get('username'):
    twitter_message += 'Username: {}\n'.format(microblog_data['username'])

-- snip --

# Add twitter_message as a response block to response.
response['blocks'].append({
    'type': 'section',
    'text': {
        'type': 'mrkdwn',
        'text': twitter_message
    }
})

# Post the message to the callback url.
requests.post(response_url, json=response)

Docker

The docker-compose builds two containers. A gunicorn webserver and a Redis instance to manage the task queue. Both gunicorn and a Redis worker are started via the prestart.sh entry point to the web server.

You’ll need to set the docker-compose.yml file to use whatever environment variables store your API tokens.

You will also need to modify the host web server port from 5001 to whatever if you’re running multiple services.

End

All done. Thanks for reading.

Ping me if you have questions and hope to see cool new Slack bots in the MISP community.