Create an import script for MISP in Python, step-by-step tutorial

Script description

Example add_github_user.py

Here the goal is to push to MISP information gathered on Github. The script add_github_user.py will be used as an example.

An import script, it’s different from a MISP module.

An import script is run from a terminal to push data into a MISP, but a MISP module runs into a MISP instance. For more information about MISP modules, see here

This script will add data from GitHub as object to an existing MISP event.

Tips: in early phase or for testing purpose, it’s better to start with an import script to test the code and results. If it’s conclusive you can code a MISP module.

PyMISP install

PyMISP is a Python library to access MISP with REST API. Install the PyMISP library on your machine.

To do so, clone the repository and install the library:

git clone https://github.com/MISP/PyMISP
cd PyMISP
pip3 install .

Fork

If you wish to contribute and push your code to the MISP community, you’ll need to fork the PyMISP library.

Meaningful naming

Where to put the files and what are their names?

The script will be in PyMISP/examples and has the name of the imported service. A short key word is a good practice to describe the script action:

PyMISP/examples/add_github_user.py

tips: before naming your script, consult the list of available scripts into the PyMISP/examples folder, in order to describe accurately what your import module does.

Keys management

For all scripts , you need to have a single key file keys.py. The file is not to be committed and contains the credentials for connecting to the MISP instance: API key, URL, SSH validations.

• You create a file keys.py into the dir PyMISP/examples.

References to this file will be later set in the script:

from keys import misp_url, misp_key, misp_verifycert

Required libraries

• PyMISP
• request
• sys
• argparse

Usage block

We will go through the Python code and explain step-by-step each part.

It will be used by pyDoc, starts with """ and ends with “”". You will here describe the usage for the script.

"""
usage: add_github_user.py [-h] -e EVENT [-f] -u USERNAME
Fetch GitHub user details and add it in object in MISP
optional arguments:
-h, --help            show this help message and exit
-e EVENT, --event EVENT
		      Event ID to update
-f, --force-template-update
-u USERNAME, --username USERNAME
		      GitHub username to add
"""

Argument parser

The script needs to grab the event-id (-e), a force to update the template from the repository (-f), and here a user account (-u).

parser = argparse.ArgumentParser(description='Fetch GitHub user details and add it in object in MISP')
parser.add_argument("-e", "--event", required=True, help="Event ID to update")
parser.add_argument("-f", "--force-template-update", required=False, action="store_true")
parser.add_argument("-u", "--username", required=True, help="GitHub username to add")
args = parser.parse_args()

Request

NOTE: If the user doesn’t exist, the script is interrupted.

r = requests.get("https://api.github.com/users/{}".format(args.username))
if r.status_code != 200:
		sys.exit("HTTP return is {} and not 200 as expected".format(r.status_code))
if args.force_template_update:
		print("Updating MISP Object templates...")
		update_objects()
pymisp = PyMISP(misp_url, misp_key, misp_verifycert)

The PyMISP connection is instantiated with following parameters coming from the file keys.py : misp_url, misp_keys and misp_verifycert.

Object creation

We create a MISP object, the name is the object template name:

misp_object = MISPObject(name="github-user")

tips: you can find the object template name is the MISP object template repository.

contributing: if an object template is missing, you can propose your own.

User data

User data from the GitHub API are transformed into a JSON in order to be parsed by the following commands:

github_user = r.json()

Following HTTP GET requests to obtain the following and followers sections of the GitHub user.

rfollowers = requests.get(github_user['followers_url'])
followers = rfollowers.json()
rfollowing = requests.get("https://api.github.com/users/{}/following".format(args.username))
followings = rfollowing.json()

Same things for the SSH public keys attached to the GitHub user:

rkeys = requests.get("https://api.github.com/users/{}/keys".format(args.username))
keys = rkeys.json()

Then, filling of the object from the github_user json object:

misp_object.add_attributes("follower", *[follower['login'] for follower in followers])
misp_object.add_attributes("following", *[following['login'] for following in followings])
misp_object.add_attributes("ssh-public-key", *[sshkey['key'] for sshkey in keys])
misp_object.add_attribute('bio', github_user['bio'])
misp_object.add_attribute('link', github_user['html_url'])
misp_object.add_attribute('user-fullname', github_user['name'])
misp_object.add_attribute('username', github_user['login'])
misp_object.add_attribute('twitter_username', github_user['twitter_username'])
misp_object.add_attribute('location', github_user['location'])
misp_object.add_attribute('company', github_user['company'])
misp_object.add_attribute('public_gists', github_user['public_gists'])
misp_object.add_attribute('public_repos', github_user['public_repos'])
misp_object.add_attribute('blog', github_user['blog'])
misp_object.add_attribute('node_id', github_user['node_id'])

Finally, you add the new object created, then pushed to the specified event id, on your MISP instance:

retcode = pymisp.add_object(args.event, misp_object)

Now it’s your turn, to do a MISP import script.