Create Your Engine Template

Clone the microengine-webhooks-py repo

The first step is to get a copy of our reference Engine implementation from GitHub. We'll use git to clone the microengine-webhooks-py repo.

(psvenv) $ git clone https://github.com/polyswarm/microengine-webhooks-py.git
(psvenv) $ cd microengine-webhooks-py/

Install dependencies

Next, let's install the Python dependencies.

(psvenv) $ pip install -r requirements.txt

When this is done, you have everything you need to build an Engine.

What is microengine-webhooks-py?

The microengine-webhooks-py repo is a fully functional Engine that you can use as a reference to build your own engine. As-is, this template Engine can only detect EICAR. The purpose of a trivial EICAR Engine is to understand how an Engine works and to test protocols and connectivity.

What is EICAR?

The Antivirus industry uses the EICAR test file to verify an Engines' ability to return a "malicicous" detection result. The file is not malicious, but is flagged as such by most vendors.

The EICAR test file is defined as a file that begins with the following string: X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H* followed by a variable amount of whitespace.

What's in this repo?

This repo uses a standard Python package layout, but here is a quick overview of the contents:

chart - Kubernetes chart and values
docker - Dockerfiles to build Docker images
nginx - Configs for nginx unit
src/microenginewebhookspy - Python code for your Engine's web service and worker
tests - Unit and Integration tests

The three most important files for Engine developers are scan.py, settings.py, and models.py.

scan.py

scan.py is the primary source file defining Engine behavior. The first of the two functions that already exist in scan.py is scan(). scan() takes a Bounty and returns a ScanResult. The primary function of scan() is to execute a scan on the Bounty's artifact and determine whether it is malicious. The result of the scan is stored in the ScanResult's fields: verdict, confidence, and metadata.

Verdict can be one of four states: malicious, suspicious, benign, or unknown.
Confidence is a measure of how sure the Engine feels about the Verdict. A decimal between 0.0 and 1.0.
Metadata is a ScanMetadata object containing a collection of details about the artifact and/or scanning application.

Developers should override scan() and replace the function contents to perform their own scan.

The second function is compute_bid(). compute_bid() takes a Bounty & ScanResult pair, then returns an int as the amount of NCT to bid. compute_bid() is covered in more detail in the section on bidding strategy.

settings.py

settings.py has environment variables for tuning Engine behavior. By default, there are variables for the Webhook secret, Celery broker, and log level. Developers should add more environment variables as needed to configure the behavior of their Engine.

models.py

models.py contains all the definitions for objects like Bounty, ScanResult, and the Verdict enum.

Remaining Python Files

The remainder of the source files can be quickly summarized, and they won't be discussed in any detail in these docs.

middleware.py - WSGI middleware that checks the signature with the shared secret
tasks.py - Celery task definition that handles the bounty and coordinates scanning, bidding, and responding to the PolySwarm Marketplace
utils.py - Helper function definitions
views.py - Flask Blueprint and routes to receive and parse Webhooks
wsgi.py - Flask application definition

Engine Architecture Overview

This diagram shows a high-level overview of the architecture implemented by this Engine template. Click on the image to make it larger.

This template architecture provides a foundation for you to build upon. The docker directory of the microengine-webhooks-py repo has 2 Dockerfiles. One for the Web Server and a second for the Worker. This allows you to quickly build and test your first engine. As you develop your own engine, maybe you want to do things differently. If so, you can customize or replace any technologies used in the Engine box.

Next Steps

Now we have a working Engine template, let's modify it to build our own Engine.