PolySwarmPolySwarmPolySwarmPolySwarm
Go to PolySwarm
Home

Developing a Production Engine

As an engine developer, it's important to monitor your engine's performance and hone your NCT staking strategy to reflect your level of confidence concerning artifacts seen in the PolySwarm marketplace. Filtering which artifacts your engine asserts on and maintaining a strong correlation between amount staked and your confidence in your engine's assertion is essential for maintaining an effective (and profitable) engine.

All engine developers should routinely:

  1. Monitor your engine's marketplace performance.
  2. Engineer a first-pass "triage" filter to quickly identify which artifacts are of interest.
  3. Maintain a tight correlation between your engine's assertion confidence and the amount of NCT staked.
  4. Ensure their engine is capable of scaling in response to demand.

Monitoring Your Engine's Marketplace Performance

Any PolySwarm user can track any engine's profit / loss from PolySwarm Web's Engines page. As an engine developer, you'll want to:

  1. Create an account on PolySwarm Web
  2. Track your engines' performance in comparison to other engines' performance and
  3. Claim ownership in your engine(s) so you can more easily track your engines' performance

Claim Ownership Over Your Engine(s)

When you claim (and prove) ownership of your Engines, you're able to:

  1. (Optionally) name your engine
  2. (Coming Soon) assign ownership to a Team
  3. View your engines' performance in a single view without searching for each engine in the unfiltered Engines listing

We encourage every engine developer to claim all of their Engines.

We're continually rolling out new features that extend the management capabilities of owned Engines.

Taking ownership of your engine is a necessary first step for admission to PolySwarm's various Private Communities, unlocking private and often higher-value bounties.

To get started, navigate to My Engines and click the "Claim Engine" button (or the plus button if you've previously claimed an engine). Follow the on-screen prompts until it asks you for a Signture proving your ownership.

Sign Claim Message

Step 1: Start an interactive node Docker container
$ docker run -it node bash
Step 2: Install prerequisites in the running Docker container
# npm install --save web3
Step 3: Use your favorite text editor to author sign.js in the running Docker container

apt-get update && apt-get install -y vim will install Vim in your Docker container.

const web3 = new (require('web3'))(),fs = require('fs');

const keystore = "/keyfile";
const password = "<keystore password>";
const message = "<message to sign>";

const keyfile = JSON.parse(fs.readFileSync(keystore));
const account = web3.eth.accounts.decrypt(keyfile, password);

const { signature } = web3.eth.accounts.sign(
  message,
  account.privateKey
);

console.log(signature);

Be sure to customize:

  • password: Your Ethereum keyfile's encryption password.
  • message: The message you'll be signing. This is presented to you in the signature verification prompt.

With these customizations complete, save this file as sign.js.

Step 4: In another shell outside of the Docker container, copy your engine's keyfile into the running container

Get the name of your running Docker container with docker ps.

$ docker cp <the location of your keyfile outside of the Docker container> <the name of your running Docker container>:/keyfile
Step 5: Run sign.js inside the running Docker container
node sign.js
Step 6: Copy the output of sign.js and paste into the Claim Engine prompt.

Your engine should now be claimed!

Track Performance

With your Engines claimed, quickly view profit / loss information across your engine footprint using My Engines. When deploying new detections to your engines, use My Engines to get a sense for their economic impact. Review these charts often in order to quickly identify when a problem with an engine arises (e.g. a precipitous drop in profitability).

Close the Loop

Visibility into underperforming engines allows the engine developer to take an underperforming engine offline as a means to stop loss. Still, without additional context, determining how to improve the engine before bringing it back online can be difficult. For this reason, we strongly recommend that all engines archive the artifacts they have evaluated along with their assertion and NCT stake for any given artifact.

We're designing bookkeeping functions that make this a proactive process, helping the engine developer more quickly close the loop on engine improvement. As they become available, these features will be accessible on the My Engines page.


Triaging Artifacts

We recommend engine developers architect their engines in a two-stage process:

  1. a very fast, lightweight triage process that determines whether the artifact is worthy of a full investigation
  2. a full interrogation of the artifact, determining malintent and responding to bounties within the assertion time window

By implementing a triage pass, engine developers can save time and money, reducing execution burden and quickly ignoring uninteresting artifacts. Based on conversations with engine providers, a popular triage tactic is artifact downselection based on file type.

Below is a simple example of a triage pass in an Engine's scan() function:

import magic
...
class Scanner(AbstractScanner):

  ...

  async def scan(self, guid, artifact_type, content, metadata, chain):

    # Reject artifacts that aren't files
    if not artifact_type == ArtifactType.FILE:
      return ScanResult()

    # Reject files that libmagic does not identify as an ELF or PE by returning an empty ScanResult object
    if not ((magic.from_buffer(content)[0:3] == "ELF") or (magic.from_buffer(content)[0:2] == "PE")):
      return ScanResult()
...

Developing an Effective Staking Strategy

Example Lifecycle

Let's run through a simplified example of a bounty lifecycle, noting the impact of staking strategy design.

Bounty Creation

Suppose the fictitious ACME Enterprises discovers something suspicious on their network and wants to enrich their telemetry with intelligence produced by the PolySwarm marketplace. ACME submits an artifact to the PolySwarm marketplace via PolySwarm Web, PolySwarm API or a third party Ambassador.

An Ambassador creates a bounty for ACME's submission. This bounty contains: (1) the artifact, and (2) a configurable amount of NCT into the initial reward bucket for the bounty. For the purposes of illustration, let's assume 5 NCT was placed into the reward bucket.

Active PolySwarm engines are notified of this new bounty and the amount of NCT placed on the artifact.

Let's assume 8 engines* find the initial NCT reward placed by the Ambassador to be sufficient for triaging of the artifact to determine whether the artifact falls within the the engine's area of expertise.

*The number of active PolySwarm engines is far beyond 8 and is growing by the day, but we'll keep this example simple for illustrative purposes.

Engines Conduct Triage Pass

Engines conduct their first-pass triage and determine:

  • engines A, B, C, D, E: the artifact is within their area of expertise
  • engines F, G, H: the artifact is outside their area of expertise

Engines F, G and H ignore the bounty, choosing not to respond, whereas engines A-E take a closer look.

Engines Conduct Full Analysis

During their analysis, each engine identifies key characteristics (high confidence indicators) and/or general patterns (lower-confidence indicators) that helped them arrive at their conclusion. These engines telegraph their confidence in terms of the amount of NCT they stake, arriving at:

  • engine A: 1 NCT / malicious
  • engine B: 1 NCT / benign
  • engine C: 2 NCT / malicious
  • engine D: 1 NCT / malicious
  • engine E: 2 NCT / benign

Roughly speaking, engines C and E are twice as confident in their assertion than the engines that agree with them.

These assertions and their NCT stakes are sent to the Ambassador immediately after the assertion window closes. The Ambassador analyzes these assertions and optionally combines them into a single piece of finished intelligence for delivery to ACME.

Total NCT is Computed by PolySwarm's BountyRegistry Contract

The initial reward plus the amounts staked are escrowed into PolySwarm's BountyRegistry contract. All the funds are summed:

  • initial reward: 5 NCT +
  • engine A: 1 NCT +
  • engine B: 1 NCT +
  • engine C: 2 NCT +
  • engine D: 1 NCT +
  • engine E: 2 NCT =
  • total reward: 12 NCT

Arbiters Determine Ground Truth

Later, Arbiters weigh in on the ground truth concerning the artifact at hand. Arbiters determine that the artifact was, in fact, malicious.

The bounty is then open for claims against the reward by the engines that correctly asserted. engines' total winnings are proportional to their stake:

  • engine A: 3 NCT (2 NCT profit)
  • engine B: 0 NCT (incorrect assertion)
  • engine C: 6 NCT (4 NCT profit)
  • engine D: 3 NCT (2 NCT profit)
  • engine E: 0 NCT (incorrect assertion)

engine C is the largest winner. By staking double the amount staked by A and D, C is rewarded with double the proportion of the reward.

The example presented here is a simplification of what would actually happen in the PolySwarm market.

In the actual marketplace, far more engines would respond, stake amounts need not be integers and fees would be assessed by the marketplace in order to compensate Arbiters.

Maintain Tight Correlation Between Confidence and NCT Stake Amount

When engines telegraph their confidence in assertions via NCT stake amounts, everyone benefits. On one side, Ambassadors are provided with another dimension of signal concerning the malintent of artifacts the engines' NCT stake amount. On the other side of the market, engines that modulate NCT stakes based on confidence have the potential to increase profit.

The best engines will have a good sense of their confidence in each scan and will deliver a "confidence score" between 0.0 and 1.0 while returning scan results via the ScanResult object. These confidences are used in AbstractMicroengine's bid() method according to a BidStrategy.

polyswarm-client provides a default bid strategy via the class BidStrategyBase. Variants of this default strategy (with different weights applied) can be found in polyswarm-client.

You may use the default bid strategy, some variant thereof, or develop a fully custom bid strategy by subclassing BidStrategyBase. participant-template will produce a BidStrategy class for Engine participants. Refer to the comments surrounding this subclass for more information.

During testing, it may be convenient to quickly swap bid strategies. You can choose which strategy to use when you launch your engine by providing the --bid-strategy command line option or setting the BID_STRATEGY variable in your environment.

Let's take a look at the default bid strategy in BidStrategyBase's bid() method:

async def bid(self, guid, mask, verdicts, confidences, metadatas, chain):
  """Override this to implement custom bid calculation logic
  Args:
      guid (str): GUID of the bounty under analysis, use to correlate with artifacts in the same bounty
      masks (list[bool]): mask for the from scanning the bounty files
      verdicts (list[bool]): scan verdicts from scanning the bounty files
      confidences (list[float]): Measure of confidence of verdict per artifact ranging from 0.0 to 1.0
      metadatas (list[str]): metadata blurbs from scanning the bounty files
      chain (str): Chain we are operating on
  Returns:
      int: Amount of NCT to bid in base NCT units (10 ^ -18)
  """
  min_allowed_bid = await self.client.bounties.parameters[chain].get('assertion_bid_minimum')
  min_bid = max(self.min_bid, min_allowed_bid)
  max_bid = max(self.max_bid, min_allowed_bid)

  asserted_confidences = [c for b, c in zip(mask, confidences) if b]
  avg_confidence = sum(asserted_confidences) / len(asserted_confidences)
  bid = int(min_bid + ((max_bid - min_bid) * avg_confidence))

  # Clamp bid between min_bid and max_bid
  return max(min_bid, min(bid, max_bid))

Currently, only one NCT stake may be placed per bounty. This presents a problem for multi-artifact bounties: How can a single NCT stake accurately convey potentially variable confidence across multiple bounties?

We're working to remove this limitation. Future releases will support N stakes for N artifacts in a single bounty.

Currently, staking strategies take an average confidence over all artifacts in a given bounty to arrive at a single NCT stake amount for that bounty.

Determining Confidence

The specifics of determining confidence cannot easily be generalized; each engine will have an optimal strategy. Generally, we've found that engine developers are choosing one of several strategies (in order of increasing efficacy):

  1. No confidence can be derived - the engine is equally confident in all responses. This is the least optimal strategy and often manifests in a static amount of NCT staked on every artifact. We're working with these engine developers on Discord to develop a better staking strategy and would be happy to help you as well!
  2. Confidence based on file type. Some engines use file type information twice: once to exclude artifacts during their triage process and again to assign weights to files that make it through their triage process. This can be as simple as assigning a static weight per supported file type that influences the confidence score passed from the Scanner class. This type of strategy is better, but still is not ideal.
  3. Confidence based on specific indicators. This should be the goal of all well-performing engines. There are several engines on the PolySwarm marketplace that do this today, e.g. by dissecting Microsoft Word documents and identifying auto-executing Macro scripts that are known to be bad. This sort of artifact interrogation is optimal - it provides very high confidence signal that will allow these engine developers to develop an optimal staking strategy.
import magic
...
class Scanner(AbstractScanner):

  ...

  async def scan(self, guid, artifact_type, content, metadata, chain):

    confidence_delta = 0

    # Increase confidence score for ELF and PE files
    if not ((magic.from_buffer(content)[0:3] == "ELF") or (magic.from_buffer(content)[0:2] == "PE")):
      confidence_delta += 0.2

    ...

    # Conduct the full analysis, arriving at a base confidence score
    confidence_base = do_analysis()

    ...

    # Take file-based confidence into account when returning result
    return ScanResult(bit=True, verdict=True, confidence=confidence_base+confidence_delta)
...

Operating at Scale

As more enterprises rely on PolySwarm for scanning suspect artifacts, engines need to scale in order to meet demand.

Engines built using participant-template use a producer / consumer model* for horizontal scaling:

  1. 1 frontend (producer): responsible for communicating with the PolySwarm marketplace: ingesting bounties, triaging artifacts, producing pub/sub scan events for backends, implementing a staking strategy and posting assertions. The frontend translates marketplace bounties into events on a pub/sub queue for backends to consume and distills responses from backends into marketplace actions.
  2. N backends (consumers): the actual scanners that process artifacts and produce assertions (malicious / benign) coupled with confidence ratings. These backends are tasked by the frontend.

Engines created with participant-template prior to June 18th 2019 will need to be upgraded to the producer / consumer model.

The producer / consumer model makes it simple to horizontally scale your engine. As demand increases, launch additional consumer instances. As demand decreases, it's safe to retire some consumer instances. In other words, engine's resource footprint should scale elastically in response to demand.

Relative to a traditional monolithic model, producer / consumer provides additional benefits:

  1. The producer houses bid / staking logic, disjoint from consumer-held scanning logic. This separation provides a more maintainable and secure architecture: consumers, responsible for complex scanning functions, need not (and should not) have access to the engine's wallet. All wallet-related functions can be handled by the comparatively simple producer component.
  2. The pub/sub interface permits parallel scans by design without complex (or even explicit) client synchronization.
  3. Makes it easier to build disjoint, multi-backend engines. It becomes possible to mix and match lighter weight (e.g. static analysis) and heavier weight (e.g. sandbox) backends, with the single producer frontend mediating scan results, allowing the engine to respond as best it can within the assertion timeframe.
  4. Reduces computational cost via elastic resource consumption.

Next Steps

With a staking strategy in place, it's time to connect your engine to the PolySwarm marketplace!

2020 © PolySwarm Pte. Ltd.