PolySwarmPolySwarmPolySwarmPolySwarm
Help

Developing a Production Microengine

As a microengine developer, it's important to monitor your microengine's performance and hone your NCT staking strategy to reflect your level of confidence concerning artifacts seen in the PolySwarm marketplace. Filtering which artifacts your microengine asserts on and maintaining a strong correlation between amount staked and your confidence in your microengine's assertion is essential for maintaining an effective (and profitable) microengine.

All microengine developers should routinely:

  1. Monitor your microengine's marketplace performance.
  2. Engineer a first-pass "triage" filter to quickly identify which artifacts are of interest.
  3. Maintain a tight correlation between your microengine's assertion confidence and the amount of NCT staked.
  4. Ensure their microengine is capable of scaling in response to demand.

Monitoring Your Microengine's Marketplace Performance

Any PolySwarm user can track any microengine's profit / loss from PolySwarm Web's Microengines page. As an microengine developer, you'll want to:

  1. Create an account on PolySwarm Web
  2. Track your microengines' performance in comparison to other microengines' performance and
  3. Claim ownership in your microengine(s) so you can more easily track your microengines' performance

Claim Ownership Over Your Microengine(s)

When you claim (and prove) ownership of your Microengines, you're able to:

  1. (Optionally) name your microengine
  2. (Coming Soon) assign ownership to a Team
  3. View your microengines' performance in a single view without searching for each microengine in the unfiltered Microengines listing

We encourage every microengine developer to claim all of their Microengines.

We're continually rolling out new features that extend the management capabilities of owned Microengines.

Taking ownership of your microengine is a necessary first step for admission to PolySwarm's various Private Communities, unlocking private and often higher-value bounties.

Track Performance

With your Microengines claimed, quickly view profit / loss information across your microengine footprint using My Microengines. When deploying new detections to your microengines, use My Microengines to get a sense for their economic impact. Review these charts often in order to quickly identify when a problem with an microengine arises (e.g. a precipitous drop in profitability).

Close the Loop

Visibility into underperforming microengines allows the microengine developer to take an underperforming microengine offline as a means to stop loss. Still, without additional context, determining how to improve the microengine before bringing it back online can be difficult. For this reason, we strongly recommend that all microengines archive the artifacts they have evaluated along with their assertion and NCT stake for any given artifact.

We're designing bookkeeping functions that make this a proactive process, helping the microengine developer more quickly close the loop on microengine improvement. As they become available, these features will be accessible on the My Microengines page.


Triaging Artifacts

We recommend microengine developers architect their microengines in a two-stage process:

  1. a very fast, lightweight triage process that determines whether the artifact is worthy of a full investigation
  2. a full interrogation of the artifact, determining malintent and responding to bounties within the assertion time window

By implementing a triage pass, microengine developers can save time and money, reducing execution burden and quickly ignoring uninteresting artifacts. Based on conversations with microengine providers, a popular triage tactic is artifact downselection based on file type.

Below is a simple example of a triage pass in a Microengine's scan() function:

import magic
...
class Scanner(AbstractScanner):

  ...

  async def scan(self, guid, artifact_type, content, metadata, chain):

    # Reject artifacts that aren't files
    if not artifact_type == ArtifactType.FILE:
      return ScanResult()

    # Reject files that libmagic does not identify as an ELF or PE by returning an empty ScanResult object
    if not ((magic.from_buffer(content)[0:3] == "ELF") or (magic.from_buffer(content)[0:2] == "PE")):
      return ScanResult()
...

Developing an Effective Staking Strategy

Example Lifecycle

Let's run through a simplified example of a bounty lifecycle, noting the impact of staking strategy design.

Bounty Creation

Suppose the fictitious ACME Enterprises discovers something suspicious on their network and wants to enrich their telemetry with intelligence produced by the PolySwarm marketplace. ACME submits an artifact to the PolySwarm marketplace via PolySwarm Web, PolySwarm API or a third party Ambassador.

An Ambassador creates a bounty for ACME's submission. This bounty contains: (1) the artifact, and (2) a configurable amount of NCT into the initial reward bucket for the bounty. For the purposes of illustration, let's assume 5 NCT was placed into the reward bucket.

Active PolySwarm microengines are notified of this new bounty and the amount of NCT placed on the artifact.

Let's assume 8 microengines* find the initial NCT reward placed by the Ambassador to be sufficient for triaging of the artifact to determine whether the artifact falls within the the microengine's area of expertise.

*The number of active PolySwarm microengines is far beyond 8 and is growing by the day, but we'll keep this example simple for illustrative purposes.

Microengines Conduct Triage Pass

Microengines conduct their first-pass triage and determine:

  • microengines A, B, C, D, E: the artifact is within their area of expertise
  • microengines F, G, H: the artifact is outside their area of expertise

Microengines F, G and H ignore the bounty, choosing not to respond, whereas microengines A-E take a closer look.

Microengines Conduct Full Analysis

During their analysis, each microengine identifies key characteristics (high confidence indicators) and/or general patterns (lower-confidence indicators) that helped them arrive at their conclusion. These microengines telegraph their confidence in terms of the amount of NCT they stake, arriving at:

  • microengine A: 1 NCT / malicious
  • microengine B: 1 NCT / benign
  • microengine C: 2 NCT / malicious
  • microengine D: 1 NCT / malicious
  • microengine E: 2 NCT / benign

Roughly speaking, microengines C and E are twice as confident in their assertion than the microengines that agree with them.

These assertions and their NCT stakes are sent to the Ambassador immediately after the assertion window closes. The Ambassador analyzes these assertions and optionally combines them into a single piece of finished intelligence for delivery to ACME.

Total NCT is Computed by PolySwarm's BountyRegistry Contract

The initial reward plus the amounts staked are escrowed into PolySwarm's BountyRegistry contract. All the funds are summed:

  • initial reward: 5 NCT +
  • microengine A: 1 NCT +
  • microengine B: 1 NCT +
  • microengine C: 2 NCT +
  • microengine D: 1 NCT +
  • microengine E: 2 NCT =
  • total reward: 12 NCT

Arbiters Determine Ground Truth

Later, Arbiters weigh in on the ground truth concerning the artifact at hand. Arbiters determine that the artifact was, in fact, malicious.

The bounty is then open for claims against the reward by the microengines that correctly asserted. microengines' total winnings are proportional to their stake:

  • microengine A: 3 NCT (2 NCT profit)
  • microengine B: 0 NCT (incorrect assertion)
  • microengine C: 6 NCT (4 NCT profit)
  • microengine D: 3 NCT (2 NCT profit)
  • microengine E: 0 NCT (incorrect assertion)

microengine C is the largest winner. By staking double the amount staked by A and D, C is rewarded with double the proportion of the reward.

The example presented here is a simplification of what would actually happen in the PolySwarm market.

In the actual marketplace, far more microengines would respond, stake amounts need not be integers and fees would be assessed by the marketplace in order to compensate Arbiters.

Maintain Tight Correlation Between Confidence and NCT Stake Amount

When microengines telegraph their confidence in assertions via NCT stake amounts, everyone benefits. On one side, Ambassadors are provided with another dimension of signal concerning the malintent of artifacts the microengines' NCT stake amount. On the other side of the market, microengines that modulate NCT stakes based on confidence have the potential to increase profit.

The best microengines will have a good sense of their confidence in each scan and will deliver a "confidence interval" between 0.0 and 1.0 while returning scan results via the ScanResult object. These confidences are used in AbstractMicroengine's bid() method according to a BidStrategy.

polyswarm-client provides a default bid strategy via the class BidStrategyBase. Variants of this default strategy (with different weights applied) can be found in polyswarm-client.

You may use the default bid strategy, some variant thereof, or develop a fully custom bid strategy by subclassing BidStrategyBase. participant-template will produce a BidStrategy class for Microengine participants. Refer to the comments surrounding this subclass for more information.

During testing, it may be convenient to quickly swap bid strategies. You can choose which strategy to use when you launch your microengine by providing the --bid-strategy command line option or setting the BID_STRATEGY variable in your environment.

Let's take a look at the default bid strategy in BidStrategyBase's bid() method:

async def bid(self, guid, mask, verdicts, confidences, metadatas, chain):
  """Override this to implement custom bid calculation logic
  Args:
      guid (str): GUID of the bounty under analysis, use to correlate with artifacts in the same bounty
      masks (list[bool]): mask for the from scanning the bounty files
      verdicts (list[bool]): scan verdicts from scanning the bounty files
      confidences (list[float]): Measure of confidence of verdict per artifact ranging from 0.0 to 1.0
      metadatas (list[str]): metadata blurbs from scanning the bounty files
      chain (str): Chain we are operating on
  Returns:
      int: Amount of NCT to bid in base NCT units (10 ^ -18)
  """
  min_allowed_bid = await self.client.bounties.parameters[chain].get('assertion_bid_minimum')
  min_bid = max(self.min_bid, min_allowed_bid)
  max_bid = max(self.max_bid, min_allowed_bid)

  asserted_confidences = [c for b, c in zip(mask, confidences) if b]
  avg_confidence = sum(asserted_confidences) / len(asserted_confidences)
  bid = int(min_bid + ((max_bid - min_bid) * avg_confidence))

  # Clamp bid between min_bid and max_bid
  return max(min_bid, min(bid, max_bid))

Currently, only one NCT stake may be placed per bounty. This presents a problem for multi-artifact bounties: How can a single NCT stake accurately convey potentially variable confidence across multiple bounties?

We're working to remove this limitation. Future releases will support N stakes for N artifacts in a single bounty.

Currently, staking strategies take an average confidence over all artifacts in a given bounty to arrive at a single NCT stake amount for that bounty.

Determining Confidence

The specifics of determining confidence cannot easily be generalized; each microengine will have an optimal strategy. Generally, we've found that microengine developers are choosing one of several strategies (in order of increasing efficacy):

  1. No confidence can be derived - the microengine is equally confident in all responses. This is the least optimal strategy and often manifests in a static amount of NCT staked on every artifact. We're working with these microengine developers on Discord to develop a better staking strategy and would be happy to help you as well!
  2. Confidence based on file type. Some microengines use file type information twice: once to exclude artifacts during their triage process and again to assign weights to files that make it through their triage process. This can be as simple as assigning a static weight per supported file type that influences the confidence score passed from the Scanner class. This type of strategy is better, but still is not ideal.
  3. Confidence based on specific indicators. This should be the goal of all well-performing microengines. There are several microengines on the PolySwarm marketplace that do this today, e.g. by dissecting Microsoft Word documents and identifying auto-executing Macro scripts that are known to be bad. This sort of artifact interrogation is optimal - it provides very high confidence signal that will allow these microengine developers to develop an optimal staking strategy.
import magic
...
class Scanner(AbstractScanner):

  ...

  async def scan(self, guid, artifact_type, content, metadata, chain):

    confidence_delta = 0

    # Increase confidence score for ELF and PE files
    if not ((magic.from_buffer(content)[0:3] == "ELF") or (magic.from_buffer(content)[0:2] == "PE")):
      confidence_delta += 0.2

    ...

    # Conduct the full analysis, arriving at a base confidence score
    confidence_base = do_analysis()

    ...

    # Take file-based confidence into account when returning result
    return ScanResult(bit=True, verdict=True, confidence=confidence_base+confidence_delta)
...

Operating at Scale

As more enterprises rely on PolySwarm for scanning suspect artifacts, microengines need to scale in order to meet demand.

Microengines built using participant-template use a producer / consumer model* for horizontal scaling:

  1. 1 frontend (producer): responsible for communicating with the PolySwarm marketplace: ingesting bounties, triaging artifacts, producing pub/sub scan events for backends, implementing a staking strategy and posting assertions. The frontend translates marketplace bounties into events on a pub/sub queue for backends to consume and distills responses from backends into marketplace actions.
  2. N backends (consumers): the actual scanners that process artifacts and produce assertions (malicious / benign) coupled with confidence ratings. These backends are tasked by the frontend.

Microengines created with participant-template prior to June 18th 2019 will need to be upgraded to the producer / consumer model.

The producer / consumer model makes it simple to horizontally scale your microengine. As demand increases, launch additional consumer instances. As demand decreases, it's safe to retire some consumer instances. In other words, microengine's resource footprint should scale elastically in response to demand.

Relative to a traditional monolithic model, producer / consumer provides additional benefits:

  1. The producer houses bid / staking logic, disjoint from consumer-held scanning logic. This separation provides a more maintainable and secure architecture: consumers, responsible for complex scanning functions, need not (and should not) have access to the microengine's wallet. All wallet-related functions can be handled by the comparatively simple producer component.
  2. The pub/sub interface permits parallel scans by design without complex (or even explicit) client synchronization.
  3. Makes it easier to build disjoint, multi-backend microengines. It becomes possible to mix and match lighter weight (e.g. static analysis) and heavier weight (e.g. sandbox) backends, with the single producer frontend mediating scan results, allowing the microengine to respond as best it can within the assertion timeframe.
  4. Reduces computational cost via elastic resource consumption.

Next Steps

With a staking strategy in place, it's time to connect your microengine to the PolySwarm marketplace!