PolySwarmPolySwarmPolySwarmPolySwarm
Help

Ambassadors

対象者

Ambassadors are gateways to the PolySwarm marketplace. It is Ambassadors' responsibility to translate queries into marketplace actions and aggregate results on behalf of consumers of PolySwarm threat intelligence (e.g. enterprise customers). If you'd like to act as a conduit to the PolySwarm marketplace on your own behalf or on behalf third party consumers, continue reading.

When a consumer uses the PolySwarm web interface or polyswarm-api they will, by default, use an Ambassador hosted by Swarm Technologies, Inc.

Consumers may choose to speak to other Ambassadors using polyswarm-api's --api-uri argument or POLYSWARM_API_URI environment variable.


マーケットプレイスにおけるアンバサダーの役割

Ambassadors act as intermediaries between consumers of PolySwarm threat intelligence and the PolySwarm marketplace. Broadly, Ambassadors broker artifact uploads, hash searches, hunts and other features as they are developed, creating actionable events in the PolySwarm marketplace on behalf of consumers and then deliver results to consumers.

アーティファクト送信のライフサイクル

The artifact submission lifecycle at a conceptual level:

  1. The consumer submits an artifact, and optionally, a PolySwarm community preference to the Ambassador*
  2. The Ambassador hosts the artifact in a manner that is accessible to microengines in the appropriate community/communities.
  3. The Ambassador creates a bounty for the artifact in each community, specifying: initial bounty amount, a signed transaction that escrows the initial bounty amount from the Ambassador's wallet to the community's BountyRegistry contract**, the bounty's assertion duration, the URI of the artifact, and, optionally, additional metadata (e.g. artifact filetype). これは、polyswarmd API を使用して、各コミュニティーの polyswarmd インスタンスを利用することで行います。
  4. 各コミュニティー内のエンジンは、アサーション期間が終わるまでに、アーティファクトに関するアサーションを返す必要があります。
  5. Engines submit their assertions (and stake amounts) to the community's polyswarmd.
  6. Each community's polyswarmd instance delivers the aggregated results to the Ambassador.
  7. (Optional) the Ambassador distills the various assertions into digestible intelligence for the consumer.
  8. The Ambassador delivers finished intelligence to the consumer.

*アンバサダーまたは利用者がコミュニティーを割り当てます。

**It is the Ambassador's responsibility to ensure that they have adequate funds relayed to each community for the payment of network fees and to cover the initial bounty amount.

ハッシュ検索のライフサイクル

Providing hash search capability to consumers requires:

  1. 送信されたアーティファクトと、各アーティファクトに対するマーケットの応答のアーカイブ
  2. ハッシュ・ミスを処理する戦略

The hash search lifecycle at a conceptual level:

  1. The Ambassador maintains an archive of all incoming queries and responses. このアーカイブには、過去の照会について説明する属性 (例えば、送信された各アーティファクトのハッシュとファイル・タイプなど) や応答 (例えば、エンジンとアサーション (および NCT 投資金額) のマッピングなど) が含まれます。
  2. コンスーマーが、特定のアーティファクトのハッシュに関する情報を求める要求を送信します。
  3. The Ambassador consults their archive for information about the hash of the file.
  4. If the Ambassador has data about the hash, it returns that data.

The Ambassador may not have data about a hash because:

  1. ハッシュが、PolySwarm マーケットに送信されたことのないアーティファクトに対応している。
  2. The hash corresponds to an artifact that was not submitted to the PolySwarm market via this Ambassador. Another Ambassador, e.g. Swarm Technologies' Ambassador, may have seen the artifact.
  3. The Ambassador previously handled the artifact but did not archive for various reasons: error, legal requirements, etc.

If your archive does not have the data needed to directly respond to your consumer's query, you may considering:

  1. マーケットプレイスへのアーティファクトの (再) 送信をオファーする。
  2. Forwarding the consumer's request to Swarm Technologies' or a third party Ambassador that may have record of the results.

ハントのライフサイクル

Providing hunt capabilities to consumers requires:

  1. 送信されたアーティファクトのアーカイブ
  2. 利用者がアップロードしたルールをアーカイブに照らして評価するための、拡張性が高くて経済的に健全なインフラストラクチャー

The hunt lifecycle at a conceptual level:

  1. The consumer submits hunt criteria (e.g. a YARA rule) to the Ambassador.
  2. The Ambassador invokes a search process for the criteria.
  3. The Ambassador returns results to the consumer.

As an Ambassador, it's largely up to you to define how (or whether) you'd like to offer hunting functionality to your consumers. Swarm Technologies' Ambassador will initially support YARA rule scanning. We encourage others to support the same.


Developing Your Ambassador

アンバサダーとして成功するには

As an Ambassador, you'll be your consumer's interface to the PolySwarm marketplace. It's important that you strive to provide a service that is:

  • 使いやすい
  • 拡張性が高い
  • 低遅延
  • 高スループット
  • 費用対効果が高い

Developing an Ambassador

Windows-based Ambassadors are not supported; we strongly recommend developing Ambassadors under Linux.

Prerequisites

Configure your Linux-based development environment.

polyswarm-client ベースの作成

polyswarm-client provides a convenient basis for Ambassador development by abstracting polyswarmd API complexities and providing ready-to-use Ambassador examples. By building on top of polyswarm-client, you won't have to worry about maintaining polyswarmd API compatibility, freeing time to focus on your Ambassador's differentiating features and developing your business logic. This tutorial will build on polyswarm-client and use the examples provided therein.

If you'd like to build an Ambassador from scratch, we encourage you to complete this tutorial first and then consult the polyswarmd API for a description of interfaces your Ambassador must support. It will be your responsibility to track polyswarmd releases and update your API interface as necessarily to ensure uninterrupted service to your customers.

The below sections break down the example Ambassadors available in polyswarm-client master as of commit 3c2e432289276f69be96db4e8eb587a997900af9.

All example Ambassadors assume a 1:1 relationship between Ambassadors and communities. As an Ambassador developer, you may want to interface with multiple communities. This and other real-world concerns are covered in a subsequent section.

例: EICAR を送信するアンバサダー

EICAR is a test file used by the Antivirus industry to test engines' ability to detect malware. The file is not malicious, but is flagged as such by many antivirus engines.

Here we'll discuss submitting EICAR as a PolySwarm Ambassador. Elsewhere, we discuss building a Microengine that detects EICAR.

polyswarm-client には、EICAR を送信するアンバサダー (eicar.py) が付属しています。このアンバサダーは、公開分散ファイル共有ネットワークである IPFS でアーティファクトをホストします。

eicar.py begins:

import base64
import logging
import random
import os

from concurrent.futures import CancelledError
from polyswarmclient.abstractAmbassador import AbstractAmbassador

logger = logging.getLogger(__name__)

EICAR = base64.b64decode(
    b'WDVPIVAlQEFQWzRcUFpYNTQoUF4pN0NDKTd9JEVJQ0FSLVNUQU5EQVJELUFOVElWSVJVUy1URVNULUZJTEUhJEgrSCo=')
NOT_EICAR = 'this is not malicious'
ARTIFACTS = [('eicar', EICAR), ('not_eicar', NOT_EICAR)]

インポートとロギングの構成後に、EICAR の文字列と、明確に EICAR ではない文字列がハードコーディングされています。 これらの文字列は、ARTIFACTS 配列に入れられています。

Continuing:

BOUNTY_TEST_DURATION_BLOCKS = int(os.getenv('BOUNTY_TEST_DURATION_BLOCKS', 5))

eicar.py では、デフォルトのアサーション期間を 5 ブロックに設定しています。 実時間のブロック期間は、コミュニティーのホストによって決定されます。 Swarm Technologies がホストしているコミュニティーでは、1 ブロックは約 1 秒ごとにチェーンに追加されます。そのため、5 ブロックの期間は約 5 秒です。 このデフォルト値は、環境変数を使用してオーバーライドできます。

class Ambassador(AbstractAmbassador):
    """Ambassador which submits the EICAR test file"""

    def __init__(self, client, testing=0, chains=None, watchdog=0, submission_rate=30):
        """
        Initialize {{ cookiecutter.participant_name }}
        Args:
            client (`Client`): Client to use
            testing (int): How many test bounties to respond to
            chains (set[str]): Chain(s) to operate on
            watchdog: interval over which a watchdog thread should verify bounty placement on-chain (in number of blocks)
            submission_rate: if nonzero, produce a sleep in the main event loop to prevent the Ambassador from overloading `polyswarmd` during testing
        """
        init_logging([__name__], log_format='json')
        super().__init__(client, testing, chains, watchdog, submission_rate)

eicar.py's Ambassader is built on polyswarm-client's AbstractAmbassador. 特に、AbstractAmbassador は、ホストされている polyswarmd への接続を確立し、client というインスタンス変数でその接続を管理しています。

AbstractAmbassador declares a single method, generate_bounties, as abstract. AbstractAmbassador のサブクラスはすべて、このメソッドを定義する必要があります。

お分かりと思いますが、eicar.py におけるこのメソッドの実装は以下のように非常に単純です。

    async def generate_bounties(self, chain):
        """EICAR テスト文字列または無害の検体を送信する

        引数:
            chain (str): サンプルの要求先チェーン
        """
        amount = await self.client.bounties.parameters[chain].get('bounty_amount_minimum')

        while True:
            try:
                filename, content = random.choice(ARTIFACTS)

                logger.info('Submitting %s', filename)
                ipfs_uri = await self.client.post_artifacts([(filename, content)])
                if not ipfs_uri:
                    logger.error('Error uploading artifact to IPFS, continuing')
                    continue

                await self.push_bounty(amount, ipfs_uri, BOUNTY_TEST_DURATION_BLOCKS, chain)
            except CancelledError:
                logger.warning('Cancel requested')
                break
            except Exception:
                logger.exception('Exception in bounty generation task, continuing')
                continue

このメソッドでは、以下 (剰余エラー検査) を実行します。

  1. polyswarmd に最小初期報奨金額を照会します。
  2. 無限ループに入ります。
  3. eicar または not_eicar 文字列をアーティファクトとしてランダムに選択します。
  4. アーティファクトを IPFS でホストするように polyswarmd に指示します。
  5. 初期報奨金額 (許可される最小)、アーティファクトの URI、アサーション期間、chain を指定して、報奨金をポストするように polyswarmd に指示します。*

*chain refers to which blockchain to post the bounty on: "homechain" or "sidechain". This argument should always be side; it will be removed in a future polyswarm-client release.

Notes:

  1. polyswarm-client-derived Ambassadors are multi-threaded by default handling events asynchronously. This infinite loop will be isolated to the thread responsible for posting bounties; the remainder of the Ambassador will function normally.
  2. ループ内に明示的なスリープはありません。 This is intentional; the thread responsible for generate_bounties effectively sleeps while blocking on bounty submission each time it calls self.client.post_artifacts (blocking on IPFS host) and self.client.push_bounty (blocking on the announcement of the bounty in the marketplace by polyswarmd).

eicar.py is a trivial example that does not account for many real-world Ambassador operating concerns. Next, we'll expand on this example to an Ambassador that submits on-disk artifacts.

例: 「ファイル・システム」アンバサダー

polyswarm-client's filesystem.py Ambassador expands on the eicar.py Ambassador, submitting artifacts from a local filesystem.

先頭部分は、以下のように同様です。

import logging
import random
import os

from concurrent.futures import CancelledError
from polyswarmclient.abstractAmbassador import AbstractAmbassador
from polyswarmclient.corpus import DownloadToFileSystemCorpus

logger = logging.getLogger(__name__)

ARTIFACT_DIRECTORY = os.getenv('ARTIFACT_DIRECTORY', 'docker/artifacts')
ARTIFACT_BLACKLIST = os.getenv('ARTIFACT_BLACKLIST', 'truth.db').split(',')
BOUNTY_TEST_DURATION_BLOCKS = int(os.getenv('BOUNTY_TEST_DURATION_BLOCKS', 5))

今回も、インポートを処理し、報奨金期間をハードコーディングし、ロギングを構成しています。 filesystem.py makes use of polyswarmclient.corpus, a helper class that will download, decrypt and extract an artifact collection. Swarm Technologies uses this class internally during continuous integration to ensure that legitimately malicious artifacts are detected as such by microengines.

Continuing:

class Ambassador(AbstractAmbassador):
    """Ambassador which submits artifacts from a directory"""

    def __init__(self, client, testing=0, chains=None, watchdog=0, submission_rate=30):
        """Initialize a filesystem Ambassador
        Args:
            client (`Client`): Client to use
            testing (int): How many test bounties to respond to
            chains (set[str]): Chain(s) to operate on
        """
        init_logging([__name__], log_format='json')
        super().__init__(client, testing, chains, watchdog, submission_rate)

filesystem.py は、テストに役立つ、AbstractAmbassador へのさらに多くの引数を利用しています。

  • testing: when nonzero, this parameter specifies the maximum number of bounties the Ambassador will generate before exiting.
  • watchdog: ブロック間隔。 Bounties placed by this Ambassador are checked against each new block to ensure that the bounty has been successfully placed on-chain.
  • submission_rate: if nonzero, this produces a sleep in the main event loop to prevent the Ambassador from overloading polyswarmd during testing.

続けて、以下のようになっています。

        self.artifacts = []
        u = os.getenv("MALICIOUS_BOOTSTRAP_URL")
        if u:
            logger.info("Unpacking malware corpus at {0}".format(u))
            d = DownloadToFileSystemCorpus()
            d.download_and_unpack()
            bfl = d.get_benign_file_list()
            mfl = d.get_malicious_file_list()
            logger.info("Unpacking complete, {0} malicious and {1} benign files".format(len(mfl), len(bfl)))
            self.artifacts = bfl + mfl
        else:
            for root, dirs, files in os.walk(ARTIFACT_DIRECTORY):
                for f in files:
                    self.artifacts.append(os.path.join(root, f))

If the environment variable MALICIOUS_BOOTSTRAP_URL is set, the Ambassador downloads artifacts from a testing repository. If it's not set, ARTIFACT_DIRECTORY directory is walked relative to the Ambassador's current working directory. 報奨金を生成する準備として、ファイルが収集されます。

filesystem.py では、以下のように generate_bounties メソッドをオーバーライドしています。

    async def generate_bounties(self, chain):
        """ファイル・システムから報奨金を送信
        引数:
            chain (str): 検体の要求先チェーン
        """
        amount = await self.client.bounties.parameters[chain].get('bounty_amount_minimum')

        while True:
            try:
                filename = random.choice(self.artifacts)

                logger.info('Submitting file %s', filename)
                ipfs_uri = await self.client.post_artifacts([(filename, None)])
                if not ipfs_uri:
                    logger.error('Error uploading artifact to IPFS, continuing')
                    continue

                await self.push_bounty(amount, ipfs_uri, BOUNTY_TEST_DURATION_BLOCKS, chain)
            except CancelledError:
                logger.warning('Cancel requested')
                break
            except Exception:
                logger.exception('Exception in bounty generation task, continuing')
                continue

This is identical to the logic contained withing eicar.py, refer to the previous section for a breakdown.

filesystem.py は、eicar.py をベースに作成しており、ディスク上に (オプションで、リモート URL に) アーティファクトを作成します。 In a real-world Ambassador, these artifacts would come from the consumer's submissions to the Ambassador.

participant-template を使用したアンバサダーの作成

The easiest way to get started is to build an Ambassador using participant-template. By using the template, your Ambassador will be based on polyswarm-client, allowing you to focus on business logic.

participant-template からマイクロエンジンを作成します。 これを行うには、cookiecutter が必要です。以下のようにします。

pip install cookiecutter

With cookiecutter installed, jump-starting your microengine from our participant-template is as easy as:

cookiecutter https://github.com/polyswarm/participant-template

And answering some prompts. Read about these prompts here.

Choose:

  • participant_type: Ambassador
  • platform: linux (Windows Ambassadors are not supported)
  • participant_name: helloworld
  • 残りはデフォルトを受け入れます

You'll be left with an Ambassador-helloworld directory. Change directory (cd) into Ambassador-helloworld:

$ cd Ambassador-helloworld

アンバサダーのカスタマイズ

Here we'll implement a simple, minimum viable Ambassador, re-creating the EICAR Ambassador described above.

最小限の実行可能なアンバサダーの実装はシンプルであり、Ambassadorgenerate_bounties メソッドを実装するだけです。 This method is found in Ambassador_<participant_name_slug>/src/<author_org_slug>_<participant_name_slug>/__init__.py (Ambassador_helloworld/src/polyswarm_helloworld/__init__.py if you followed the cookiecutter prompts as described above). Production Ambassadors will, of course, need to do far more than this.

Open Ambassador_helloworld/src/polyswarm_helloworld/__init__.py.

Customize the file to include the EICAR and not-EICAR definitions we saw in the EICAR Ambassador:

...
logger = logging.getLogger(__name__)

EICAR = base64.b64decode(
    b'WDVPIVAlQEFQWzRcUFpYNTQoUF4pN0NDKTd9JEVJQ0FSLVNUQU5EQVJELUFOVElWSVJVUy1URVNULUZJTEUhJEgrSCo=')
NOT_EICAR = 'this is not malicious'
ARTIFACTS = [('eicar', EICAR), ('not_eicar', NOT_EICAR)]

BOUNTY_TEST_DURATION_BLOCKS = int(os.getenv('BOUNTY_TEST_DURATION_BLOCKS', 5))
...

次に、以下のように、EICAR と非 EICAR を送信するように generate_bounty メソッドをカスタマイズします。

    async def generate_bounties(self, chain):
        """EICAR テスト文字列または無害の検体を送信する

        引数:
            chain (str): サンプルの要求先チェーン
        """
        amount = await self.client.bounties.parameters[chain].get('bounty_amount_minimum')

        while True:
            try:
                filename, content = random.choice(ARTIFACTS)

                logger.info('Submitting %s', filename)
                ipfs_uri = await self.client.post_artifacts([(filename, content)])
                if not ipfs_uri:
                    logger.error('Error uploading artifact to IPFS, continuing')
                    continue

                await self.push_bounty(amount, ipfs_uri, BOUNTY_TEST_DURATION_BLOCKS, chain)
            except CancelledError:
                logger.warning('Cancel requested')
                break
            except Exception:
                logger.exception('Exception in bounty generation task, continuing')
                continue

Once these changes are made, you now have an EICAR Ambassador built on participant-template!

Next, we'll consider some real-world concerns that go beyond the current scope of this document and then test our EICAR-submitting Ambassador.


Production Ambassador Considerations

The eicar.py and filesystem.py Ambassadors are proof of concepts that do not address many requirements desirable of production Ambassadors, including, but not limited to:

  1. 利用者と対話する API*
  2. 複数のコミュニティーとやり取りする機能
  3. 速度制限や課金などのために利用者の要求を追跡する手段
  4. 低遅延と高スループットを確保するために、需要に応じて調整できる拡張性の高いインフラストラクチャー
  5. ハッシュ検索、ハントなどの機能の基盤となる過去の照会と結果の拡張性の高いアーカイブ

Ready to build your ambassador and serve as your clients' window into the PolySwarm marketplace?

I want to build an Ambassador →

Ambassadors are only supported under Linux.