PolySwarmPolySwarmPolySwarmPolySwarm
Help

Testing in Windows (Docker)

概览

这些说明涵盖测试基于 Windows 的:

  • microengines
  • arbiters (coming soon)

Windows-based Ambassadors are not supported.

In this page, we use microengine-eicar as the name of the participant's directory. Substitute this directory name for your own in the commands that follow.

Additionally, we've shortened PowerShell prompts to PS > and Linux prompts to $ to make commands easier to read.

Finally, unless otherwise stated, all commands are to be executed inside your PolySwarm Python virtual envionment, established while configuring your development environment.


建立参与者 Docker 映像

您需要建立参与者 Docker 映像,以进行单元测试和集成测试:

PS > docker build -t (Get-Item -Path ".\").Name -f docker/Dockerfile .

这将生成一个标记有目录名称的 Docker 映像,例如 microengine-eicar


Unit Testing

We'll use tox to test our Microengine. tox runs whatever unit tests you add to tests/scan_test.py. EICAR-detecting unit tests are included for microengine participants.

Run tox in your Windows docker image to conduct unit tests:

PS > docker run --isolation=process microengine-eicar tox

Output for microengine unit tests will look similar to the following:

GLOB sdist-make: C:\microengine\setup.py
py36 create: C:\microengine\.tox\py36
py36 inst: C:\microengine\.tox\dist\polyswarm_eicar-0.1.zip
py36 installed: aiodns==1.2.0,aiohttp==3.5.4,aioredis==1.2.0,aioresponses==0.6.0,aiorwlock==0.6.0,asn1crypto==0.24.0,async-timeout==3.0.1,asynctest==0.12.2,atomicwrites==1.3.0,attrdict==2.0.1,attrs==19.1.0,base58==0.2.5,certifi==2019.3.9,cffi==1.12.3,chardet==3.0.4,click==6.7,coincurve==11.0.0,colorama==0.4.1,coverage==4.5.1,cytoolz==0.9.0.1,eth-abi==1.3.0,eth-account==0.3.0,eth-hash==0.2.0,eth-keyfile==0.5.1,eth-keys==0.2.1,eth-rlp==0.1.2,eth-typing==2.1.0,eth-utils==1.5.1,ethereum==2.3.2,future==0.17.1,hexbytes==0.1.0,hiredis==1.0.0,hypothesis==3.82.1,idna==2.7,idna-ssl==1.1.0,lru-dict==1.1.6,malwarerepoclient==0.1,more-itertools==7.0.0,multidict==4.5.2,mypy-extensions==0.4.1,parsimonious==0.8.1,pbkdf2==1.3,pluggy==0.9.0,polyswarm-client==1.4.1,polyswarm-eicar==0.1,py==1.8.0,py-ecc==1.6.0,pycares==3.0.0,pycparser==2.19,pycryptodome==3.8.1,pyethash==0.1.27,pypiwin32==223,pysha3==1.0.2,pytest==3.9.2,pytest-asyncio==0.9.0,pytest-cov==2.6.0,pytest-timeout==1.3.2,python-json-logger==0.1.9,python-magic==0.4.15,pywin32==224,PyYAML==5.1,repoze.lru==0.7,requests==2.20.1,rlp==1.1.0,scrypt==0.8.13,six==1.12.0,toml==0.10.0,toolz==0.9.0,tox==3.4.0,typing==3.6.6,typing-extensions==3.7.2,urllib3==1.24.2,virtualenv==16.5.0,web3==4.8.2,websockets==6.0,yara-python==3.7.0,yarl==1.3.0
py36 run-test-pre: PYTHONHASHSEED='511'
py36 runtests: commands[0] | pytest -s
============================= test session starts =============================
platform win32 -- Python 3.6.7, pytest-3.9.2, py-1.8.0, pluggy-0.9.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('C:\\microengine\\.hypothesis\\examples')
rootdir: C:\microengine, inifile:
plugins: timeout-1.3.2, cov-2.6.0, asyncio-0.9.0, hypothesis-3.82.1
collected 1 item

tests\test_eicar.py .

============================== warnings summary ===============================
c:\python36\lib\site-packages\eth_utils\applicators.py:32: DeprecationWarning: combine_argument_formatters(formatter1, formatter2)([item1, item2])has been deprecated and will be removed in a subsequent major version release of the eth-utils library. Update your calls to use apply_formatters_to_sequence([formatter1, formatter2], [item1, item2]) instead.
  "combine_argument_formatters(formatter1, formatter2)([item1, item2])"

-- Docs: https://docs.pytest.org/en/latest/warnings.html
==================== 1 passed, 1 warnings in 3.37 seconds =====================
___________________________________ summary ___________________________________
  py36: commands succeeded
  congratulations :)

You can safely ignore the combine_argument_formatters deprecation warning.


集成测试

Integration testing a Windows Docker participant requires:

  1. A Docker image of your participant (we've already made this).
  2. A Linux VM for standing up a PolySwarm testnet (we'll make this now).

The recommendations presented here are hard-won. We strongly recommend that you test using the exact recommendations presented here. Using any other configuration will make it difficult for us to provide you with support.

创建一个 Linux Guest

In the first step, you installed Hyper-V, Microsoft's flagship hypervisor. We'll use Hyper-V to create a Linux VM for hosting our testnet.

  1. Search "Hyper-V Quick Create" in the desktop search bar.
  2. Run Hyper-V Quick Create.
  3. Select Ubuntu 18.04.1 as the operating system.
  4. Click "More Options".
  5. Name the VM polyswarm_lin.
  6. Choose a network connection that can communicate with both Docker containers and the Internet. Typically, this is called Default Switch, but the name may vary. If in doubt, confirm that your chosen Network appears in docker network list.
  7. Click "Create Virtual Machine".

It will take a few minutes to create the VM. When creation is done, you're offered the option to "Connect" to the VM or "Edit settings...". Choose "Edit settings...".

In the settings window, reconfigure for:

  • Memory: 8192MB

Click "OK" to save and close settings. Click "Connect" to connect to the VM. Click "Start" in the VM window.

Follow the installation prompts and complete the Ubuntu installation.

Once the installation is complete, shut down the VM.

Configure Networking

On the host, you'll be creating a new virtual switch with docker network create (a thin wrapper around the Hyper-V VS fabric manager). This permits placing the Linux VM into the same virtual switching fabric as the Docker containers. This virtual switch will be tied to an interface you select in windowsshim.interface. Although you may use any interface name, we recommend you use an Ethernet adaptor when running this command as some users have reported issues running this command with a Wireless adaptor.

The name of this Ethernet adapter may vary. On your Windows host:

  1. Open the Control Panel.
  2. Select Network & Internet.
  3. Look on the right panel under "Network status".
  4. It should say that your host is connected to the Internet.
  5. In the network diagram above "You're connected to the Internet", look at the Ethernet jack icon.
  6. Note the Ethernet adapter displayed in parens (). This adapter will likely be something similar to Ethernet0.

In an elevated PowerShell prompt, create the transparent Docker network switch, substituting Ethernet0 for the name of your network connect as appropriate:

PS > $networkid=docker network create -d transparent -o com.docker.network.windowsshim.interface="Ethernet0" "PolySwarm Network"

Attach your Linux VM to the same switch (the VM must be powered off for this to work):

PS> Connect-VMNetworkAdapter -VMName polyswarm_lin -SwitchName $networkid

We need to create a transparent network, because Docker uses IP Address Management (IPAM) to issue IP addresses. If you were to connect your Linux VM directly to an existing network, your VM would not acquire an IP address automatically.

Verify Network Configuration

Before we test connectivity, we need to verify that our Linux VM and Windows Docker container have IP addresses on the same network.

Launch your Linux VM and execute the following in a terminal:

$ ip a show eth0
eth0: <BROADCAST,MULTICAST,UP> mtu 1500 group default qlen 1
    link/ether 00:15:5d:07:c9:c1
    inet 192.168.86.31/24 brd 192.168.86.1 scope global dynamic
       valid_lft forever preferred_lft forever

Of note here is 192.168.86.31/24, meaning the Linux VM is on the 192.168.86.0 subnet with a mask of 255.255.255.0. Let's see if this subnet and mask matches our Windows Docker container. The exact value here will likely vary

Launch your Windows container (note the --network argument):

PS > docker run --isolation=process --network "PolySwarm Network" -it microengine-eicar

Inside your Windows Docker container, run:

[container] PS > ipconfig

Windows IP Configuration


Ethernet adapter vEthernet (Ethernet):

   Connection-specific DNS Suffix  . : localdomain
   Link-local IPv6 Address . . . . . : fe80::fc35:45f8:8c2e:3032%47
   IPv4 Address. . . . . . . . . . . : 192.168.48.137
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.48.2

It may take several minutes for your Windows Docker container to acquire an IP address. If the IP address listed via ipconfig is not on the same subnet as the Linux VM, wait a few minutes and query again.

With both our Linux VM and Windows Docker container on the same subnet, let's verify connectivity.

First, within your Windows Docker container, alias the Linux host's address to polyswarmd (substitute the IP address with the address of the Linux VM):

[container] PS > Add-Content C:\Windows\System32\Drivers\etc\hosts -Value "192.168.86.31 polyswarmd"

Ping the IPv4 address of the Linux host polyswarmd:

[container] PS > ping polyswarmd
Pinging 192.168.86.30 with 32 bytes of data:
Reply from 192.168.86.30: bytes=32 time<1ms TTL=128
...

Success!

设置 Linux VM 来托管一个本地 Testnet

Install Docker & docker-compose

The PolySwarm testnet will run on the Linux VM using Docker (& docker-compose). The first step is to install Docker in the Linux VM:

$ sudo apt-get update && sudo apt-get install -y curl
$ curl -fsSL https://get.docker.com -o get-docker.sh
$ chmod +x get-docker.sh
$ ./get-docker.sh
$ sudo usermod -aG docker $USER

Once installed, verify that the installation works by running:

$ docker ps

This should print something like:

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Next, install docker-compose:

$ curl -L "https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m)" -o docker-compose
$ sudo mv docker-compose /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose

Once installed, verify that the installation works.

$ docker-compose -v

This should output at least: docker-compose version 1.21.1, build 5a3f1a3

After installing both Docker and docker-compose, we recommend adding your user to the docker group so that you can easily issue docker commands without sudo:

$ sudo usermod -aG docker ${USER}

You'll need to reboot the Linux VM in order for the change to take effect.

Install Git

We need to download several PolySwarm source code repositories to use in testing. For that, you need to install Git into your Linux VM:

$ sudo apt update && sudo apt install -y git

Download orchestration

PolySwarm 市场由众多参与者及技术所组成:以太坊、IPFS 节点、合约、微引擎、代表、仲裁者、工件等等。 测试一个单独的组件通常需要能够获得所有其他组件。

借助 orchestration 项目,可简便、顺畅地建立一个完整的测试网。 顾名思义,orchestration 协调所有必要的组件,以便在本地开发计算机上搭建和拆分整个 PolySwarm 市场环境。

Clone orchestration:

$ git clone https://github.com/polyswarm/orchestration

cd into the orchestration directory.

orchestration 存储库中的以下 docker-compose 文件将有可帮助测试:

  • base.yml:定义测试网的基础结构基础知识(例如 geth 节点和 relays
  • ambassador-eicar.yml:使用 eicar.py 定义代表服务。
  • arbiter-verbatim.yml:使用 verbatim.py 定义仲裁者服务。
  • microengine-eicar.yml:使用 eicar.py 定义微引擎服务。
  • microengine-clamav.yml:使用 clamav.py + ClamAV 守护进城定义微引擎服务。
  • microengine-multi.yml:使用 multi.py 定义微引擎服务。

测试您的参与者

We're going to have to bounce back and forth between our Linux VM and Windows Docker container. We will first start the testnet in the Linux VM. Then we will start your participant in the Windows Docker container. Finally, we will start the ambassador inside the Linux VM.

Linux VM: Launch the Testnet

Create the orchestration_default network, if you haven't already:

$ docker network create orchestration_default

Launch the base network + an arbiter:

$ docker-compose -f base.yml -f arbiter-verbatim.yml pull
$ docker-compose -f base.yml -f arbiter-verbatim.yml up

If you get an error about orchestration_default network not existing, manually create it with docker network create orchestration_default and try again.

Wait for polyswarmd to become available. It will take several minutes for polyswarmd to become available. During this time, you will see many messages like Problem with dial... dial tcp connection refused. and chain for config not available in consul yet. These errors are normal while the testnet is initializing, so have patience.

View polyswarmd logs in a new shell session (from the orchestration directory):

docker-compose -f base.yml logs --follow polyswarmd

In this new shell, wait for polyswarmd:

Attaching to orchestration_polyswarmd_1_c1d732ac5355
polyswarmd_1_c1d732ac5355 | [2019-05-13 20:52:36 +0000] [1] [INFO] Starting gunicorn 19.9.0
polyswarmd_1_c1d732ac5355 | [2019-05-13 20:52:36 +0000] [1] [INFO] Listening at: http://0.0.0.0:31337 (1)
...
polyswarmd_1_c1d732ac5355 | CRITICAL:polyswarmd.wsgi:polyswarmd is ready!
polyswarmd_1_c1d732ac5355 | CRITICAL:polyswarmd.wsgi:polyswarmd is ready!
polyswarmd_1_c1d732ac5355 | CRITICAL:polyswarmd.wsgi:polyswarmd is ready!
polyswarmd_1_c1d732ac5355 | CRITICAL:polyswarmd.wsgi:polyswarmd is ready!

When ready, polyswarmd will print polyswarmd is ready! once for every polyswarmd thread (4 by default).

Windows Docker Container: Test polyswarmd Connection & Health

Inside your Windows container, it's time to verify that polyswarmd is available and ready to respond to your Microengine:

[container] PS > (curl -UseBasicParsing http://polyswarmd:31337/status).Content

Which should emit the following:

{"result":{"community":"gamma","home":{"block":350,"reachable":true,"syncing":false},"ipfs":{"reachable":true},"side":{"block":349,"reachable":true,"syncing":false}},"status":"OK"}
...

The receipt of the key-datum pair "status":"OK" indicates that polyswarmd is ready.

Windows VM: Launch balancemanager

balancemanager is a utility to maintain an acceptable balance of (fake) PolySwarm Nectar (NCT) on the sidechain of our local testnet where all transactions will take place. It identifies your balance by the 'keyfile', a JSON-encoded file holding all information required to create and verify cryptographic material in the network. The utility and the keyfile will have been installed alongside the Docker container build process.

If you modified or copied your keyfile, it may not have an encoding compatible with Window's default encoding. If any program below emits errors concerning invalid JSON, try the following:

PS> Set-Content -path C:\microengine\microengine_keyfile -Encoding ascii C:\microengine\microengine_keyfile

Inside your Windows container's PowerShell session, launch balancemanager as follows:

[container[ PS > balancemanager --log INFO maintain --keyfile microengine_keyfile --password password --polyswarmd-addr polyswarmd:31337 --insecure-transport 100000 500000

It will print output similar to the following:

INFO:balancemanager:2019-04-23 16:46:54,787 Logging in text format.
INFO:polyswarmclient:2019-04-23 16:46:54,787 Logging in text format.
INFO:balancemanager.__main__:2019-04-23 16:46:54,787 Maintaining the minimum balance by depositing 500000.0 NCT when it falls below 100000.0 NCT
INFO:polyswarmclient:2019-04-23 16:46:56,474 Using account: 0x05328f171b8c1463eaFDACCA478D9EE6a1d923F8
INFO:polyswarmclient:2019-04-23 16:46:57,175 Received connected on chain side: {'start_time': '1556063097.7256606'}
INFO:polyswarmclient:2019-04-23 16:46:57,176 Connected to event socket at: 1556063097.7256606
INFO:polyswarmclient:2019-04-23 16:46:57,178 Received connected on chain home: {'start_time': '1556063097.7256606'}
INFO:polyswarmclient:2019-04-23 16:46:57,178 Connected to event socket at: 1556063097.7256606
INFO:balancemanager:2019-04-23 16:46:57,487 Sidechain balance (0 NCT) is below minimum (100000000000000000000000 NCT). Depositing 500000000000000000000000 NCT
WARNING:polyswarmclient.transaction:2019-04-23 16:46:58,568 Updated nonce to 1 on home
INFO:balancemanager:2019-04-23 16:47:00,571 Waiting for 26 more blocks
INFO:balancemanager:2019-04-23 16:47:00,583 Waiting for 25 more blocks
...
INFO:balancemanager:2019-04-23 16:47:23,136 Waiting for 1 more blocks
INFO:balancemanager:2019-04-23 16:47:24,280 Waiting for 0 more blocks
INFO:balancemanager:2019-04-23 16:47:25,436 Checking NCT balance
INFO:balancemanager:2019-04-23 16:47:25,436 Balance increased to 500000000000000000000000 NCT

When it starts printing Received block on chain messages, you are ready to launch your participant.

Windows VM: Launch Your Participant

You will want to keep balancemanager continually running but will need to continue to work with PowerShell, you can do in two different ways:

  1. (Recommended) You can start another PowerShell window with docker exec $CONTAINER_ID, where $CONTAINER_ID refers to the id value for your container printed by docker ps.
  2. Restart the process as a service: Install-Service -Name balancemanager -Path "balancemanager --log INFO maintain --keyfile microengine_keyfile --password password --polyswarmd-addr polyswarmd:31337 --insecure-transport 100000 500000" followed by Start-Service balancemanager. This will emit all error and log messages to C:\balancemanager.log

Now, you will run your participant using a command similar to the following command. Be sure to update the value for the --backend argument to match the name of your participant's package directory (i.e. the name in setup.py):

[container] PS > microengine --log INFO --keyfile microengine_keyfile --password password --polyswarmd-addr polyswarmd:31337 --insecure-transport --testing 2 --backend polyswarm_eicar

This will print output similar to the following:

INFO:root:2018-12-06 16:56:20,674 Logging in text format.
INFO:polyswarmclient:2018-12-06 16:56:21,299 Using account: 0x05328f171b8c1463eaFDACCA478D9EE6a1d923F8
INFO:polyswarmclient:2018-12-06 16:56:21,690 Received connected on chain side: {'start_time': '1544126035.507124'}
INFO:root:2018-12-06 16:56:21,690 Connected to event socket at: 1544126035.507124
INFO:polyswarmclient:2018-12-06 16:56:22,691 Received block on chain side: {'number': 18255}
...
INFO:polyswarmclient:2018-12-06 16:56:44,205 Received block on chain side: {'number': 18277}
INFO:polyswarmclient:2018-12-06 16:56:44,283 Received bounty on chain side: {'author': '0x4B1867c484871926109E3C47668d5C0938CA3527', 'expiration': '18297', 'uri': 'QmVoLQJ2nm4V6XiZXC9vEUrCaTHdkXS7y3crztZ5HwC9iK', 'guid': '48dd5360-47a3-4e12-a975-eb30fed5cc22', 'amount': '62500000000000000'}
INFO:polyswarmclient.abstractmicroengine:2018-12-06 16:56:44,283 Testing mode, 1 bounties remaining
INFO:polyswarmclient.abstractmicroengine:2018-12-06 16:56:44,455 Responding to bounty: 48dd5360-47a3-4e12-a975-eb30fed5cc22
INFO:polyswarmclient:2018-12-06 16:56:45,237 Received block on chain side: {'number': 18278}
INFO:polyswarmclient:2018-12-06 16:56:46,393 Received block on chain side: {'number': 18279}
INFO:polyswarmclient.events:2018-12-06 16:56:46,440 OnNewBountyCallback callback results: [[{'bounty_guid': '48dd5360-47a3-4e12-a975-eb30fed5cc22', 'mask': [True], 'bid': '62500000000000000', 'commitment': '44296088244268214239924675885675264686302131561550908677050134822720003742540', 'author': '0x05328f171b8c1463eaFDACCA478D9EE6a1d923F8', 'index': 0}]]
INFO:polyswarmclient:2018-12-06 16:56:46,456 Received bounty on chain side: {'author': '0x4B1867c484871926109E3C47668d5C0938CA3527', 'expiration': '18299', 'uri': 'QmVjWbqv8aXEPE53vDYS9r3wG7odJjrHXf7ci1xfLyNAEU', 'guid': '40862925-3e00-41b2-a946-365135d87070', 'amount': '62500000000000000'}
INFO:polyswarmclient:2018-12-06 16:56:46,456 Received assertion on chain side: {'bounty_guid': '48dd5360-47a3-4e12-a975-eb30fed5cc22', 'mask': [True], 'bid': '62500000000000000', 'commitment': '44296088244268214239924675885675264686302131561550908677050134822720003742540', 'author': '0x05328f171b8c1463eaFDACCA478D9EE6a1d923F8', 'index': 0}
INFO:polyswarmclient.abstractmicroengine:2018-12-06 16:56:46,456 Testing mode, 0 bounties remaining
INFO:polyswarmclient.abstractmicroengine:2018-12-06 16:56:46,643 Responding to bounty: 40862925-3e00-41b2-a946-365135d87070
INFO:polyswarmclient:2018-12-06 16:56:47,409 Received block on chain side: {'number': 18280}
INFO:polyswarmclient.events:2018-12-06 16:56:48,222 OnNewBountyCallback callback results: [[{'bounty_guid': '40862925-3e00-41b2-a946-365135d87070', 'mask': [True], 'bid': '62500000000000000', 'commitment': '26135711486835189252810507112407250051211627558503078858520125577864847775053', 'author': '0x05328f171b8c1463eaFDACCA478D9EE6a1d923F8', 'index': 0}]]
INFO:polyswarmclient:2018-12-06 16:56:48,440 Received block on chain side: {'number': 18281}
INFO:polyswarmclient:2018-12-06 16:56:48,503 Received bounty on chain side: {'author': '0x4B1867c484871926109E3C47668d5C0938CA3527', 'expiration': '18301', 'uri': 'QmVoLQJ2nm4V6XiZXC9vEUrCaTHdkXS7y3crztZ5HwC9iK', 'guid': 'b41ef0f8-039f-4448-aadf-4d4135cdd94b', 'amount': '62500000000000000'}
INFO:polyswarmclient:2018-12-06 16:56:48,503 Received assertion on chain side: {'bounty_guid': '40862925-3e00-41b2-a946-365135d87070', 'mask': [True], 'bid': '62500000000000000', 'commitment': '26135711486835189252810507112407250051211627558503078858520125577864847775053', 'author': '0x05328f171b8c1463eaFDACCA478D9EE6a1d923F8', 'index': 0}
WARNING:polyswarmclient.abstractmicroengine:2018-12-06 16:56:48,503 Received new bounty, but finished with testing mode

Running with --testing 2 means that your Microengine will respond to 2 bounties and then refuse to respond to further bounties by shutting itself off. You can adjust this number if you want it to process more bounties in your tests.

Linux VM: Launch the Ambassador

Back in the Linux VM, launch the ambassador, so it can place bounties on the testnet. Run the following commands in the orchestration directory:

$ docker-compose -f ambassador-eicar.yml pull
$ docker-compose -f ambassador-eicar.yml up

Observe your Microengine's output, after a few seconds, you'll see your engine is seeing and responding to new bounties being placed on the test network!