paper002-Sonata

《Sonata: Query-Driven Streaming Network Telemetry》

SIGCOMM ‘18

总结

　　现有遥测系统可以实时收集和分析测量数据，但存在两个普遍问题：（1）仅支持单一遥测任务；（2）处理和存储开销随流量及查询量增加而增大。

学术界的研究人员也关注到了上面的问题，Arpit Gupta等人针对遥测提供了一个声明式接口，可以满足11种遥测任务的查询需求[7]。这11种遥测任务包括：TCP新建连接查询、Slowloris DOS攻击查询、Zorro Attacks查询、SSH暴力破解攻击查询、Superspreader、端口扫描查询、DDoS攻击查询、TCP SYN Flood攻击查询、TCP Incomplete Flows查询、DNS Tunneling检测查询、DNS反向攻击查询。

Sonata搭建步骤

软件下载和安装

参考教程：

virtualbox+vagrant学习-1-环境安装及vagrantfile的简单配置-Mac系统

Git安装和使用

Xming

Putty

sonata在github上的教程

备注：1.clone时打git的目录即为下载的目录；2.vagrant up耗时较长（约1-2h，可以看两集电视剧）请保持网络畅通，请耐心等待；3.全程开VPN（学长说的；快连VPN）4.上述软件安装即可，无须其他配置；比如：git只需要下载，安装，无需执行上述教程中后续的步骤；5.上述命令的执行均在git中执行；

在SONATA-DEV目录下（vagrantfile所在目录）打开git，运行vagrant ssh出现：

yuan@DESKTOP-9IE8SJC MINGW64 /f/rj/sonatagit2020_12_13/SONATA-DEV (tutorial)
$ vagrant ssh
Welcome to Ubuntu 14.04.6 LTS (GNU/Linux 3.13.0-170-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Sun Dec 13 09:31:01 UTC 2020

  System load:  0.18               Processes:           90
  Usage of /:   16.8% of 39.34GB   Users logged in:     0
  Memory usage: 10%                IP address for eth0: 10.0.2.15
  Swap usage:   0%

  Graph this data and manage this system at:
    https://landscape.canonical.com/

New release '16.04.7 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


Last login: Sun Dec 13 09:31:01 2020 from 10.0.2.2
vagrant@vighata:~$

到此为止，Windows环境下虚拟机远程登陆配置成功（sonata及其相关配置文件在配置好的虚拟机中：vagrant/目录下；该目录下的dev是共享文件夹）；

sonata使用：教程1

根据步骤2.出现vagrant@vighata:~$后依次执行指令1. cd ~/dev、2. export SPARK_HOME=/home/vagrant/spark/、3. sudo $SPARK_HOME/bin/spark-submit sonata/examples/newly_opened_connections/test_app.py之后会出现

错误：ImportError: No module named mysql.connector

解决方案：ImportError: No module named mysql.connector）在vagrant@vighata:后面输入：pip install mysql-connector-python-rf==2.1.3；

备注：定位到` File “/home/vagrant/dev/sonata/dataplane_driver/p4/emitter/emitter.py”, line 7, in
```
import mysql.connector`初步推测该错误原因和python的版本以及python包mysql-connector有关；
```

此时再次执行sudo $SPARK_HOME/bin/spark-submit sonata/examples/newly_opened_connections/test_app.py出现system ready表示sonata成功启动（应该是sonata启动过程中的相应配置，需要读一下，所以复制下来了，该终端运行结束的方法在本步骤的备注里）：

vagrant@vighata:~/dev$ sudo $SPARK_HOME/bin/spark-submit sonata/examples/newly_opened_connections/test_app.py
*********************************************************************
*                   Receiving User Queries                          *
*********************************************************************


*********************************************************************
*                   Generating Query Plan                           *
*********************************************************************


2020-12-13 10:18:40,343 - P4Target - INFO - init
2020-12-13 10:18:40,343 - P4DataPlane - INFO - init
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/12/13 10:18:40 INFO SparkContext: Running Spark version 1.6.1
20/12/13 10:18:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/12/13 10:18:40 WARN Utils: Your hostname, vighata resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface eth0)
20/12/13 10:18:40 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
20/12/13 10:18:40 INFO SecurityManager: Changing view acls to: root
20/12/13 10:18:40 INFO SecurityManager: Changing modify acls to: root
20/12/13 10:18:40 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
20/12/13 10:18:41 INFO Utils: Successfully started service 'sparkDriver' on port 47073.
20/12/13 10:18:41 INFO Slf4jLogger: Slf4jLogger started
20/12/13 10:18:41 INFO Remoting: Starting remoting
20/12/13 10:18:41 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.0.2.15:54932]
20/12/13 10:18:41 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 54932.
20/12/13 10:18:41 INFO SparkEnv: Registering MapOutputTracker
20/12/13 10:18:41 INFO SparkEnv: Registering BlockManagerMaster
20/12/13 10:18:41 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-b8b9310b-e6f9-41d8-85ec-a5e4a021eea9
20/12/13 10:18:41 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
20/12/13 10:18:41 INFO SparkEnv: Registering OutputCommitCoordinator
20/12/13 10:18:42 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/12/13 10:18:42 INFO SparkUI: Started SparkUI at http://10.0.2.15:4040
20/12/13 10:18:42 INFO Utils: Copying /home/vagrant/dev/sonata/examples/newly_opened_connections/test_app.py to /tmp/spark-098d43ad-67c4-4a9d-81d2-ef3b54c13593/userFiles-146cb70a-708c-45f9-9e47-afa2a299ecea/test_app.py
20/12/13 10:18:42 INFO SparkContext: Added file file:/home/vagrant/dev/sonata/examples/newly_opened_connections/test_app.py at file:/home/vagrant/dev/sonata/examples/newly_opened_connections/test_app.py with timestamp 1607854722257
20/12/13 10:18:42 INFO Executor: Starting executor ID driver on host localhost
20/12/13 10:18:42 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 53540.
20/12/13 10:18:42 INFO NettyBlockTransferService: Server created on 53540
20/12/13 10:18:42 INFO BlockManagerMaster: Trying to register BlockManager
20/12/13 10:18:42 INFO BlockManagerMasterEndpoint: Registering block manager localhost:53540 with 511.1 MB RAM, BlockManagerId(driver, localhost, 53540)
20/12/13 10:18:42 INFO BlockManagerMaster: Registered BlockManager
2020-12-13 10:18:43,372 - P4Target - INFO - run
2020-12-13 10:18:43,372 - P4Target - INFO - init P4 application object
2020-12-13 10:18:43,372 - P4Application - INFO - init
2020-12-13 10:18:43,372 - P4Application - DEBUG - create query pipeline for qid: 10024
2020-12-13 10:18:43,373 - P4Application - DEBUG - create query pipeline for qid: 10032
2020-12-13 10:18:43,374 - P4Target - INFO - generate p4 code and commands
2020-12-13 10:18:43,377 - P4Target - INFO - compile p4 code to json
2020-12-13 10:18:43,377 - P4DataPlane - INFO - compile p4 to json
2020-12-13 10:18:43,734 - P4Target - INFO - initialize the dataplane with the json configuration
2020-12-13 10:18:43,735 - P4DataPlane - INFO - initialize
2020-12-13 10:18:43,735 - P4DataPlane - INFO - create interfaces
*********************************************************************
*                   Updating data-plane driver                       *
*********************************************************************


*********************************************************************
*                   Updating data-plane driver                       *
*********************************************************************


*********************************************************************
*                   Updating Streaming Driver                       *
*********************************************************************


Calling target program-options parser
Adding interface m-veth-1 as port 11
Adding interface m-veth-2 as port 12
Adding interface m-veth-3 as port 13
For 10024 stream processor sending  []  at time 7.76877212524
For 10032 stream processor sending  []  at time 7.79664301872
Traceback (most recent call last):
  File "/home/vagrant/bmv2/targets/simple_switch/sswitch_CLI.py", line 22, in <module>
    import runtime_CLI
  File "/home/vagrant/bmv2/tools/runtime_CLI.py", line 1887
    print("  -> (L1h={}, rid={})".format(L1h, rid), end=' ')
                                                       ^
SyntaxError: invalid syntax
Command '['/home/vagrant/bmv2/targets/simple_switch/sswitch_CLI', '/home/vagrant/dev/sonata/examples/newly_opened_connections/generated_src/compiled.json', '22222']' returned non-zero exit status 1
Using Thrift port 22222

2020-12-13 10:18:49,614 - P4Target - INFO - start the emitter
Emitter Started
Waiting for socket
*********************************************************************
*                           System Ready                            *
*********************************************************************


Now start sniffing the packets from switch
Interface confirming:  out-veth-2
For 10024 stream processor sending  []  at time 11.6746091843
For 10032 stream processor sending  []  at time 11.6951150894
For 10024 stream processor sending  []  at time 15.6822590828
For 10032 stream processor sending  []  at time 15.6919710636

备注：终端2发送完特定流量后自动结束，结束后输入下述命令，则终端1也会结束;

$ cd ~/dev
$ sudo ./cleanup.sh
//终端1会显示：
Killed
vagrant@vighata:~/dev$

步骤3中运行：sudo ./cleanup.sh出现错误及解决办法

错误：

1 2	vagrant@vighata:~/dev$ sudo ./cleanup.sh sudo: unable to execute ./cleanup.sh: No such file or directory

解决办法：Linux运行shell脚本提示No such file or directory错误的解决办法

cd ~/dev
sudo apt-get install vim//暂时不清楚这一步用不用做
vi cleanup.sh
:set ff //打开之后不要按任何键，直接输入此命令
回车，显示fileformat=dos
:set ff=unix //fileformat=dos在输入此命令时会自动清空
保存退出: 
:wq

备注：好好学一下Linux的Vim吧。

错误：强制从VitualBox中关闭虚拟机，导致下次运行：vagrant ssh时无反映。（解决耗时：2晚上+一上午，重装数次）

解决办法：

1 2	vagrant destroy # 销毁当前虚拟机 vagrant up # 启动虚拟机

备注：

1.学习一下VitualBox和vagrant以及Xming、Putty的基本功能、命令；

2.操作完成之后要关闭虚拟机必须使用命令：vagrant suspend，不要在VitualBox中操作；vagrant + virtualBox模式: windows开发 - 文件本地共享 — 直接访问虚拟机的方式；(vagrant相关命令)；

vagrant up （启动虚拟机）
vagrant halt （关闭虚拟机——对应就是关机）
vagrant suspend （暂停虚拟机——只是暂停，虚拟机内存等信息将以状态文件的方式保存在本地，可以执行恢复操作后继续使用）
vagrant resume （恢复虚拟机—— 与前面的暂停相对应）
vagrant destroy （删除虚拟机，删除后在当前虚拟机所做进行的除开Vagrantfile中的配置都不会保留）
vagrant destroy 命令实际上并不删除下载的 box 文件。可以使用 vagrant box remove 命令彻底删除 box 文件。

3.虚拟机的位置：C:\Users\yuan\VirtualBox VMs\SONATA-DEV_default_1607914797886_56231

4.在VitualBox中点击：显示(绿箭头)，等待片刻（5min），会出现登录界面；账号，密码均是：vagrant；登录之后会出现：vagrant@vighata:~$这应该就是进入虚拟机的终端里了；（！！！除非你想毁掉虚拟机重新配置，否则不要在VitualBox中做任何操作！！！）

5.vagrant halt命令关机之后在VitualBox中点击：显示，无法启动。具体原因不清楚。

6.经测试，在git中运行vagrant suspend关闭虚拟机后，再运行vagrant resume可以正常启动虚拟机；

8.上述问题均可通过销毁再重新启动的方式解决；

9.初步推测上述问题和SSH有关；

vagrant-安装教程及常见问题