3.3.5. 脚本定制

因为业务千变万化, TH-Nebula并不能预知到所有的业务场景, 所以需要自定义化, 可以对一个接口写一个, 甚至多个脚本去触发策略.

如何定制TH-Nebula脚本？

TH-Nebula脚本的总体流程

概要:

  1. 首先取得所需要监控网站的接口以及页面(page)
  2. 对接口的参数需要收集起来,并对参数要明确了解
  3. 对接口的类型分类为所属的策略, 根据接口参数所提供的, 在定制脚本上定制相应的类,也就是处理的函数,分配到策略上, 就可以在触发事件的时候, 通过处理函数,将事件分配到策略上, 在详情页面拿到所定制的流量数据.

根据之前的例子, 新建了一个事件叫做账号-实名验证的策略, 那么它的类型如图所示它的类型为: ACCOUNT_CERTIFICATION

22.脚本定制2

已经定制好策略, 然后在脚本中定义此策略相应的类, 以下是测试的类
例如在/etc/nebula/sniffer/sniffer.conf中配置脚本名, 以下testparser即是脚本名, 脚本的所在位置在于TH-Nebula_sniffer项目:

sniffer.conf位置:

  1. sources: [eth0]
  2. en0:
  3.     driver: bro
  4.     interface: en0
  5.     ports: [80, 81, 1080, 3128, 8000, 8080, 8888, 9001, 8081-8083]
  6.     start_port: 48880
  7.     instances: 1
  8.     parser:
  9.         name: test
  10.         module: testparser

testparser.py位置:

配置：

  1. #!/usr/bin/env python
  2. # -*- coding: utf-8 -*-
  3. import re
  5. import logging
  7. from threathunter_common.util import millis_now
  8. from threathunter_common.event import Event
  10. from ..parser import Parser, extract_common_properties, extract_http_log_event
  11. from ..parserutil import extract_value_from_body, get_md5, get_json_obj
  12. from ..msg import HttpMsg
  14. logger = logging.getLogger("sniffer.parser.{}".format("testparser"))
  16. def arg_from_get(data):
  17.     d = data.split('?')[1].split('&')
  18.     result = {}
  19.     for e in d:
  20.         a, b = e.split('=')
  21.         result[a] = b
  22.     return result
  24. def account_test(httpmsg):
  25.     if not isinstance(httpmsg, HttpMsg):
  26.         return
  27.     if "/loign" not in httpmsg.uri:
  28.         return
  29.     data = httpmsg.get_dict()    
  30.     data = arg_from_get(data['uri'])
  32.     properties["result"] = "T"
  33.     properties["new_password"] = data.get('new_password')
  34.     properties["old_password"] = data.get('old_password')
  35.     properties["register_realname"] = ""
  36.     properties["register_channel"] = "not"
  37.     properties["email"] = data.get('email')
  38.     properties["user_name"] = extract_value_from_body(r_mobile_pattern, httpmsg.req_body)
  39.     properties["password"] = data.get('password')
  40.     properties["captcha"] = ""
  41.     properties["register_verification_token"] = "none"
  42.     properties["register_verification_token_type"] = "null"
  43.     return Event("nebula", "ACCOUNT_CERTIFICATION", "", millis_now(), properties)
  46. #############Parser############################
  47. class TestParser(Parser):
  48.     def __init__(self):
  49.         super(TestParser, self).__init__()
  50.         self.http_msg_parsers = [
  51.             account_test,
  52.         ]
  54.     def name(self):
  55.         return "test customparsers"
  57.     def get_logbody_config(self):
  58.         return ["login", "user"]
  60.     def get_events_from_http_msg(self, http_msg):
  61.         if not http_msg:
  62.             return []
  64.         result = list()
  65.         for p in self.http_msg_parsers:
  66.             try:
  67.                 ev = p(http_msg)
  68.                 if ev:
  69.                     result.append(ev)
  70.             except:
  71.                 logger.debug("fail to parse with {}".format(p))
  73.         return result
  75.     def get_events_from_text_msg(self, text_msg):
  76.         return []
  78.     def filter(self, msg):
  79.             return False
  81.         return False

要注意最后return中的Event第二个参数是: ACCOUNT_CERTIFICATION
这个参数决定了触发了策略的类型, 系统会根据这个类型去匹配策略, 如果触发了策略, 那么会根据策略的处理流程处理.

httpmsg是TH-Nebula系统中的sniffer定义的请求所包含的数据, 设备信息, IP等等所有能抓到的所有信息

注意请求字段 在req_body字段中,可以以此找到所有请求参数,这是定制脚本重点

  1. {
  2.     'uid': '',
  3.     'status_code': 404,
  4.     'resp_content_type': 'text/plain;charset=utf-8',
  5.     'resp_headers': {
  6.         'CONTENT-LENGTH': '356',
  7.         'VARY': 'Accept-Encoding',
  8.         'SERVER': 'openresty',
  9.         'CONNECTION': 'close',
  10.         'DATE': 'Fri,26Oct201808:13:52GMT',
  11.         'CONTENT-TYPE': 'text/plain;charset=utf-8'
  12.     },
  13.     'id': ObjectId('5bd2ccc012a365682aee1fbd'),
  14.     'dest_port': 9001,
  15.     'resp_body_len': 356,
  16.     'log_body': False,
  17.     'resp_body': 'Traceback(mostrecentcalllast):\nFile"/home/threathunter/nebula/nebula_web/venv/lib/python2.7/site-packages/tornado/web.py",line1422,in_execute\nresult=self.prepare()\nFile"/home/threathunter/nebula/nebula_web/venv/lib/python2.7/site-packages/tornado/web.py",line2149,inprepare\nraiseHTTPError(self._status_code)\nHTTPError:HTTP404:NotFound\n',
  18.     'req_content_type': 'application/json',
  19.     'debug_processing': False,
  20.     'req_headers': {
  21.         'CONTENT-LENGTH': '78',
  22.         'ACCEPT-ENCODING': 'gzip,deflate',
  23.         'HOST': '112.74.58.210:9001',
  24.         'ACCEPT': '*/*',
  25.         'USER-AGENT': 'python-requests/2.11.1',
  26.         'CONNECTION': 'close',
  27.         'COOKIE': 'auth=2|1:0|10:1540368603|4:auth|44:NGJlZTQwMDI0NTExYjM2NDVkNjkzOTM1ZTJmMDllMWY=|34718dcbcac603ee1a38daaa0a05fbafc524873af0fafb0013077a53a28b2d4b;group_id=2|1:0|10:1540368603|8:group_id|4:Mg==|30789028a7e8399f5f5ef115fc050e1b3424df25b966755be7554e0048dd29a4;user=weihong;user_id=2|1:0|10:1540368603|7:user_id|4:Mg==|141555b29c711f95ddebea3987820fb90c30a48f506eae9f8cee9b334372ae79',
  28.         'CONTENT-TYPE': 'application/json'
  29.     },
  30.     'method': 'POST',
  31.      // 这里需要注意 req_body 字段值
  32.      'req_body': '{"password":"777777777","order":"999999999999999999","user":"weihong999"}',
  33.     'req_body_len': 78,
  34.     'host': '112.74.58.210',
  35.     'referer': '',
  36.     'xforward': '',
  37.     'did': '',
  38.     'status_msg': 'notfound',
  39.     'source_ip': '61.141.65.209',
  40.     'uri': '112.74.58.210/history',
  41.     'user_agent': 'python-requests/2.11.1',
  42.     'source_port': 62840,
  44. }