[bug] Data Normalization - TypeError: unhashable type: 'dict'

[chunyong-lin]

Background

If you have AWS SecurityHub service enabled in your AWS account, it will generate cloudwatch events and it may trigger exception with TypeError: unhashable type: 'dict' error due to StreamAlert normalizes source field into sourceAddress while the source field in SecurityHub events containing a dictionary eventually. See the normalization conf here.

            "source": {
                "owner": "AWS",
                "sourceIdentifier": "RESTRICTED_INCOMING_TRAFFIC"
            },

The walkaround is remove source key in conf/normalized_types.json.

The long term solution is to handle this exception. Other CloudWatch events may contain source address in source field.

For your reference, here is a CloudWatch event generated by SecurityHub

{
    "eventID": "79fc6769-d50c-4947-8573-4790f0f24395",
    "responseElements": null,
    "sharedEventID": "",
    "eventType": "AwsApiCall",
    "streamalert:envelope_keys": {
        "owner": "123456789012",
        "logStream": "123456789012_CloudTrail_us-east-1",
        "subscriptionFilters": [
            "cloudtrail_delivery"
        ],
        "messageType": "DATA_MESSAGE",
        "logGroup": "CloudTrail/DefaultLogGroup"
    },
    "errorCode": "",
    "eventSource": "config.amazonaws.com",
    "requestID": "fc51187c-241f-11e9-97ca-bb002a57e007",
    "userAgent": "securityhub.amazonaws.com",
    "recipientAccountId": "123456789012",
    "eventName": "PutConfigRule",
    "eventTime": "2019-01-29T23:45:40Z",
    "serviceEventDetails": [],
    "resources": [],
    "eventVersion": "1.05",
    "additionalEventData": [],
    "apiVersion": "",
    "errorMessage": "",
    "readOnly": false,
    "requestParameters": {
        "configRule": {
            "configRuleState": "ACTIVE",
            "description": "Checks whether the incoming RDP traffic is Allowed from 0.0.0.0/0. This rule is compliant when incoming RDP traffic is restricted.",
            "configRuleName": "securityhub-restricted-rdp-b69qbh",
            "configRuleArn": "arn:aws:config:us-east-1:123456789012:config-rule/aws-service-rule/securityhub.amazonaws.com/config-rule-memyg0",
            "source": {
                "owner": "AWS",
                "sourceIdentifier": "RESTRICTED_INCOMING_TRAFFIC"
            },
            "createdBy": "securityhub.amazonaws.com",
            "inputParameters": "{}",
            "scope": {
                "complianceResourceTypes": [
                    "AWS::EC2::SecurityGroup"
                ]
            },
            "configRuleId": "config-rule-memyg0"
        }
    },
    "awsRegion": "us-east-1",
    "userIdentity": {
        "principalId": "BBBBBBBBBBBBBBBBBBBBB:securityhub",
        "accessKeyId": "ABCDABCDABCDABCDABCD",
        "invokedBy": "securityhub.amazonaws.com",
        "sessionContext": {
            "sessionIssuer": {
                "userName": "AWSServiceRoleForSecurityHub",
                "type": "Role",
                "arn": "arn:aws:iam::123456789012:role/aws-service-role/securityhub.amazonaws.com/AWSServiceRoleForSecurityHub",
                "principalId": "BBBBBBBBBBBBBBBBBBBBB",
                "accountId": "123456789012"
            },
            "attributes": {
                "creationDate": "2019-01-29T23:45:38Z",
                "mfaAuthenticated": "false"
            }
        },
        "type": "AssumedRole",
        "arn": "arn:aws:sts::123456789012:assumed-role/AWSServiceRoleForSecurityHub/securityhub",
        "accountId": "123456789012"
    },
    "vpcEndpointId": "",
    "sourceIPAddress": "securityhub.amazonaws.com",
    "managementEvent": false
}

The traceback is

Traceback (most recent call last):
  File "test_kinesis_events.py", line 27, in <module>
    main()
  File "/Users/chunyong_lin/Work/virtual_environments/streamalert/lib/python2.7/site-packages/moto/core/models.py", line 74, in wrapper
    result = func(*args, **kwargs)
  File "/Users/chunyong_lin/Work/virtual_environments/streamalert/lib/python2.7/site-packages/mock/mock.py", line 1724, in _inner
    return f(*args, **kw)
  File "/Users/chunyong_lin/Work/virtual_environments/streamalert/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "test_kinesis_events.py", line 23, in main
    Classifier().run(data.get('Records', []))
  File "/Users/chunyong_lin/Work/streamalert/stream_alert/classifier/classifier.py", line 244, in run
    self._classify_payload(payload)
  File "/Users/chunyong_lin/Work/streamalert/stream_alert/classifier/classifier.py", line 184, in _classify_payload
    Normalizer.normalize(parsed_rec, record.log_type)
  File "/Users/chunyong_lin/Work/streamalert/stream_alert/shared/normalize.py", line 105, in normalize
    record.update({cls.NORMALIZATION_KEY: cls.match_types(record, log_normalized_types)})
  File "/Users/chunyong_lin/Work/streamalert/stream_alert/shared/normalize.py", line 62, in match_types
    for key, keys_to_normalize in normalized_types.iteritems()
  File "/Users/chunyong_lin/Work/streamalert/stream_alert/shared/normalize.py", line 62, in <dictcomp>
    for key, keys_to_normalize in normalized_types.iteritems()
TypeError: unhashable type: 'dict'

Desired Change

Handle TypeError due to the dictionary in the normalized field correctly.

[0xdabbad00]

I hit the same bug but with Macie for recipientAccountId from https://github.com/airbnb/streamalert/blob/master/conf/normalized_types.json#L89 due to Macie using the format:

"recipientAccoundId": { "123456789012": 431 }

(I have no idea what that 431 means)

See example Macie event at https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#macie-event-types

[Ryxias]

Resolved #1031