Skip to content

[hashicorp_vault] Test with Vault 1.11.2 #4036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
andrewkroh merged 2 commits into from
Aug 25, 2022
Merged

[hashicorp_vault] Test with Vault 1.11.2 #4036

andrewkroh merged 2 commits into from
Aug 25, 2022

Conversation

@andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Aug 18, 2022 •
edited
Loading

What does this PR do?

Update tests to use Hashicorp Vault 1.11.2.
Update mappings for Vault 1.11 audit data schema.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues

Logs

1.11 event with new fields.

{
  "agent": {
    "name": "docker-fleet-agent",
    "id": "03109bfa-7015-46bd-9433-3879357210cd",
    "type": "filebeat",
    "ephemeral_id": "c6777a07-0ee5-4efc-b9d5-efe38e545ab5",
    "version": "8.3.2"
  },
  "hashicorp_vault": {
    "audit": {
      "request": {
        "client_token": "hmac-sha256:9670f60ae136c9fb5ec99472291977aa9c61ad4d49829cf4dffabc21d5a05483",
        "path": "auth/token/lookup-self",
        "namespace": {
          "id": "root"
        },
        "remote_port": 46026,
        "id": "6a2dae54-47cc-80ab-43d5-aaa60bc95745",
        "mount_type": "token",
        "remote_address": "172.30.0.4",
        "client_token_accessor": "hmac-sha256:dd2e0395c657ad5a10dfd832e841b0f1d91e0e95911fa58d6ee7fe7e622912c5",
        "operation": "read",
        "mount_accessor": "auth_token_df374312",
        "client_id": "0DHqvq2D77kL2/JTPSZkTMJbkFVmUu0TzMi0jiXcFy8="
      },
      "auth": {
        "token_policies": [
          "root"
        ],
        "client_token": "hmac-sha256:9670f60ae136c9fb5ec99472291977aa9c61ad4d49829cf4dffabc21d5a05483",
        "token_issue_time": "2022-08-18T17:01:47Z",
        "accessor": "hmac-sha256:dd2e0395c657ad5a10dfd832e841b0f1d91e0e95911fa58d6ee7fe7e622912c5",
        "policies": [
          "root"
        ],
        "policy_results": {
          "granting_policies": [
            {
              "namespace_id": "root",
              "name": "root",
              "type": "acl"
            }
          ],
          "allowed": true
        },
        "display_name": "token",
        "token_type": "service"
      },
      "response": {
        "data": {
          "creation_time": 1660842107,
          "creation_ttl": 0,
          "accessor": "hmac-sha256:dd2e0395c657ad5a10dfd832e841b0f1d91e0e95911fa58d6ee7fe7e622912c5",
          "policies": [
            "hmac-sha256:2d6e645cbe7182171f0dc8dafb5959ff1a9f28ffbebf505d98ede04f7360ba92"
          ],
          "expire_time": null,
          "num_uses": 0,
          "display_name": "hmac-sha256:976ba670270142f2507af8db8a08ab80e39fb8e59dd990cd97a94744597d7ae2",
          "entity_id": "hmac-sha256:508ac971ef30d807928dca23750b3dcf6b0b2e251a06b7ca636c4d111877d5c2",
          "orphan": false,
          "type": "hmac-sha256:f00410e9496c6521003b940705e6c8f076614212de1bbbe9867555871d65629e",
          "ttl": 0,
          "explicit_max_ttl": 0,
          "path": "hmac-sha256:80821fa705df690f161f9b5e041f7f5460c827cad881c50b6caafc5f54bbb2ea",
          "meta": null,
          "renewable": false,
          "id": "hmac-sha256:9670f60ae136c9fb5ec99472291977aa9c61ad4d49829cf4dffabc21d5a05483",
          "issue_time": "2022-08-18T17:01:47.694373174Z"
        },
        "mount_type": "token",
        "mount_accessor": "auth_token_df374312"
      },
      "type": "response"
    }
  },
  "log": {
    "file": {
      "path": "/tmp/service_logs/vault/audit.json"
    },
    "offset": 22102
  },
  "elastic_agent": {
    "id": "03109bfa-7015-46bd-9433-3879357210cd",
    "version": "8.3.2",
    "snapshot": false
  },
  "source": {
    "port": 46026,
    "ip": "172.30.0.4"
  },
  "tags": [
    "preserve_original_event",
    "hashicorp-vault-audit"
  ],
  "input": {
    "type": "log"
  },
  "@timestamp": "2022-08-18T17:01:47.754Z",
  "ecs": {
    "version": "8.4.0"
  },
  "related": {
    "ip": [
      "172.30.0.4"
    ]
  },
  "data_stream": {
    "namespace": "ep",
    "type": "logs",
    "dataset": "hashicorp_vault.audit"
  },
  "host": {
    "hostname": "docker-fleet-agent",
    "os": {
      "kernel": "5.10.76-linuxkit",
      "codename": "focal",
      "name": "Ubuntu",
      "family": "debian",
      "type": "linux",
      "version": "20.04.4 LTS (Focal Fossa)",
      "platform": "ubuntu"
    },
    "containerized": false,
    "ip": [
      "172.23.0.7"
    ],
    "name": "docker-fleet-agent",
    "mac": [
      "02:42:ac:17:00:07"
    ],
    "architecture": "x86_64"
  },
  "event": {
    "agent_id_status": "verified",
    "ingested": "2022-08-18T17:02:23Z",
    "original": "{\"time\":\"2022-08-18T17:01:47.754081924Z\",\"type\":\"response\",\"auth\":{\"client_token\":\"hmac-sha256:9670f60ae136c9fb5ec99472291977aa9c61ad4d49829cf4dffabc21d5a05483\",\"accessor\":\"hmac-sha256:dd2e0395c657ad5a10dfd832e841b0f1d91e0e95911fa58d6ee7fe7e622912c5\",\"display_name\":\"token\",\"policies\":[\"root\"],\"token_policies\":[\"root\"],\"policy_results\":{\"allowed\":true,\"granting_policies\":[{\"name\":\"root\",\"namespace_id\":\"root\",\"type\":\"acl\"}]},\"token_type\":\"service\",\"token_issue_time\":\"2022-08-18T17:01:47Z\"},\"request\":{\"id\":\"6a2dae54-47cc-80ab-43d5-aaa60bc95745\",\"client_id\":\"0DHqvq2D77kL2/JTPSZkTMJbkFVmUu0TzMi0jiXcFy8=\",\"operation\":\"read\",\"mount_type\":\"token\",\"mount_accessor\":\"auth_token_df374312\",\"client_token\":\"hmac-sha256:9670f60ae136c9fb5ec99472291977aa9c61ad4d49829cf4dffabc21d5a05483\",\"client_token_accessor\":\"hmac-sha256:dd2e0395c657ad5a10dfd832e841b0f1d91e0e95911fa58d6ee7fe7e622912c5\",\"namespace\":{\"id\":\"root\"},\"path\":\"auth/token/lookup-self\",\"remote_address\":\"172.30.0.4\",\"remote_port\":46026},\"response\":{\"mount_type\":\"token\",\"mount_accessor\":\"auth_token_df374312\",\"data\":{\"accessor\":\"hmac-sha256:dd2e0395c657ad5a10dfd832e841b0f1d91e0e95911fa58d6ee7fe7e622912c5\",\"creation_time\":1660842107,\"creation_ttl\":0,\"display_name\":\"hmac-sha256:976ba670270142f2507af8db8a08ab80e39fb8e59dd990cd97a94744597d7ae2\",\"entity_id\":\"hmac-sha256:508ac971ef30d807928dca23750b3dcf6b0b2e251a06b7ca636c4d111877d5c2\",\"expire_time\":null,\"explicit_max_ttl\":0,\"id\":\"hmac-sha256:9670f60ae136c9fb5ec99472291977aa9c61ad4d49829cf4dffabc21d5a05483\",\"issue_time\":\"2022-08-18T17:01:47.694373174Z\",\"meta\":null,\"num_uses\":0,\"orphan\":false,\"path\":\"hmac-sha256:80821fa705df690f161f9b5e041f7f5460c827cad881c50b6caafc5f54bbb2ea\",\"policies\":[\"hmac-sha256:2d6e645cbe7182171f0dc8dafb5959ff1a9f28ffbebf505d98ede04f7360ba92\"],\"renewable\":false,\"ttl\":0,\"type\":\"hmac-sha256:f00410e9496c6521003b940705e6c8f076614212de1bbbe9867555871d65629e\"}}}",
    "kind": "event",
    "action": "read",
    "id": "6a2dae54-47cc-80ab-43d5-aaa60bc95745",
    "category": [
      "authentication"
    ],
    "type": [
      "access"
    ],
    "dataset": "hashicorp_vault.audit",
    "outcome": "success"
  }
}

@andrewkroh
hashicorp_vault - Test with Vault 1.11.2
569feca 
Update tests to use Hashicorp Vault 1.11.2.
Update mappings for Vault 1.11 audit data schema.

Closes #4035
@andrewkroh andrewkroh marked this pull request as ready for review August 18, 2022 17:16
@andrewkroh andrewkroh requested a review from a team as a code owner August 18, 2022 17:16
@elasticmachine
Copy link

elasticmachine commented Aug 18, 2022

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@andrewkroh andrewkroh mentioned this pull request Aug 18, 2022
Closed
@andrewkroh andrewkroh changed the title hashicorp_vault - Test with Vault 1.11.2 [hashicorp_vault] Test with Vault 1.11.2 Aug 18, 2022
@elasticmachine
Copy link

elasticmachine commented Aug 18, 2022 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-08-24T16:18:23.107+0000

  • Duration: 19 min 0 sec

Test stats 🧪

Test Results
Failed 0
Passed 18
Skipped 0
Total 18

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Aug 18, 2022 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (2/2) 💚
Files 100.0% (3/3) 💚 2.814
Classes 100.0% (3/3) 💚 2.814
Methods 92.593% (25/27) 👍 3.225
Lines 86.124% (180/209) 👎 -4.802
Conditionals 100.0% (0/0) 💚

efd6
efd6 reviewed Aug 22, 2022
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From looking in the Go definitions for the types that serialise these fields, it looks like there are some missing fields.

  • audit.auth.request.replication_cluster: keyword
  • audit.auth.request.client_certification_serial_number: keyword
  • audit.auth.entity_created: boolean

audit.response.auth.policies is missing a type in the existing file, it looks like it should be keyword.

@andrewkroh
Add more audit fields
3756813 
audit.auth.request.replication_cluster
audit.auth.request.client_certificate_serial_number
audit.auth.entity_created

Add `type: keyword` to audit.response.auth.policies.
@andrewkroh
Copy link
Member Author

andrewkroh commented Aug 24, 2022

@efd6 Thank you for spotting those fields. I was mostly programming to the errors (and checking the Go types to make sure I had the type correct), but the generated logs in the system tests didn't have all the fields.

@andrewkroh andrewkroh requested a review from efd6 August 24, 2022 17:14
@andrewkroh andrewkroh merged commit 3875ca2 into elastic:main Aug 25, 2022
@jamiehynds
Copy link

jamiehynds commented Aug 26, 2022

@andrewkroh am I ok to communicate to Hashicorp that we can support v1.11 (when the package is promoted)?

Can we also update on integration docs to reflect support for v1.11?

@andrewkroh
Copy link
Member Author

andrewkroh commented Aug 26, 2022

am I ok to communicate to Hashicorp that we can support v1.11 (when the package is promoted)?

Yes. 👍

Can we also update on integration docs to reflect support for v1.11?

This change did update docs contained in the package to mention 1.11.

@andrewkroh andrewkroh mentioned this pull request Sep 13, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@efd6 efd6 efd6 approved these changes

Assignees

No one assigned

Labels

enhancement New feature or request Integration:hashicorp_vault Hashicorp Vault

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Hashicorp Vault] Support for Vault v1.11

4 participants

@andrewkroh @elasticmachine @jamiehynds @efd6