![© 2022 SPLUNK INC.
What Can Admins Accomplish before
Ingest Actions?
Edit props.conf:
[source::/var/log/messages]
TRANSFORMS-null= setnull
Edit transforms.conf:
[setnull]
REGEX = DEBUG
DEST_KEY = queue
FORMAT = nullQueue
Filtering and masking data
involves:
• Memorizing syntax
• Handwriting stanzas
• Expensive iteration
• Editing of many conf files
• Manual deployment Source:
https://www.memesmonkey.com/topic/confused+dog](https://image.slidesharecdn.com/20221102sfbaingestactions-221107222512-b4e916b6/85/SFBA-Usergroup-meeting-November-2-2022-8-320.jpg)
![© 2022 SPLUNK INC.
Changes to outputs.conf
[rfs:s3]
path = s3://data-actions-ingest/data-actions-service-acct/
remote.s3.endpoint = https://s3.us-west-2.amazonaws.com
remote.s3.access_key = key
remote.s3.secret_key = secret
Note:
● If on Standalone (incl. HWF), use the UI!
● If on IDXC, use the UI on the CM or SH!
● If on DS, no UI yet (9.1), must configure HWF
directly
● Much of SmartStore’s configuration from
indexes.conf works in this stanza](https://image.slidesharecdn.com/20221102sfbaingestactions-221107222512-b4e916b6/85/SFBA-Usergroup-meeting-November-2-2022-18-320.jpg)
![© 2022 SPLUNK INC.
Configuring Metrics
# transforms.conf
[_ruleset:global_settings]
metrics.disabled = false
metrics.report_interval = 30s
metrics.rule_filter = *<your rule name>*
Turned off by default](https://image.slidesharecdn.com/20221102sfbaingestactions-221107222512-b4e916b6/85/SFBA-Usergroup-meeting-November-2-2022-23-320.jpg)
SFBA Usergroup meeting November 2, 2022
- 1. During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates sampled on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product Forward- Looking Statements © 2022 SPLUNK INC.
- 2. © 2022 SPLUNK INC. Introducing Ingest Actions: Filter, Mask, Route, Repeat San Francisco Bay Area Splunk User Group Nov 2, 2022 Divya Vijayan Software Engineer | Splunk Inc. Samat Jain Principal Software Engineer | Splunk Inc.
- 3. © 2022 SPLUNK INC. Software Engineer | Splunk Inc. Divya Vijayan Principal Software Engineer | Splunk Inc. Samat Jain
- 4. © 2022 SPLUNK INC. Thanks to… Senior Product Manager | Splunk Inc. Felix Jiang Cloud Solutions Architect | Splunk Inc. Russell Uman Product Management Director | Splunk Inc. Izzy Park
- 5. © 2022 SPLUNK INC. $ Value of Data Age of Data Potential Splunk Use Cases Common Splunk Use Cases Real and Near-Real Time Ad Hoc Data Lake and Archive Forensics, Summaries, and Data Retention Compliance <1 sec <1 min <10 sec 1 week 1 year 1 month 1 hour 1 day 10 year Data Value Changes With Age
- 6. © 2022 SPLUNK INC. Data Optimization Brings a Value-Based Approach to Data Strategy TIER A TIER B TIER C Higher Value Low Volume Low Value High Volume Use Cases Monitoring, Investigation, RCA, Premium Solutions Value / Volume Ratio High Value, Med-High Volume Use Cases Troubleshooting, Forensic Investigation, Forensic Analysis Value / Volume Ratio Low Value, Med-High Volume Use Cases Compliance, Future Proofing Value / Volume Ratio Low Value, High Volume B U S I N E S S C R I T I C A L L O W S I G N A L L O W V A L U E
- 7. © 2022 SPLUNK INC. Data Tiering Flows - GDI HF / IDX IDXC DDAS Flex Index S3: RFS S3: DDSS (Frozen) DDAA Syslog Frozen Hadoop Splunk INGEST INDEX ARCHIVE
- 8. © 2022 SPLUNK INC. What Can Admins Accomplish before Ingest Actions? Edit props.conf: [source::/var/log/messages] TRANSFORMS-null= setnull Edit transforms.conf: [setnull] REGEX = DEBUG DEST_KEY = queue FORMAT = nullQueue Filtering and masking data involves: • Memorizing syntax • Handwriting stanzas • Expensive iteration • Editing of many conf files • Manual deployment Source: https://www.memesmonkey.com/topic/confused+dog
- 9. © 2022 SPLUNK INC. A new user interface and backend enhancements to enable admins to easily author and deploy rules on existing Splunk Enterprise-derived infrastructure. This means you can now: • Filter: discard unwanted events – Remove noisy events, DEBUG logs, etc • Mask: change the contents of events – Mask PII, IP addresses, usernames • Route: Events can be routed to any combination of original Splunk index, different Splunk index, clone, or sent to Amazon S3 • Use the UI to preview and validate rules / logic – Does my regex work? – How did one rule interact with others? How Do Ingest Actions Achieve This?
- 10. © 2022 SPLUNK INC. Save Time, Save $ Less iteration time between authoring and deployment in prod Filtering and routing events do not count against the ingest license meter Why? • We heard you loud and clear • In the long term, we still want to help you operate and derive value on your most mission-critical data
- 11. © 2022 SPLUNK INC. IA enables masking with PCRE regex compatibility For audit & compliance contexts, store unmasked data on S3 for compliance, but mask and de-identify for everyday search and reporting Address Compliance Related Use Cases with Ingest Actions
- 12. © 2022 SPLUNK INC. Platform and Licensing Support Deployment Customer-Managed (Splunk® Enterprise) Splunk-Managed (Splunk® Cloud) Licensing - Ingest - vCPU *No new SKU required for IA - Ingest - SVC *No new SKU required for IA Stack - N/A - Upgraded (“Victoria”) Stacks - Classic Stacks (excluding GCP and FedRAMP in 8.2.2203) Platform Tier - Forwarding Tier: Deployment Server → Heavyweight Forwarder via app distribution to all clients - Indexing Tier: Cluster Manager → Indexers via cluster bundle push - Indexing Tier: Rules deployed via Splunk Cloud Platform internal mechanisms UI location - Forwarding Tier: Deployment Server - Indexing Tier: Cluster Manager - Indexing Tier: Search Head
- 13. © 2022 SPLUNK INC. Demo!
- 14. © 2022 SPLUNK INC. Demo Architecture Splunk cloud Search Head Self-Managed Forwarding Tier
- 15. © 2022 SPLUNK INC. What Do Rulesets Look Like?
- 16. © 2022 SPLUNK INC. Where is IA configuration written? DS $SPLUNK_HOME/etc/ deployment-apps/ splunk_ingest_actions Standalone (incl. HWF) $SPLUNK_HOME/etc/ apps/ splunk_ingest_actions SH, CM $SPLUNK_HOME/etc/ manager-apps/ splunk_ingest_actions
- 17. © 2022 SPLUNK INC. Changes to props.conf and transforms.conf props.conf o RULESET-* Works the same as TRANSFORMS-* class, but will run transforms on cooked data o RULESET_DESC-* Description of ruleset transforms.conf o STOP_PROCESSING_IF Used for certain types of rules to conditionally stop processing subsequent rules o Basically, routing rules will have events “exit early” STOP_PROCESSING_IF = <evaluator expression> * An evaluator expression that the regexreplacement processor uses to determine whether or not further processing is to occur for this event. * If you set STOP_PROCESSING_IF, and the regexreplacement processor evaluates the expression that you supply to be true, then the processor stops further processing of this event. * When you set STOP_PROCESSING_IF, like INGEST_EVAL, this setting overrides all of the other index-time settings (such as REGEX, DEST_KEY, etc) except for INGEST_EVAL. STOP_PROCESSING_IF executes after INGEST_EVAL.
- 18. © 2022 SPLUNK INC. Changes to outputs.conf [rfs:s3] path = s3://data-actions-ingest/data-actions-service-acct/ remote.s3.endpoint = https://s3.us-west-2.amazonaws.com remote.s3.access_key = key remote.s3.secret_key = secret Note: ● If on Standalone (incl. HWF), use the UI! ● If on IDXC, use the UI on the CM or SH! ● If on DS, no UI yet (9.1), must configure HWF directly ● Much of SmartStore’s configuration from indexes.conf works in this stanza
- 19. © 2022 SPLUNK INC. Where do rulesets execute? Ingest Action Rulesets are executed after existing transforms, e.g. TAs Universal Forwarder Indexer Heavy Forwarder Indexer Universal Forwarder Unparsed Data Unparsed Data Parsed Data ● Parsing ● Merging ● Typing ○ TRANSFORMS ○ RULESET ● Ruleset ○ RULESET ● Parsing ● Merging ● Typing ○ TRANSFORMS ○ RULESET Parse Boundary
- 20. © 2022 SPLUNK INC. Samat: Updated Masa
- 21. © 2022 SPLUNK INC. What’s on S3?
- 22. © 2022 SPLUNK INC. File Format is Valid JSON, “HEC JSON”
- 23. © 2022 SPLUNK INC. Configuring Metrics # transforms.conf [_ruleset:global_settings] metrics.disabled = false metrics.report_interval = 30s metrics.rule_filter = *<your rule name>* Turned off by default
- 24. © 2022 SPLUNK INC. What metrics are logged Metrics - group=transforms, name=typing, rule="_rule:ruleset_splunkd_ui_access:mask:m7yeuix8", sourcetype="splunkd", hit=216, cpu_seconds=0.1 in=38426, out.splunk=38000, out.drop=426 ● rule is the name of the rule, and can be mapped to a rule in an Ingest Action ruleset ● hit is the times the rule is hit in the report interval (number of events) ● cpu_seconds is the cpu time spent by the rule during the report interval ● in is the raw bytes the rule processes in the report interval ● out.x is the raw bytes the rule routes to each destination ● Ingest Action rules have a special prefix _rule ○ _rule:ruleset_splunkd_ui_access:mask:m7yeuix8 New
- 25. © 2022 SPLUNK INC. Key new concepts for users who already have experience with props/transforms, pipelines ● In Ingest Actions rulesets only, filtering and routing rules will stop further processing on events, by default & by design ● A new pipeline “ruleset” was added ○ The pipeline will accept “cooked” data from HFs (by design) ○ This also means there’s another queue to monitor, previously only needed to mntiro ● A new output “rfs” pipeline was created for S3 ○ Using output to S3 is not immune to issues such as backpressure https://confluence.splunk.com/display/PROD/Data+Actions+Performance+Plan
- 26. © 2022 SPLUNK INC. How Do I Get Started? (1) Capability prereqs: • list_ingest_ruleset: list existing rulesets • edit_ingest_ruleset: create / edit rulesets *Admins get these capabilities automatically (2) Create your first ruleset!
- 27. © 2022 SPLUNK INC. New Since .conf’s 9.0 ● “Set Index”: Route events to different Splunk indexes ● Health Report for S3 destinations
- 28. Thank You © 2022 SPLUNK INC.
- 29. © 2022 SPLUNK INC. Additional Resources 1. What is a Victoria Cloud Stack? 2. Monitoring vCPU Consumption 3. Monitoring SVC Consumption 4. Using Ingest Actions to improve the data input process