splunk

Reference

• Release and maintain Splunk apps for Splunk Cloud or Splunk Enterprise

add-on design and name convention

• About the ES solution architecture
  • SA-: Supporting add-ons
  • DA-: Domain add-ons
  • TA-: Technology add-ons
• Planning your integration for ES

列出一些，目前使用 splunk 有用到的方式

一些有找到，還滿好用的有關 splunk 的 project

• splunk/splunk-app-examples - App examples for Splunk Enterprise
  • 可以參考這之中的 docker-compose.yml 用法
• hire-vladimir/SA-cim_vladiator - Data validator agains Splunk Common Information Model (CIM)
  • 用來 validate CIM 滿好用的
• Kintyre/ksconf - Kintyre’s Splunk Configuration tool
  • 網路上找到，可用來管理 splunk conf 的工具，僅看過說明，還沒有嘗試過

有用到的 vim plugin

• yorokobi/vim-splunk - Syntax highlighting for Splunk .conf files

Reference

• Splunk Field(s) of Dreams: How-to Create Calculated Fields and Aliases
• A Beginner’s Guide to Regular Expressions in Splunk

• 列出目前系統中有的 CIM

  • 於 web 中 search | datamodelsimple type=models

• 確定要符合那種 data model

  • 可由 “Splunk Common Information Model” App 來看
  • http://localhost:8000/en-US/app/Splunk_SA_CIM/cim_setup?action=edit
  • https://docs.splunk.com/Documentation/CIM/4.18.0/User/Overview => 也可由這邊選 data models

• 接著設定 data model

  • 由 Settings -> Data models -> 選對應的 data model
  • 進行 tag 的設定
    • 由 Settings -> Event Types 進行新增
    • 透過設定檔, app/default/tags.conf

• 建立需要的 lookup table

  • 可以由 | inputlookup lookup_defintion 可以看到 lookup 的定義
  • 定義 lookup table
  • 設定 lookup 欄位定義

• 設定 lookup 欄位的步驟

  • 建立 & copy csv 檔案
    • 檔案放置於, app/lookup 目錄中
  • 定義 lookup define 資料
    • 編輯檔案，app/default/transforms.conf
  • 使用 lookup 來 alias 欄位
    • 編輯檔案，app/default/props.conf

• regex

  • 可以由 regex101 (https://regex101.com/ ) 這個 site 來測試

• 進行 CIM validate 的工具

  • SA-cim_validator
  • 安裝時，無法由 splunk base 進行安裝。自行 copy to $SPLUNK/etc/apps 目錄中，手動安裝
    • hire-vladimir/SA-cim_vladiator

一些筆記

• Two Knowledge Objects included in the CIM

  • Tags
  • Fields

• Process for Tags

  • Identity
    • identity the data model and dataset revelant to your event
  • Observe
    • Which Tags are required for the dataset ?
    • Are there any constraints ?
  • Apply
    • create event types and tag them with the appropriate tags
  • Verify
    • verify the tags work as expected using Pivot tool

• Process of Fields

  • Compare the fileds in the data model with the fields in your data
  • Map your fields name to the CIM filed names
    • Aliases
    • Extractions
    • Lookups

• Two Way to validate your data

  • using the Pivot editor
  • searching the data model itself

• Six steps in the data normalization pipeline

  • get data in
  • examine data
  • tag events
  • verify tags
  • normalize field
  • validate against data model

Reference

• Splunk Common Information Model (CIM) - splunk app
  • Common Information Model Add-on Manual

列出使用 Splunk Packaging Toolkit 的記錄

cmd document

• Packaging Toolkit CLI
• install

cmd list

• cmd: slim, slim <command> --help
• slim generate-manifest
• slim package
  • slim package [-h] [--debug] [--quiet] [-o <output-dir>] [-r <repository>] [-u <level>] <app-source>
• slim validate

Reference

• Splunk App Lifecycle Management - with More Peace, Love and Rock-n-Roll! - pdf, 包&測試 splunk app 的說明
• Splunk App Lifecycle Management - Take Control of Your Apps in the Cloud! - pdf
• Package apps for Splunk Cloud or Splunk Enterprise using the Splunk Packaging Toolkit

列出，常用的 splunk cmd 作為參考用

command

• 手動 restart splunkWeb 用

  • $SPLUNK_HOME/bin/splunk restart splunkweb -auth <USER>:<PASSWORD>
  • Is there a configuration to restart Splunk web without asking for credentials?

• remove index 用

  • Remove indexes and indexed data
  • help command, splunk help clean
  • remove all index, splunk clean eventdata
  • remove single index, splunk clean eventdata -index <index_name>

• 看 splunk log 用

  • Is there a way to search for all Splunk error messages?
    • index=_internal source="*/splunkd.log" | ...

記錄使用 email 來收 splunk alert 時，遇到的情形

修改 mail notificition 的相關設定

• 修改 from email
  • How to change the “FROM” address when alert emails are sent?
  • 編輯檔案 /opt/splunk/etc/system/local/alert_actions.conf add from 變數
• 修改信件中的 hostname
  • Is there a way to modify alert url $results.url$ in email alerts?
  • 編輯檔案 /opt/splunk/etc/system/local/alert_actions.conf add hostname 變數

Trouble Shooting

• Why is my triggered alert email not sending?
  • 直接來去看 log , log path: ~/splunk/var/log/splunk/python.log

最近在測試使用 splunk app，要 trouble shooting 需要花些功夫

splunk log path

• 在 $SPLUNK_HOME/var/log/splunk folder 中，有需要的 log folder
  • for example /opt/splunk/var/log/splunk, linux 中的 path

可以 trouble shooting 的方式

• 使用 query 的方式

  1	index=_internal " error " NOT debug source=*splunkd.log* earliest=-24h latest=now

記錄一下，要開發 splunk app 時，需要閱讀的資料 & 閱讀的記錄

Visualize data in a Splunk app for Splunk Cloud or Splunk Enterprise

開發工具

• Splunk>Blogs: Splunk Enterprise + Visual Studio Code = Better Together * 要來試試看

一些可以先事前看看的文件

• Before you create a Splunk app
• Anatomy of a Splunk app in Splunk Cloud or Splunk Enterprise * 可以清楚的了解 app 的架構，重要
• Understand file precedence in apps for Splunk Cloud or Splunk Enterprise * 要看檔案如何找時，滿有用的
• Understand asset caching and state changes in Splunk Enterprise
• Start and stop Splunk Enterprise in the Splunk Enterprise Admin Manual

Splunk conf material

• splunk developer 相關的 topic
• DEV1722C - Publishing your app on Splunkbase
  • session video, session slides
• FN1390 - App Sorcery - Building Splunk Apps With Best Practice
  • session video, session slides
• FN1097 - App Sorcery 2: Building Better Splunk Apps with Best Practice
  • session video, session slides
• DEV1545 - Go From Dashboards to Applications With Ease: SplunkJS and Splunk Python for Non-Developers
  • session video, session slides
• DEV1293 - Worst practices for building Splunk Apps and Add-ons and how to avoid them
  • session video, session slides

publish

• Publish apps for Splunk Cloud or Splunk Enterprise to Splunkbase
• DEV1722C - Publishing your app on Splunkbase
  • session video, session slides

Debug

• DEV1188C - Debugging Splunk Apps, Add-ons, and Configurations using Visual Studio Code
  • session video, session slides

大神的 blog

• David Veuve - Tech Blog
• Splunk Presentations
• Splunk Apps

教學 site

• learnsplunk

Reference

• splunkbase - develop page
• App fundamentals
• Building a Splunk App * 這篇很重要，必讀
• Package a Splunk App - Package apps for Splunk Cloud or Splunk Enterprise using the Splunk Packaging Toolkit

List Splunk AppInspect 的相關資料，當寫好一個 splunk app 時，可以使用 AppInspect 檢查 app 的寫法是否符合 splunk 的 specs

sample command

$ splunk-appinspect inspect --included-tags splunk_appinspect $SPLUNK_HOME/etc/apps/[SPLUNK_APP_NAME]

Reference

• Validate quality of apps or add-ons for Splunk Cloud or Splunk Enterprise with Splunk AppInspect
• Dev Resources