Automating Graylog Pipelines
Part of our job at Recon relies on fine tuning our threat signatures that make up the bulk of our pipeline rules in our Graylog environment.
Because of this, they are constantly changing, growing, being tuned, and ultimately becoming more effective over time at detecting anomalous and malicious activity.
The rules change frequently, sometimes daily, and keeping track of who changed what and when is important. They are in plain text format, which makes them a perfect candidate for GitHub.
We also don't want to have situations where the rules that already live in Graylog don't match the rules sitting in our GitHub repository.
WE NEEDED TO CI OUR RULES.
CI, or continuous integration, takes the output from our GitHub commits, and kicks off a job that we tell it to run. This maintains a level of accountability for who approves and makes changes, gives us revision history, and ensures that what we put in GitHub is now in Graylog.
It also eliminates the manual process of copy/pasting a rule from GitHub into Graylog, and potentially altering it, or even assigning it to the wrong pipelines.
We decided to harness the power of Python and Ansible to build Graylog Ansible modules (wrappers for various pieces of the Graylog API), which are then used to deploy all of our pipeline rule changes to Graylog via CircleCI.
The one we are focused on primarily, in this case, is
graylog_pipelines. This module allows us to create and modify pipelines, and pipeline rules, and we use it whenever we make modifications to the rules in our repository.
LET'S CHECK OUT AN EXAMPLE.
Say I have a pipeline called
threats, and a rule called
check_src_ip in stage 3 of that pipeline that checks source IPs against known bad IP addresses from various threat data feeds.
In this case, the path to this particular rule might be
I can then, within CircleCI (or your tool of choice), kick off a job that runs a script (Bash, Python, you name it) that checks what rules were just added/modified/deleted in GitHub, and loop through that list of rule paths, grabbing the pipeline/rule name/stage from each.
The playbook below, using the
graylog_pipelines module, will validate and create a rule sitting at
ansible-playbook ~/my_repo/.circleci/playbook.yml -e \ "rule_path=pipelines/threats/stage_3/check_src_ip \ rule_pipeline=threats \ rule_stage=3 \ rule_name=check_src_ip \ commit=c26cb8afd03c423679908392f8a6617a7e7922a2"
- name: Get rule info set_fact: pipeline_name: "" stage: "" rule_path: "" rule_name: "" commit: "" username: "" password: "" endpoint: "graylog.mydomain.com" no_log: true - name: Get rule source from file set_fact: rule_source: "" - name: Validate pipeline rule graylog_pipelines: action: parse_rule endpoint: "" graylog_user: "" graylog_password: "" source: "" register: validate_rule - name: Fail if rule does not validate fail: msg: "Rule does not validate" when: validate_rule.status != 200 - name: Create pipeline rule graylog_pipelines: action: create_rule endpoint: "" graylog_user: "" graylog_password: "" title: "" description: "Created , Commit ID " source: "" register: pipeline_rule - name: Get pipeline ID from pipeline name graylog_pipelines: action: query_pipelines endpoint: "" graylog_user: "" graylog_password: "" pipeline_name: "" register: pipeline - name: Get pipeline graylog_pipelines: action: list endpoint: "" graylog_user: "" graylog_password: "" pipeline_id: "" register: pipeline - name: Create file with pipeline source copy: content: "" dest: ./pipeline - name: Make sure stage is in pipeline source lineinfile: path: ./pipeline regexp: '^stage ' line: 'stage match either' insertafter: '^pipeline' - name: Add rule to pipeline source lineinfile: path: ./pipeline regexp: '^rule ""' line: 'rule ""' insertafter: '^stage ' - name: Get modified pipeline content set_fact: pipeline_source: "" - name: Update pipeline with newly added rule graylog_pipelines: action: update endpoint: "" graylog_user: "" graylog_password: "" pipeline_id: "" source: ""
This has added a lot of value to our team, and allows us to better streamline an important part of our stack. What was once an unmanaged and completely manual process, is now tracked, validated, and repeatable.
However, these rules can't just be thrown into a pipeline on a whim, especially in a production environment with hundreds/thousands/millions of events coming in.
We still have to do our due diligence in making sure the rules we commit do, in fact, behave how we intend for them to behave. Pull requests are our way of ensuring that rules have had multiple sets of eyeballs on them before they get accepted and merged.
Adding validation steps in our CircleCI jobs and Ansible playbooks is another layer of security before pushing any breaking changes to Graylog.
BUT WAIT! THERE'S MORE.
If you want to learn more about Graylog pipelines and rules, their documentation is fantastic.
Additionally, we are continuously building upon these modules, and other tools, and thrive on giving back to a community that has provided us with so much of what we use and rely on. Everything we've open sourced can be found in our GitHub, @ReconInfoSec.
WANT HANDS ON EXPERIENCE?
1. These modules are currently awaiting being merged into the official Ansible project here.