Cribl Engineer

GovCIO is currently hiring for  Cribl Engineer to support our Administrative Office of the US Courts NLS project. The NLS currently ingest an average of 18-20TB of logging data daily across 60 indexers distributed in 2 data centers. This position is located within the United States and is fully remote.

• Develop apply best practices and tools for data ingestion, indexing, and management to optimize data sources and refine data collection processes to capture only pertinent data.
• Plan and perform Cribl platform upgrades (Leader, Worker, and Edge nodes) following defined change control procedures.
• Manage and optimize the Cribl distributed infrastructure, ensuring scalability, stability, and efficient data routing.
• Continuously monitor Cribl performance, including throughput, queue depth, and worker health metrics.
• Develop and maintain Cribl pipelines for new data sources, implementing filtering, sampling, and enrichment logic.
• Migrate existing Splunk forwarder-based data inputs to Cribl for improved control and flexibility.
• Build and maintain Cribl Packs for standardized configurations across multiple environments.
• Implement data reduction and enhancement workflows to minimize ingestion volume and improve data quality.
• Maintain and enhance Ansible playbooks for automated deployments, configurations, and upgrades.
• Integrate GitOps CI/CD pipelines (e.g., GitLab, Jenkins, Terraform) to manage configuration-as-code for both Splunk and Cribl.
• Develop, test, and review merge requests related to dashboards, alerts, saved searches, and data onboarding pipelines.
• Perform Splunk core upgrades (indexers, search heads, cluster masters, deployers) ensuring backward compatibility and minimal downtime.
• Upgrade and validate Splunk Add-ons and Apps, maintaining functionality and CIM compliance.
• Develop and maintain custom props, transforms, eventtypes, and lookups to normalize data consistently.
• Ensure CIM compliance for all add-ons and sourcetypes used across the platform.
• Handle escalations from Operations and perform deep-dive troubleshooting on ingestion, parsing, or performance issues.
• Perform break/fix analysis on Splunk core services such as KVStore, clustering, deployment server, and scheduler.
• Conduct performance tuning for search optimization, bucket management, and scheduler balancing across SHC.
• Design and maintain retention, archival, and index management strategies to align with business and compliance goals.
• Manage license allocation, volume forecasting, and capacity planning across indexer clusters.
• Develop and maintain monitoring and alerting integrations for Cribl and Splunk infrastructure health.
• Collaborate with Operations on incident triage, root cause analysis, and postmortem documentation.
• Create and maintain runbooks and engineering guides for deployments, upgrades, and troubleshooting.
• Participate in architecture and design discussions to ensure Splunk and Cribl meet enterprise scaling and reliability needs.
• Implement security and compliance controls including token rotation, TLS configurations, and secret management via Vault or GCP Secret Manager.
• Perform disaster recovery testing and validate replication and failover processes across clusters.
• Collaborate with governance teams to align on data retention, anonymization, and privacy requirements.
• Support continuous improvement by analyzing ingestion efficiency, performance benchmarks, and automation opportunities.
• Lead knowledge-sharing sessions and technical handoffs with Operations for newly deployed features or pipelines.

Bachelor's with 8+ years (or commensurate experience)

Required Skills and Experience

• 3 years of experience with Cribl data engine
• Understanding of Data from a Security Perspective
  
  

Clearance Required:  Must be able to attain and maintain AOUSC Public Trust

Preferred Skills and Experience

• Cribl Admin Cert
• Experience in a Splunk Environment
• Cloud Experience