OSMC 2015: The Assimilation Project by Alan Robertson

Painlessly Discovering and MonitoringPainlessly Discovering and Monitoring
Systems, Services and ComplianceSystems, Services and Compliance
#AssimProj @OSSAlanR
http://assimproj.org/
Alan Robertson <alanr@assimilationsystems.com>
Assimilation Systems Limited
http://assimilationsystems.com

© 2015 Assimilation Systems Limited
2
BiographyBiography
● 35+ years in IT/development – 10 years in
system management (SysAdmin)
● Founded Linux-HA project - led 1998-2007 –
aka “Heartbeat” - now called Pacemaker
● Founded Assimilation Project in 2010
● Founded Assimilation Systems Limited in 2013
● Alumnus of Bell Labs, SuSE, IBM

3
Assimilation Project EvolutionAssimilation Project Evolution
● Inspired by 2 million core computer
(cyclops64)
● Concerns for extreme scale
● Topology aware monitoring
● Topology discovery w/out security issues
=►Discovery of everything!

4
A 6-dimensional overviewA 6-dimensional overview
1.System Management Suite Overview
2.Basic Technology
3.Best Practice Analyses
4.Demo
5.Current Status
6.What You Need To Do!

5
Disturbing Trends...Disturbing Trends...
30% of break-ins come through “lost” systems (Verizon)
90% have had failures of unmonitored services (Turnbull)
80% are unable to stay in compliance (Verizon)
30% only start monitoring only after a problem (Turnbull)
30% of systems doing nothing useful (Koomey)

6
More Trends...More Trends...
● Larger sites have trouble scaling monitoring
(Turnbull)
● Larger site admins often don’t know dependencies
● Documentation is incomplete, out of date, expensive

7
You can't manage what you can'tYou can't manage what you can't
see...see...
We give you X-Ray vision into your infrastructure
● Provides insight and details through a graph-model CMDB
● Helps you understand and automate your environment
– Reduce Errors
– Speed up problem resolution
● Reduces Manual Documentation
● CMDB-driven configuration => near-zero configuration
● Automates Monitoring
● Enhances Security
● Designed for Extreme Scale

8
What's in the Suite?What's in the Suite?
● Graph CMDB
● Exception Monitoring
● Security Discovery
● Network Connections

9
Our Unique ValueOur Unique Value
● Security – continuous security compliance
● Scalability – scales like nothing else
● Complexity Management – reduces and
helps you manage complexity

10
ComplexityComplexity
“Complexity is the enemy of reliability”
● Complexity likely your single biggest
problem
– Near-zero configuration reduces complexity
– Tight service integration reduces complexity
– Accurate detailed information provides insights
which help manage complexity

11
Highly Scalable Discovery-DrivenHighly Scalable Discovery-Driven
AutomationAutomation
Continuous Discovery drives everything
● Continuous extensible discovery (CMDB)
– systems, switches, services, dependencies – zero
network footprint discovery process
● Extensible exception monitoring
– more than 100K systems
● Discovery Drives Best Practice Analyses
– Initially concentrating on security
● All data goes into central graph CMDB

12
This all sounds unreasonable...This all sounds unreasonable...
● Huge scalability without complexity?
● Discovery without pings or port scans?
Really?

13
SSimple Scalabilityimple Scalability
I can explain how we scale so your
grandmother would understand...
istockphoto
©bowdenimages

14
Massive Scalability –Massive Scalability – oror
“I see dead servers in“I see dead servers in OO(1) time”(1) time”
● Adding systems does not increase the monitoring work on any system
● Each server monitors 2 (or 4) neighbors
● Each server monitors and discovers its own services
● Ring repair and alerting is O(n) – but a very small amount of work
Current Implementation

15
Minimizing Network FootprintMinimizing Network Footprint
(in our roadmap)(in our roadmap)
● Support diagnosing switch issues
● Minimize network traffic
● Ideal for multi-site arrangements

16
Service Monitoring based on HAService Monitoring based on HA
TechnologiesTechnologies
● Well-proven architecture:
– reliable “no news is good news”
● Implements Open Cluster Framework
standard, LSB and Nagios remote agent APIs
● Each system monitors own services
● Can also start, stop, migrate services

17
How does discovery work?How does discovery work?
Nanoprobe scripts perform discovery
● Each discovers one kind of information
● Can take arguments from environment
● Output JSON
CMA stores Discovery Information
● JSON stored in Neo4j database
● CMA discovery plugins => graph nodes and relationships

18
OS discovery JSON SnippetOS discovery JSON Snippet
{ "nodename": "alanr-1225B",
"operating-system": "GNU/Linux",
"machine": "x86_64",
"processor": "x86_64",
"hardware-platform": "x86_64",
"kernel-name": "Linux",
"kernel-release": "3.8.0-31-generic",
"kernel-version": "#46-Ubuntu SMP ...",
"Distributor ID": "Ubuntu",
"Description": "Ubuntu 13.04",
"Release": "13.04",
"Codename": "raring" }

19
"sshd": {
"exe": "/usr/sbin/sshd",
"cmdline": [ "/usr/sbin/sshd", "-D" ],
"uid": "root",
"gid": "root",
"cwd": "/",
"listenaddrs": {
"0.0.0.0:22": { "proto": "tcp" }
}
sshdsshd ServiceService JSON SnippetJSON Snippet
(from netstat and /proc)(from netstat and /proc)

20
"ssh": {
"exe": "/usr/sbin/ssh",
"cmdline": [ "ssh", "servidor" ],
"uid": "alanr",
"gid": "alanr",
"cwd": "/home/alanr/monitor/src",
"clientaddrs": {"10.10.10.5:22": {"proto": "tcp"}
}
sshssh ClientClient JSON SnippetJSON Snippet
(from netstat and /proc)(from netstat and /proc)

21
Service Dependency GraphService Dependency Graph

22
Switch Discovery GraphSwitch Discovery Graph
from LLDP (or CDP)from LLDP (or CDP)

23
Why a graph database? (Neo4j)Why a graph database? (Neo4j)
● Humans describe systems as graphs
● Dependency & Discovery information: graph
● Speed of graph traversals depends on size of subgraph,
not total graph size
● Root cause queries  graph traversals – notoriously slow
in relational databases
● Visualization is Natural
● Schema-less design: good for constantly changing
heterogeneous environment
● Graph Model === Object Model

24
A Few Canned QueriesA Few Canned Queries
allipports get all port/ip/service/hosts
allswitchports get switch connections
crashed get crashed servers
shutdown get gracefully shutdown servers
downservices get nonworking services
findip get system owning IP
findmac get system owning MAC
unknownips get unknown IP addresses
unmonitored get unmonitored services

25
BBest Practice Analysesest Practice Analyses
Under active development
● Triggered by Discovery Updates
– Analysis occurs within seconds of change
– No change => No analysis
● We can analyze anything discovered
● Expect to create alerts and reports
● SIEM integration

26
Sample Security Best PracticesSample Security Best Practices
● Inappropriate services (telnet, etc)
● Settings in /proc/sys/
● Security Patch Coverage
– OS vendor (RedHat, SuSE, Canonical, etc)
– Application (Oracle, IBM, WordPress, etc)
● Other OS settings
● Common Application Settings
● Looking at best practices
FYI: Collaborating with Lynis project and Linux Foundation

27
Other Sample Security FeaturesOther Sample Security Features
● Discovery of “forgotten” IP addresses
● Monitoring of Open Ports and Services
● Collection of network-facing app checksums
● Nmon profiling of new MAC addresses
● Checksum outliers analysis
● Security Best Practice Analyses

28
IT Best Practices ProjectIT Best Practices Project
ITBestPractices.info
● IT-Bestpractices GitHub project
● Working on Linux Foundation Sponsorship
● Apache 2 License (or similar)
● Initial Sources
– DISA STIGs
– Lynis project
– PCI DSS rules
– Individual contributions

29
IT Best Practices GoalsIT Best Practices Goals
● Make Best Practice rules available in JSON
– Curate mechanically-verifiable practices
– Human-readable descriptions of issues and
remedies
– Multiple language support
– Not limited to security best practices
– Web server under development

30
Sample short descriptionSample short description
The system must limit the ability of processes to
have simultaneous write and execute access to
memory.

31
Sample long descriptionSample long description
ExecShield uses the segmentation feature on all
x86 systems to prevent execution in memory
higher than a certain address. It writes an address
as a limit in the code segment descriptor, to control
where code can be executed, on a per-process
basis. When the kernel places a process's memory
regions such as the stack and heap higher than
this address, the hardware prevents execution in
that address range.

32
Sample Security Rule checkSample Security Rule check
The status of the "kernel.exec-shield" kernel parameter can
be queried by running the following command:
$ sysctl kernel.exec-shield
$ grep kernel.exec-shield /etc/sysctl.conf
The output of the command should indicate a value of "1". If
this value is not the default value, investigate how it could
have been adjusted at runtime, and verify it is not set
improperly in "/etc/sysctl.conf".
If the correct value is not returned, this is a finding.

33
Assimilation /proc/sys RuleAssimilation /proc/sys Rule
Disallow executing code on writable pages
“nist_V-38597”:
{“rule”: “EQ($kernel.exec-shield, 1)”,
“category”: “security”
}

34
Assimilation Networking RuleAssimilation Networking Rule
Buffer bloat prevention
“itbp-0001”:
{“rule”: “IN($kernel.core.default_qdisc,
fq_codel, codel)”,
“category”: “networking”
}

35
DDiscovery / Monitoring / Bestiscovery / Monitoring / Best
Practices DemoPractices Demo
● Demonstrate basic capabilities
– Discovery-driven monitoring configuration
– Discovery-driven 'tripwire-like' checksums
– Monitoring – failures / successes
– Host down notification
– Best Practices
● No configuration was supplied
– everything comes from discovery
http://assimilationsystems.com/90_second_demo/

36
Current StatusCurrent Status
● 1.1.0 release out 3 November 2015
● Way-cool simplified installer!
● Continuous Security Monitoring
– ~50 best practice rules implemented
● Great unit and system tests
● Strongly encrypted communication
● Quite a few discovery methods written
● Extensible Automated Discovery Triggers
● Discovery => Automatic Monitoring + Network-Facing Checksums
● Compatible with Nagios remote monitoring agent API
● REST + Command Line Queries

37
Get Involved!Get Involved!
● Trials! Early Adopters!
● Contributors
– Testers, Continuous Integration
– Best practice experts
– Designers
– Developers (C, Python, Shell, PowerShell, JavaScript)
– Porters (esp Windows)
– Promoters, Publicists, Packagers, etc.

38
Resistance Is Futile!Resistance Is Futile!
These slides: bit.ly/DOSUG0915
Mailing List: bit.ly/AssimML
@OSSAlanR
#assimilation on irc.freenode.net
Project Web Site: assimproj.org
Company Web Site: assimilationsystems.com
Download: assimilationsystems.com/download

39
Risk Management/MitigationRisk Management/Mitigation
● Intrusions
● Vulnerable Software
● Licensed Software
● Audit Risk
● Outages
● System management

40
Monitoring Pros and ConsMonitoring Pros and Cons
Pros
Simple & Scalable
Uniform work distribution
No single point of failure
Distinguishes switch vs
host failure
Easy on LAN, WAN
Multi-tenant approach
Cons
Active agents
Potential slowness
at power-on

41
Sixth Dimension:Sixth Dimension:
Graph SchemaGraph Schema
Two Schema subgraphs
● Client / server
dependency
● Switch interconnect

First DimensionFirst Dimension::
Problems AddressedProblems Addressed
● Discovering and maintaining documentation
(CMDB) using continuous discovery
– Services, Systems, Dependencies, Switches, Interconnects,
Configuration
● Monitoring and alerting: services, systems and
compliance
● Managing compliance
● Mitigating risk

45
Why Discovery? (DevOps)Why Discovery? (DevOps)
● Documentation: incomplete, incorrect
● Dependencies: unknown
● Planning: Needs accurate data
● Best Practices: Verification needs data
● ITIL CMDB (Configuration Management
Data Base)
Our Discovery: continuous, low-profile

46
Second Dimension:Second Dimension:
Unique Powerful FeaturesUnique Powerful Features
1. Continuous Discovery
2. Discovery: Zero network footprint
3. Centralized graph database
4. We know everything that changes
5. Discover and update dependency information
6. Discovery and monitoring tightly integrated –
discovery drives automation

47
(even more) Features...(even more) Features...
7. Discovery and monitoring easily extensible
8. Naturally scalable to > 100K systems
9. Minimal network load
10.Server failures distinguishable from switch failures
11.Best practice and vulnerability alerts
12.Multi-tenant support

48
Third Dimension:Third Dimension:
Fully distributed workFully distributed work
Two philosophical underpinnings
1. Monitoring and Discovery are fully distributed
2. Reliable “no news is good news”
Only responses to changes are centralized

49
Sample /proc/sys RulesSample /proc/sys Rules
“BPC-00002-1”:
{“rule”: “OR(EQ($kernel.core_uses_pid, 1),
NE($kernel.core_pattern, ""))”
“url”: “https://trello.com/c/6LOXeyDD” },
“BPC-00003-1”: {“rule”: “EQ($kernel.ctrl-alt-del, 0)”,
“url”: “https://trello.com/c/aUmn4WFg”},
“BPC-00006-1”: {“rule”: “EQ($kernel.sysrq, 0)”,
“url”: “https://trello.com/c/QSovxhup” },

OSMC 2015: The Assimilation Project by Alan Robertson

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (20)

Similar to OSMC 2015: The Assimilation Project by Alan Robertson

Similar to OSMC 2015: The Assimilation Project by Alan Robertson (20)

Recently uploaded

Recently uploaded (20)

OSMC 2015: The Assimilation Project by Alan Robertson