Tutorial - Custom Combiner Development¶
In the Map-Reduce model, Parsers are responsible for mapping the data from a datasource and Combiners are responsible for reducing the data from multiple Parsers into a reduced dataset. Combiners help to consolidate information from different data sources, hide differences between the same data source across operating system versions, and make Parser output more rule friendly.
For example, the hostname of a system may be obtained from the Hostname
,
Facter
, and SystemID
parsers. A rule could rely on all three parsers
and might need to process each one differently depending upon which was present.
However, a rule could instead simply rely on the hostname
combiner which
does the job of evaluating all of the parser data and determining the best
source of hostname, greatly simplifying logic in the rule.
Another example is the GrubConf
combiner which evaluates multiple versions
(1, 2, non-EFI, EFI) Grub configuration data to provide one consolidated source
of information for Grub configuration on a system.
You can find the complete implementation of the combiner and test code in the
directory insights-core-tutorials/insights_examples/combiners
.
Hostname Combiner¶
Overview¶
The same development environment will be used that was setup at the beginning of the tutorial using the Preparing Your Development Environment section.
For this tutorial we will create a new hostname combiner that will consolidate
information from the insights.parsers.uname.Uname
and
insights.parsers.hostname.Hostname
parsers. There is an existing
insights.combiners.hostname.Hostname
combiner so we will call ours
HostnameUH
to avoid confusion.
Creating the Initial Combiner Files¶
First we need to create the combiner file. Combiner files are implemented in
modules. The module should be limited to one purpose. In this case we are
working with hostname
data so we will create an hostname_uh
module.
Also there is already a hostname
combiner module so we want to avoid
confusion. Create the module file mycomponents/combiners/hostname_uh.py
in the
mycomponents/combiners directory
:
(env)[userone@hostone mycomponents]$ touch combiners/hostname_uh.py
Now edit the file and create the combiner skeleton:
1from insights.core.plugins import combiner
2from insights.parsers.hostname import Hostname
3from insights.parsers.uname import Uname
4
5
6@combiner([Hostname, Uname])
7class HostnameUH(object):
8
9 def __init__(self, hostname, uname):
10 pass
We start by importing the combiner
decorator. As discussed above our
combiner will depend upon the Hostname
and Uname
parsers and these
are imported and included as arguments to the combiner
decorator. Notice
that the decorator arguments are in a list
meaning that our combiner
needs at least one of the parsers in the list. See the discussion of
Rule Decorators for more information on required,
at least one, and optional arguments to the combiner decorator.
We also need to pass the parser instance objects as arguments to the __init__
method of our combiner. If either of these objects is not present then its
value with be None
.
Next we’ll create the combiner test file mycomponents/tests/combiners/test_hostname_uh.py
as a skeleton that will aid in the combiner development process:
1from mycomponents.combiners.hostname_uh import HostnameUH
2
3
4def test_hostname_uh():
5 pass
Once you have created and saved both of these files, you can the test to make sure everything is setup correctly:
(env)[userone@hostone insights-core-tutorials]$ pytest -k hostname_uh
======================= test session starts ==============================
platform linux -- Python 3.6.6, pytest-3.0.6, py-1.7.0, pluggy-0.4.0
rootdir: /home/userone/work/insights-core-tutorials, inifile:
collected 16 items / 14 deselected
insights_examples/tests/combiners/test_hostname_uh.py . [ 50%]
mycomponents/tests/combiners/test_hostname_uh.py .
============ 2 passed, 14 deselected in .27 seconds ====================
When you invoke pytest
with the -k
option it will only run tests
which match the filter, in this case tests that match hostname_uh. So our
test passed as expected.
Hint
You may sometimes see a message that pytest
cannot be found,
or see some other related message that doesn’t make sense. The first
think to check is that you have activated your virtual environment by
executing the command source bin/activate
from the root directory
of your insights-core-tutorials project. You can deactivate the virtual
environment by typing deactivate
. You can find more information about
virtual environments here:
http://docs.python-guide.org/en/latest/dev/virtualenvs/
Combiner Implementation¶
Typically parser and combiner development is driven by rules that need facts generated by the parsers and combiners. Regardless of the specific requirements, it is important (1) to implement basic functionality by getting the raw data into a usable format, and (2) to not overdo the implementation because we can’t anticipate every use of the combiner output. In our example the output is simple, but some combiners can be complicated so keep these two criteria in mind when developing new parsers or combiners. You can always add more capability later on if needed by your rules.
Test Code¶
We will start by creating a test for the output that we want from our combiner
using the two input sources. You can look at the documentation for
insights.parsers.hostname
and insights.parsers.uname
to see
what methods will be available. In our tests we want to ensure that we can
test with the parser object so we’ll use input data to feed the parsers and
then use the parsers as input to our combiner tests.
1from mycomponents.combiners.hostname_uh import HostnameUH
2from insights.parsers.hostname import Hostname
3from insights.parsers.uname import Uname
4from insights.tests import context_wrap
5
6HOSTNAME = "hostone_h.example.com"
7UNAME = "Linux hostone_u.example.com 3.10.0-693.21.1.el7.x86_64 #1 SMP Fri Feb 23 18:54:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux"
8
9
10def test_hostname_uh():
11 hostname = Hostname(context_wrap(HOSTNAME))
12 uname = Uname(context_wrap(UNAME))
13
14 hostname_uh = HostnameUH(hostname, None)
15 assert hostname_uh.hostname == HOSTNAME
16
17 hostname_uh = HostnameUH(None, uname)
18 assert hostname_uh.hostname == "hostone_u.example.com"
19
20 hostname_uh = HostnameUH(hostname, uname)
21 assert hostname_uh.hostname == HOSTNAME
First we added an import for the combiner object and the parser objects. Next
we import a helper function context_wrap
which we’ll
use to create our parser instance objects:
1 from insights.combiners.hostname_uh import HostnameUH
2 from insights.parsers.hostname import Hostname
3 from insights.parsers.uname import Uname
4 from insights.tests import context_wrap
Next we include the sample data that will be used for the test. We will use
data for input to the parsers so we need both sample outputs of the hostname
command and the uname -a
command:
6HOSTNAME = "hostone_h.example.com"
7UNAME = "Linux hostone_u.example.com 3.10.0-693.21.1.el7.x86_64 #1 SMP Fri Feb 23 18:54:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux"
Next, to the body of the test, we add code to create instances of the necessary parser classes:
10def test_hostname_uh():
11 hostname = Hostname(context_wrap(HOSTNAME))
12 uname = Uname(context_wrap(UNAME))
Finally we add our tests using the attributes that we want to be able to
access in our rules. For our combiner we trust hostname
more than
uname
so we give hostname
priority by checking it first and then
fall back to uname
if hostname is not available. If neither of these is
available the combiner will not be called. It is always guaranteed that our
combiner will get at least one of the parsers when called.
Now here are the tests:
14hostname_uh = HostnameUH(hostname, None)
15assert hostname_uh.hostname == HOSTNAME
16
17hostname_uh = HostnameUH(None, uname)
18assert hostname_uh.hostname == "hostone_u.example.com"
19
20hostname_uh = HostnameUH(hostname, uname)
21assert hostname_uh.hostname == HOSTNAME
We use a different hostname in each parser so that we can confirm that the correct parser data is chosen.
Combiner Code¶
The class __init__
method performs all of the work in our combiner. If
your combiner is more complex you may need to add additional methods and utility
functions. Some general recommendations for the combiner class implementation
are:
Choose attributes that make sense for use by actual rules, or how you anticipate rules to use the information. If rules need to iterate over the information then a
list
might be best, or if rules could access via keywords thendict
might be better.Choose attribute types that are not so complex they cannot be easily understood or serialized. Unless you know you need something complex keep it simple.
Use the
@property
decorator to create read-only getters and simplify access to information.
Now we need to implement the combiner that will satisfy our tests.
1from insights.core.plugins import combiner
2from insights.parsers.hostname import Hostname
3from insights.parsers.uname import Uname
4
5
6@combiner([Hostname, Uname])
7class HostnameUH(object):
8
9 def __init__(self, hostname, uname):
10 if hostname:
11 self.hostname = hostname.fqdn
12 else:
13 self.hostname = uname.nodename
We’ve replaced our original __init__
to include the logic for our combiner.
The Hostname
parser is passed in as the hostname
attribute, and if it
is present then we use it to acquire the hostname data. If hostname
is
None
, meaning that there was no data or there was some error in the data
for the Hostname
parser, we fall back to use the Uname
parser data
passed in the uname
attribute.
Now save this file and run the tests again to confirm that we have successfully written our combiner to pass all tests:
(env)[userone@hostone insights-core-tutorials]$ pytest -k hostname_uh
======================= test session starts ==============================
platform linux -- Python 3.6.6, pytest-3.0.6, py-1.7.0, pluggy-0.4.0
rootdir: /home/userone/insights-core-tutorials, inifile: setup.cfg
plugins: cov-2.6.1
collected 6 items / 5 deselected
insights_examples/tests/combiners/test_hostname_uh.py . [ 50%]
mycomponents/tests/combiners/test_hostname_uh.py .
============ 2 passed, 14 deselected in 0.35 seconds ====================
Combiner Documentation and Testing¶
The last step to complete implementation of our combiner is to create the documentation. The guidelines and examples for combiner documentation is provided in the section Documentation Guidelines and parallels the information provided in the instructions for Parser Documentation. Combiner testing parallels the information provided in the instructions for the Parser Testing