<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body><div style="font-family:Helvetica,Arial;font-size:13px"><br></div><p class="airmail_on">On September 23, 2019 at 9:20:16 AM, Jeff Frontz (<a href="mailto:jeff.frontz@gmail.com">jeff.frontz@gmail.com</a>) wrote:</p> <div><blockquote type="cite" class="clean_bq" style="font-family:Helvetica,Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><span><div><div></div><div><div dir="ltr">I have a need to generate configuration files for a bunch of different (some open-source, some semi-proprietary) applications that run on a bunch of different systems. The format of the configuration files differs between applications -- TOML, json, free-form text, etc. -- and the contents vary between systems for a given application (but in a readily identifiable way - the applications/system form various mesh networks).<div><br></div><div>For example, I need to set up tor hidden services between hosts and I want each host to use a unique service to connect to each other host -- so I need to configure the services on a host and I need to configure the applications on the other hosts to use the each of the destination hosts' services.</div><div><br></div><div>I don't want to actually push the configurations to the hosts in real-time, but rather I want to store them locally on the configuration system (for Reasons).</div><div><br></div><div>I know there are several popular configuration management tools, but I have zero experience with them -- some/most seem to be tailored to actually manipulate each configured machine (vs. capturing the per-machine configurations locally).</div><div><br></div><div>Anyone use one that fits the bill?</div></div></div></div></span></blockquote></div><p>[Ed. note: While always a big of automation, I went from resisting CM to becoming a huge advocate because (as much as I want to punch Chef in the throat at least twice a week) it has made my life massively easier. I’ve become a huge advocate of CM, so... sorry not sorry for the length of the reply below.]</p><p>So yeah, you are talking about configuration management. In both Chef and Puppet, you can both capture and enforce the configuration. You’re right that CM tools are generally designed toward configuration enforcement. While I’ve never used the feature, I think that both Puppet and Chef allow reporting-only mode. (I’ve never used Salt or Ansible[1], so I can’t speak to those.) This obviously defeats the purpose of the “management” part of CM and still leaves you in the position of setting all this config manually or (possibly worse) by some other process.</p><p>I’m not sure what Reasons there might be for not wanting to enforce configuration, but this in my experience is often a _cultural_, rather than technical, battle with an org and we’ve-always-done-it-this-way folks. In many ways the culture battle can be a more difficult fight than the technical one.</p><p>There are a few of advantages of CM in your situation I can think of offhand:</p><p>* you get a level of abstraction from the hosts and varied config formats (not 100%, but it’s much better)</p><p>* you can give yourself and your systems/configs a system of shared knowledge among them in a way that allows you to use knowledge about the environment (ie host A) to configure host B.</p><p>* If you choose to enforce the configuration, you can demonstrate (configuration as code) that the config you’re managing will revert to what it should be even if a keyboard monkey gets into the system.</p><p>* Bringing up a new host (because the aforementioned keyboard monkey got mad and threw poo into the works) with the correct configuration is VASTLY easier than trying to rebuild the dead host by hand. I’ve been at this so long, I forget things at roughly the speed of light. I don’t have to remember how to configure something or how something should be configured. I let Chef do it for me.</p><p>For me, Chef (or Puppet) code also helps to serve as documentation. This is by design, and is core to how Chef/Puppet are meant to be used. They’re almost meant to be the anti-shell script. That is, they’re languages meant at their core to be readable, flexible, highly descriptive, and platform agnostic.</p><p>For example, if I need to install imagemagick using Chef:</p><p><span class="Apple-tab-span" style="white-space:pre"> </span>package ‘imagemagick’</p><p>It doesn’t matter if it’s EL (yum/rpm), Ubuntu (apt), or some other package manager. Chef figures it out. (The same sort of capability can function on Windows, but it’s a slight bit more complex.)</p><p>For configuration files, (this is just one way to do it) I can create a template file that looks like the final config format - be that JSON, TOML, yaml, whatever. I put placeholders (variable) where I want to dynamically set a value. Then in the Chef/Puppet code, I say something like</p><p><span class="Apple-tab-span" style="white-space:pre"> </span>template ‘/opt/someapp/config/urmom.json’<br><span class="Apple-tab-span" style="white-space:pre"> </span>source ‘template_source.json.erb'<br><span class="Apple-tab-span" style="white-space:pre"> </span>variables( pancakes: ‘yes’ )</p><p>This will look in your code for the file ‘template_source.json` (the extension/format doesn’t matter), look for the ‘pancakes’ placeholder, insert ‘yes’, and write the result as a file to /opt/someapp/config/urmom.json. Again, the file format doesn’t matter. Even more than not mattering, you can almost entirely forget about it (abstraction) when reading the code. Who cares that it’s JSON or whatever. It doesn’t matter. I can see that there’s a template file, and what/how the values are filled in without worrying about it. If I want to open the template file itself, then and only then do I need to think about that it’s JSON or YAML or whatever.</p><blockquote type="cite" class="clean_bq"><div dir="ltr">I need to configure the applications on the other hosts to use the each of the destination hosts' services.</div></blockquote><p>One thing you may be looking for is service discovery. We use Hashicorp Consul. It does much, much more than just this but one of the super helpful things it does is that it does host discovery with it’s own sort of dynamic DNS. When a host comes up and says “hey consul, here I am!”, Consul creates a DNS entry <host>.node.consul. Similarly, when a host advertises a service to consul (say because we want consul to monitor the service), Consul creates a DNS entry <service>.service.consul. You get to stop trying to remember what the IP address for such-and-such is.</p><p>I haven’t spent much time exploring Consul beyond that, but Hashicorp tends to provide rich APIs with their tools. There’s much more capability available in just the service discovery than we’re using right now.</p><p>HTH</p><p>[1] ha - autocorrect wants to spell “Ansible” as “unusable"</p><div></div></body></html>