Getting started with SaltStack Configuration Management

SaltStack is one of the least popular Automation and Configuration Management tool, similar to Ansible, Chef, and Puppet (also stuff that not a CM, it's IaC tool but if you are skilled enough can be used as CM: Terraform and Pulumi). Unlike Ansible, it have agent that needs to be installed to the target servers. Unlike Chef and Puppet, it uses Python. Like Ansible, SaltStack can be used for infrastructure lifecycle management too. To install Salt, you need to run these commands:

sudo add-apt-repository ppa:saltstack/salt
sudo apt install salt-master salt-minion salt-cloud

# for minion:
sudo vim /etc/salt/minion
# change line "#master: salt" into:
# master: 127.0.0.1 # should be master's IP
# id: thisMachineId
sudo systemctl restart salt-minion

There's simpler way to install, bootstrap master and minion, it's using a bootstrap script:

curl -L https://bootstrap.saltstack.com -o install_salt.sh
sudo sh install_salt.sh -M # for master

sudo sh install_salt.sh    # for minion
# don't forget change /etc/salt/minion, and restart salt-minion

To the minion will send the key to master, to list all possible minion, run:

sudo salt-key -L

To accept/add minion, you can use:

sudo salt-key --accept=FROM_LIST_ABOVE
sudo salt-key -A # accept all unaccepted, -R reject all

# test connection to all minion
sudo salt '*' test.ping

# run a command 
sudo salt '*' cmd.run 'uptime'

If you don't want open port, you can also use salt-ssh, so it would work like Ansible:

pip install salt-ssh
# create /etc/salt/roster containing:
ID1:
  host: minionIp1
  user: root
  sudo: True
ID2_usually_hostname:
  host: minionIp2
  user: root
  sudo: True

To execute a command on all roster you can use salt-ssh:

salt-ssh '*' cmd.run 'hostname'

On SaltStack, there's 5 things that you need to know:

1. Master (the controller) 
2. Minion (the servers/machines being controlled)
3. States (current state of servers)
4. Modules (same like Ansible modules)
1. Grains (facts/properties of machines, to gather facts call: salt-call --grains)
2. Execution (execute action on machines, salt-call moduleX.functionX, for example: cmd.run 'cat /etc/hosts | grep 127. ; uptime ; lsblk', or user.list_users, or grains.ls -- for example to list all grains available properties, or grains.get ipv4_interfaces, or grains.get ipv4_interfaces:docker0)
3. States (idempotent multiplatform for CM)
4. Renderers (module that transform any format to Python dictionary)
5. Pillars (user configuration properties)
5. salt-call (primary command)

To run locally just add --local. To run on every host we can use salt '*' modulename.functionname. The wildcard can be changed with compound filtering argument, more detail and example here.

To start using salt with file mode, create a directory called salt, and create a top.sls file (it's just a yaml combined with jinja) which contains list of host, filters, and state module you want to call, usually it saved on /srv/ directory, containing:

base:
  '*': # every machine

    - requirements # state module to be called
    - statemodule0
dev1:
  'osrelease:*22.04*': # only machine with specific os version
     - match: grain
     - statemodule1
dev2:
  'os:MacOS': # only run on mac
     - match: grain
     - statemodule2/somesubmodule1
prod:
  'os:Pop and host:*.domain1': # only run on PopOS with tld domain1
     - match: grain
     - statemodule3
     - statemodule4
  'os:Pop': # this too will run if the machine match
     - match: grain
     - statemodule5

Then create a directory for each those items containing init.sls or create files for each those items with .sls extension. For example requirements.sls:

essential-packages: # ID = what this state module do

  pkg.installed: # module name, function name

    - pkgs:

      - bash

      - build-essentials

      - git

      - tmux

      - byobu

      - zsh

      - curl

      - htop

      - python-software-properties

      - software-properties-common

      - apache2-utils
  file.managed: 
    - name: /tmp/a/b
    - makedirs: True
    - user: root
    - group: wheel
    - mode: 644
    - source: salt://files/b # will copy ./files/b to machine

  file.managed:

    - name: /tmp/a/c
    - contents: # this will create a file with specific lines
      - line 1
      - line 2
  service.running:
    - name: myservice1
    - watch: # will restart the service if these changed
      - file: /etc/myservice.conf
      - file: /tmp/a/b
  file.append:
    - name: /tmp/a/c
    - text: 'some line' # will append to that file
  cmd.run:
    - name: /bin/someCmd1 param1; /bin/cmd2 --someflag2

    - onchanges:

      - file: /tmp/a/c # run cmd above if this file changed
  file.directory: # ensure directory created
    - name: /tmp/d
    - user: root
    - dirmode: 700
  archive.extracted: # extract from specific archive file

    - name: /tmp/e
    - source: https://somedomain/somefile.tgz
    - force: True
    - keep_source: True
    - clean: True

To apply run: salt-call state.apply requirements

Some other example, we can create a template with jinja and yaml combined, like this:

statemodule0:
  file.managed:
    - name: /tmp/myconf.cfg # will copy file based on jinja condition

    {% if '/usr/bin' in grains['pythonpath'] %}
    - source: salt://files/defaultpython.conf
    {% elif 'Pop' == grains['os'] %}
    - source: salt://files/popos.conf
    {% else %}
    - source: salt://files/unknown.conf
    {% endif %}
    - makedirs: True
  cmd.run:
    - name: echo
    - onchanges:

      - file: statemodule0 # refering statemodule0.file.managed.name

To create a python state module, you can create a file containing something like this:

#!py
def run():
  config = {}
  a_var = 'test1' # we can also do a loop, everything is dict/array
  config['create_file_{}'.format(a_var)] = {
    'file.managed': [
      {'name': '/tmp/{}'.format(a_var)},
      {'makedirs': True},
      {'contents': [
        'line1',
        'line2'
        ]
      },
    ],
  }
  return config
 
To include another state module, you can specify on statemodulename/init.sls, something like this:

include:
  - .statemodule2 # if this a folder, will run the init.sls inside
  - .statemodule3 # if this a file, will run statemodule3.sls

To run all state it you can call salt-call state.highstate or salt-call state.apply without any parameter.
It would execute top.sls file and the includes in order recursively.
To create a scheduled state, you can create a file containing something like this:

id_for_this:
  schedule.present:
    - name: highstate
    - run_on_start: True
    - function: state.highstate
    - minutes: 60
    - maxrunning: 1
    - enabled: True
    - returner: rawfile_json
    - splay: 600 

Full example can be something like this:

install nginx:
  pkg.install:
    - nginx

/etc/nginx/nginx.conf: # used as name
   file.managed:
     source: salt://_files/nginx.j2
     template: jinja
     require:
       - install nginx

run nginx:

  service.running:
    name: nginx
    enable: true
    watch: 
      - /etc/nginx/nginx.conf

Next, to create a pillar config, just create a normal sls file, containing something like this:

user1:
  active: true
  sudo: true
  ssh_keys:
    - ssh-rsa censored user1@domain1
nginx:
  server_name: 'foo.domain1'

To reference this on other salt file, you can use jinja something like this:

{% set sn = salt['pillar.get']('nginx:server_name') -%}
server {
  listen 443 ssl;
  server_name {{ sn }};
...

That's it for now, next if you want to learn more is to create your own executor module or other topics here.

Getting started with Ansible

Ansible is one of the most popular server automation tool (other than Pulumi and Terraform), it's agentless and only need SSH access to run. It also can help you provision server or VM instances using cloud module. You can also provision vagrant/virtualbox/qemu/docker/lxc/containers inside an already running server using Ansible. Other competitor in this category includes Puppet and Chef  but they both require an agent to be installed inside the nodes that want to be controlled. To install Ansible in Ubuntu, run:

sudo add-apt-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

You can put list of servers you want to control in /etc/ansible/hosts
or any other inventory file, something like this:

[DC01]
hostname.or.ip

[DC02]
hostname.or.ip
hostname.or.ip

[DC01:vars]
ansible_user=foo
ansible_pass=bar
# it's preferred to use ssh-keygen ssh-copy-id (passwordless login)
# and sudoers set to ALL=(ALL) NOPASSWD:ALL for the ansible user
# instead of hardcoding the username and password

If you put it on another file, you can use -i inventoryFileName to set the inventory file, also don't forget to check /etc/ansible/ansible.cfg default configs, for example you can set default inventory file to another file there.

Example for checking whether all server on DC02 up:

ansible DC02 -m ping

To run arbitary command on all server on DC01:

ansible DC01 -a "cat /etc/lsb-release" 
# add -f N, to run N forks in parallel

To create a set of commands, we can create a playbook file (which contains one or more play, and has one or more tasks), which is just a yaml file that contains specific structure, something like this:

---
  - name: make sure all tools installed # a play
    hosts: DC01 # or all or "*" or localhost
    become: yes # sudo
    tasks:
      - name: make sure micro installed # a task
        apt: # a module
          name: micro
          state: latest
      - name: make sure golang ppa added
        ansible.builtin.apt_repository:
          repo: deb http://ppa.launchpad.net/longsleep/golang-backports/ubuntu/ focal main
      - name: make sure latest golang installed
        apt: name=golang-1.18 state=present # absent to uninstall
      - name: make sure docker gpg key installed
        raw: 'curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o - > /usr/share/keyrings/docker-archive-keyring.gpg'
      - name: make sure docker-ce repo added
        ansible.builtin.apt_repository:
          repo: 'deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu impish stable'
      - name: make sure docker-ce installed
        apt: name=docker-ce

That playbook, for example if you save it on playbooks/ensure-tools-installed.yml, you can run it using ansible-playbook playbooks/ensure-tools-installed.yml

How to know list of modules and what's their key-value options? visit this site https://docs.ansible.com/ansible/2.9/modules/modules_by_category.html