Update README with more detailed running instructions from prior project

This will make transition easier for an interim period. Will remove at
version 1.0

README.md

@@ -1,79 +1,156 @@
-# minitor-go
+# [minitor-go](https://git.iamthefij.com/iamthefij/minitor-go)
 
-A reimplementation of [Minitor](https://git.iamthefij.com/iamthefij/minitor) in Go
+A minimal monitoring system
+
+## What does it do?
+
+Minitor accepts a YAML configuration file with a set of commands to run and a set of alerts to execute when those commands fail. It is designed to be as simple as possible and relies on other command line tools to do checks and issue alerts.
+
+## But why?
+
+I'm running a few small services and found Sensu, Consul, Nagios, etc. to all be far too complicated for my usecase.
+
+## So how do I use it?
+
+### Running
+
+Install and execute with:
+
+```bash
+go get github.com/iamthefij/minitor-go
+minitor
+```
+
+If locally developing you can use:
+
+```bash
+make run
+```
+
+It will read the contents of `config.yml` and begin its loop. You could also run it directly and provide a new config file via the `-config` argument.
+
+
+#### Docker
+
+You can pull this repository directly from Docker:
+
+```bash
+docker pull iamthefij/minitor-go:latest
+```
+
+The Docker image uses a default `config.yml` that is copied from `sample-config.yml`. This won't really do anything for you, so when you run the Docker image, you should supply your own `config.yml` file:
+
+```bash
+docker run -v $PWD/config.yml:/app/config.yml iamthefij/minitor-go:latest
+```
+
+Images are provided for `amd64`, `arm`, and `arm64` architechtures.
+
+## Configuring
+
+In this repo, you can explore the `sample-config.yml` file for an example, but the general structure is as follows. It should be noted that environment variable interpolation happens on load of the YAML file.
+
+The global configurations are:
+
+|key|value|
+|---|---|
+|`check_interval`|Maximum frequency to run checks for each monitor|
+|`monitors`|List of all monitors. Detailed description below|
+|`alerts`|List of all alerts. Detailed description below|
+
+### Monitors
+
+All monitors should be listed under `monitors`.
+
+Each monitor allows the following configuration:
+
+|key|value|
+|---|---|
+|`name`|Name of the monitor running. This will show up in messages and logs.|
+|`command`|Specifies the command that should be executed, either in exec or shell form. This command's exit value will determine whether the check is successful|
+|`alert_down`|A list of Alerts to be triggered when the monitor is in a "down" state|
+|`alert_up`|A list of Alerts to be triggered when the monitor moves to an "up" state|
+|`check_interval`|The interval at which this monitor should be checked. This must be greater than the global `check_interval` value|
+|`alert_after`|Allows specifying the number of failed checks before an alert should be triggered|
+|`alert_every`|Allows specifying how often an alert should be retriggered. There are a few magic numbers here. Defaults to `-1` for an exponential backoff. Setting to `0` disables re-alerting. Positive values will allow retriggering after the specified number of checks|
+
+### Alerts
+
+Alerts exist as objects keyed under `alerts`. Their key should be the name of the Alert. This is used in your monitor setup in `alert_down` and `alert_up`.
+
+Eachy alert allows the following configuration:
+
+|key|value|
+|---|---|
+|`command`|Specifies the command that should be executed, either in exec or shell form. This is the command that will be run when the alert is executed. This can be templated with environment variables or the variables shown in the table below|
+
+Also, when alerts are executed, they will be passed through Go's format function with arguments for some attributes of the Monitor. The following monitor specific variables can be referenced using Go formatting syntax:
+
+|token|value|
+|---|---|
+|`{{.AlertCount}}`|Number of times this monitor has alerted|
+|`{{.FailureCount}}`|The total number of sequential failed checks for this monitor|
+|`{{.LastCheckOutput}}`|The last returned value from the check command to either stderr or stdout|
+|`{{.LastSuccess}}`|The ISO datetime of the last successful check|
+|`{{.MonitorName}}`|The name of the monitor that failed and triggered the alert|
+
+### Metrics
+
+Minitor supports exporting metrics for [Prometheus](https://prometheus.io/). Prometheus is an open source tool for reading and querying metrics from different sources. Combined with another tool, [Grafana](https://grafana.com/), it allows building of charts and dashboards. You could also opt to just use Minitor to log check results, and instead do your alerting with Grafana.
+
+It is also possible to use the metrics endpoint for monitoring Minitor itself! This allows setting up multiple instances of Minitor on different servers and have them monitor each-other so that you can detect a minitor outage.
+
+To run minitor with metrics, use the `-metrics` flag. The metrics will be served on port `8080` by default, though it can be overriden using `-metrics-port`. They will be accessible on the path `/metrics`. Eg. `localhost:8080/metrics`.
+
+```bash
+minitor -metrics
+# or
+minitor -metrics -metrics-port 3000
+```
+
+## Contributing
+
+Whether you're looking to submit a patch or just tell me I broke something, you can contribute through the Github mirror and I can merge PRs back to the source repository.
+
+Primary Repo: https://git.iamthefij.com/iamthefij/minitor.git
+
+Github Mirror: https://github.com/IamTheFij/minitor.git
+
+## Original Minitor
+
+This is a reimplementation of [Minitor](https://git.iamthefij.com/iamthefij/minitor) in Go
 
 Minitor is already a minimal monitoring tool. Python 3 was a quick way to get something live, but Python itself comes with a large footprint. Thus Go feels like a better fit for the project, longer term.
 
 Initial target is meant to be roughly compatible requiring only minor changes to configuration. Future iterations may diverge to take advantage of Go specific features.
 
-## Differences from Python version
-
+### Differences from Python version
 
 Templating for Alert messages has been updated. In the Python version, `str.format(...)` was used with certain keys passed in that could be used to format messages. In the Go version, we use a struct, `AlertNotice` defined in `alert.go` and the built in Go templating format. Eg.
 
 minitor-py:
 ```yaml
 alerts:
-  log_command:
-    command: ['echo', '{monitor_name}']
-  log_shell:
+  log:
     command: 'echo {monitor_name}'
 ```
 
 minitor-go:
 ```yaml
 alerts:
-  log_command:
-    command: ['echo', '{{.MonitorName}}']
-  log_shell:
+  log:
     command: 'echo {{.MonitorName}}'
 ```
 
-Finally, newlines in a shell command don't terminate a particular command. Semicolons must be used and continuations should not.
+For the time being, legacy configs for the Python version of Minitor should be compatible if you apply the `-py-compat` flag when running Minitor. Eventually, this flag will go away when later breaking changes are introduced.
 
-minitor-py:
-```yaml
-alerts:
-  log_shell:
-    command: >
-      echo "line 1"
-      echo "line 2"
-      echo "continued" \
-        "line"
-```
+## Future
 
-minitor-go:
-```yaml
-alerts:
-  log_shell:
-    command: >
-      echo "line 1";
-      echo "line 2";
-      echo "continued"
-        "line"
-```
-
-## To do
-There are two sets of task lists. The first is to get rough parity on key features with the Python version. The second is to make some improvements to the framework.
-
-Pairity:
-
-  - [x] Run monitor commands
-  - [x] Run monitor commands in a shell
-  - [x] Run alert commands
-  - [x] Run alert commands in a shell
-  - [x] Allow templating of alert commands
-  - [x] Implement Prometheus client to export metrics
-  - [x] Test coverage
-  - [x] Integration testing (manual or otherwise)
-  - [x] Allow commands and shell commands in the same config key
-
-Improvement (potentially breaking):
+Future, potentially breaking changes
 
   - [ ] Implement leveled logging (maybe glog or logrus)
-  - [ ] Consider switching from YAML to TOML
   - [ ] Consider value of templating vs injecting values into Env variables
-  - [ ] Consider dropping `alert_up` and `alert_down` in favor of using Go templates that offer more control of messaging
   - [ ] Async checking
-  - [ ] Use durations rather than seconds checked in event loop
   - [ ] Revisit metrics and see if they all make sense
+  - [ ] Consider dropping `alert_up` and `alert_down` in favor of using Go templates that offer more control of messaging (Breaking)
+  - [ ] Use durations rather than seconds checked in event loop (Potentially breaking)

alert.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"log"
 	"os/exec"
+	"strings"
 	"text/template"
 	"time"
 )
@@ -35,19 +36,35 @@ func (alert Alert) IsValid() bool {
 
 // BuildTemplates compiles command templates for the Alert
 func (alert *Alert) BuildTemplates() error {
+	// TODO: Remove legacy template support later after 1.0
+	legacy := strings.NewReplacer(
+		"{alert_count}", "{{.AlertCount}}",
+		"{alert_message}", "{{.MonitorName}} check has failed {{.FailureCount}} times",
+		"{failure_count}", "{{.FailureCount}}",
+		"{last_output}", "{{.LastCheckOutput}}",
+		"{last_success}", "{{.LastSuccess}}",
+		"{monitor_name}", "{{.MonitorName}}",
+	)
 	if LogDebug {
 		log.Printf("DEBUG: Building template for alert %s", alert.Name)
 	}
 	if alert.commandTemplate == nil && alert.Command.Command != nil {
 		alert.commandTemplate = []*template.Template{}
 		for i, cmdPart := range alert.Command.Command {
+			if PyCompat {
+				cmdPart = legacy.Replace(cmdPart)
+			}
 			alert.commandTemplate = append(alert.commandTemplate, template.Must(
 				template.New(alert.Name+string(i)).Parse(cmdPart),
 			))
 		}
 	} else if alert.commandShellTemplate == nil && alert.Command.ShellCommand != "" {
+		shellCmd := alert.Command.ShellCommand
+		if PyCompat {
+			shellCmd = legacy.Replace(shellCmd)
+		}
 		alert.commandShellTemplate = template.Must(
-			template.New(alert.Name).Parse(alert.Command.ShellCommand),
+			template.New(alert.Name).Parse(shellCmd),
 		)
 	} else {
 		return fmt.Errorf("No template provided for alert %s", alert.Name)
@@ -57,7 +74,7 @@ func (alert *Alert) BuildTemplates() error {
 }
 
 // Send will send an alert notice by executing the command template
-func (alert Alert) Send(notice AlertNotice) (output_str string, err error) {
+func (alert Alert) Send(notice AlertNotice) (outputStr string, err error) {
 	log.Printf("INFO: Sending alert %s for %s", alert.Name, notice.MonitorName)
 	var cmd *exec.Cmd
 	if alert.commandTemplate != nil {
@@ -92,10 +109,23 @@ func (alert Alert) Send(notice AlertNotice) (output_str string, err error) {
 
 	var output []byte
 	output, err = cmd.CombinedOutput()
-	output_str = string(output)
+	outputStr = string(output)
 	if LogDebug {
-		log.Printf("DEBUG: Alert output for: %s\n---\n%s\n---", alert.Name, output_str)
+		log.Printf("DEBUG: Alert output for: %s\n---\n%s\n---", alert.Name, outputStr)
 	}
 
-	return output_str, err
+	return outputStr, err
+}
+
+// NewLogAlert creates an alert that does basic logging using echo
+func NewLogAlert() *Alert {
+	return &Alert{
+		Name: "log",
+		Command: CommandOrShell{
+			Command: []string{
+				"echo",
+				"{{.MonitorName}} check has failed {{.FailureCount}} times",
+			},
+		},
+	}
 }

alert_test.go

@@ -34,6 +34,7 @@ func TestAlertSend(t *testing.T) {
 		expectedOutput string
 		expectErr      bool
 		name           string
+		pyCompat       bool
 	}{
 		{
 			Alert{Command: CommandOrShell{Command: []string{"echo", "{{.MonitorName}}"}}},
@@ -41,6 +42,7 @@ func TestAlertSend(t *testing.T) {
 			"test\n",
 			false,
 			"Command with template",
+			false,
 		},
 		{
 			Alert{Command: CommandOrShell{ShellCommand: "echo {{.MonitorName}}"}},
@@ -48,6 +50,7 @@ func TestAlertSend(t *testing.T) {
 			"test\n",
 			false,
 			"Command shell with template",
+			false,
 		},
 		{
 			Alert{Command: CommandOrShell{Command: []string{"echo", "{{.Bad}}"}}},
@@ -55,6 +58,7 @@ func TestAlertSend(t *testing.T) {
 			"",
 			true,
 			"Command with bad template",
+			false,
 		},
 		{
 			Alert{Command: CommandOrShell{ShellCommand: "echo {{.Bad}}"}},
@@ -62,11 +66,22 @@ func TestAlertSend(t *testing.T) {
 			"",
 			true,
 			"Command shell with bad template",
+			false,
+		},
+		{
+			Alert{Command: CommandOrShell{ShellCommand: "echo {alert_message}"}},
+			AlertNotice{MonitorName: "test", FailureCount: 1},
+			"test check has failed 1 times\n",
+			false,
+			"Command shell with legacy template",
+			true,
 		},
 	}
 
 	for _, c := range cases {
 		log.Printf("Testing case %s", c.name)
+		// Set PyCompat to value of compat flag
+		PyCompat = c.pyCompat
 		c.alert.BuildTemplates()
 		output, err := c.alert.Send(c.notice)
 		hasErr := (err != nil)
@@ -78,6 +93,8 @@ func TestAlertSend(t *testing.T) {
 			t.Errorf("Send(%v err), expected=%v actual=%v", c.name, "Err", err)
 			log.Printf("Case failed: %s", c.name)
 		}
+		// Set PyCompat back to default value
+		PyCompat = false
 		log.Println("-----")
 	}
 }

config.go

@@ -50,6 +50,19 @@ func (cos *CommandOrShell) UnmarshalYAML(unmarshal func(interface{}) error) erro
 func (config Config) IsValid() (isValid bool) {
 	isValid = true
 
+	// Validate alerts
+	if config.Alerts == nil || len(config.Alerts) == 0 {
+		// This should never happen because there is a default alert named 'log' for now
+		log.Printf("ERROR: Invalid alert configuration: Must provide at least one alert")
+		isValid = false
+	}
+	for _, alert := range config.Alerts {
+		if !alert.IsValid() {
+			log.Printf("ERROR: Invalid alert configuration: %s", alert.Name)
+			isValid = false
+		}
+	}
+
 	// Validate monitors
 	if config.Monitors == nil || len(config.Monitors) == 0 {
 		log.Printf("ERROR: Invalid monitor configuration: Must provide at least one monitor")
@@ -74,18 +87,6 @@ func (config Config) IsValid() (isValid bool) {
 		}
 	}
 
-	// Validate alerts
-	if config.Alerts == nil || len(config.Alerts) == 0 {
-		log.Printf("ERROR: Invalid alert configuration: Must provide at least one alert")
-		isValid = false
-	}
-	for _, alert := range config.Alerts {
-		if !alert.IsValid() {
-			log.Printf("ERROR: Invalid alert configuration: %s", alert.Name)
-			isValid = false
-		}
-	}
-
 	return
 }
 
@@ -117,6 +118,17 @@ func LoadConfig(filePath string) (config Config, err error) {
 		log.Printf("DEBUG: Config values:\n%v\n", config)
 	}
 
+	// Add log alert if not present
+	if PyCompat {
+		// Intialize alerts list if not present
+		if config.Alerts == nil {
+			config.Alerts = map[string]*Alert{}
+		}
+		if _, ok := config.Alerts["log"]; !ok {
+			config.Alerts["log"] = NewLogAlert()
+		}
+	}
+
 	if !config.IsValid() {
 		err = errors.New("Invalid configuration")
 		return

config_test.go

@@ -10,22 +10,29 @@ func TestLoadConfig(t *testing.T) {
 		configPath string
 		expectErr  bool
 		name       string
+		pyCompat   bool
 	}{
-		{"./test/valid-config.yml", false, "Valid config file"},
-		{"./test/does-not-exist", true, "Invalid config path"},
-		{"./test/invalid-config-type.yml", true, "Invalid config type for key"},
-		{"./test/invalid-config-missing-alerts.yml", true, "Invalid config missing alerts"},
-		{"./test/invalid-config-unknown-alert.yml", true, "Invalid config unknown alert"},
+		{"./test/valid-config.yml", false, "Valid config file", false},
+		{"./test/valid-default-log-alert.yml", false, "Valid config file with default log alert PyCompat", true},
+		{"./test/valid-default-log-alert.yml", true, "Invalid config file no log alert", false},
+		{"./test/does-not-exist", true, "Invalid config path", false},
+		{"./test/invalid-config-type.yml", true, "Invalid config type for key", false},
+		{"./test/invalid-config-missing-alerts.yml", true, "Invalid config missing alerts", false},
+		{"./test/invalid-config-unknown-alert.yml", true, "Invalid config unknown alert", false},
 	}
 
 	for _, c := range cases {
 		log.Printf("Testing case %s", c.name)
+		// Set PyCompat based on compatibility mode
+		PyCompat = c.pyCompat
 		_, err := LoadConfig(c.configPath)
 		hasErr := (err != nil)
 		if hasErr != c.expectErr {
 			t.Errorf("LoadConfig(%v), expected_error=%v actual=%v", c.name, c.expectErr, err)
 			log.Printf("Case failed: %s", c.name)
 		}
+		// Set PyCompat to default value
+		PyCompat = false
 		log.Println("-----")
 	}
 }

main.go

@@ -18,6 +18,9 @@ var (
 	// Metrics contains all active metrics
 	Metrics = NewMetrics()
 
+	// PyCompat enables support for legacy Python templates
+	PyCompat = false
+
 	// version of minitor being run
 	version = "dev"
 )
@@ -83,7 +86,10 @@ func main() {
 	// Get debug flag
 	flag.BoolVar(&LogDebug, "debug", false, "Enables debug logs (default: false)")
 	flag.BoolVar(&ExportMetrics, "metrics", false, "Enables prometheus metrics exporting (default: false)")
+	flag.BoolVar(&PyCompat, "py-compat", false, "Enables support for legacy Python Minitor config. Will eventually be removed. (default: false)")
+	flag.IntVar(&MetricsPort, "metrics-port", 8080, "The port that Prometheus metrics should be exported on, if enabled. (default: 8080)")
 	var showVersion = flag.Bool("version", false, "Display the version of minitor and exit")
+	var configPath = flag.String("config", "config.yml", "Alternate configuration path (default: config.yml)")
 	flag.Parse()
 
 	// Print version if flag is provided
@@ -93,7 +99,7 @@ func main() {
 	}
 
 	// Load configuration
-	config, err := LoadConfig("config.yml")
+	config, err := LoadConfig(*configPath)
 	if err != nil {
 		log.Fatalf("Error loading config: %v", err)
 	}

test/valid-default-log-alert.yml

@@ -0,0 +1,8 @@
+---
+check_interval: 1
+
+monitors:
+  - name: Command
+    command: ['echo', '$PATH']
+    alert_down: ['log']
+    alert_every: 0