benthos

Conditions

This document was generated with benthos --list-conditions

Conditions are boolean queries that can be executed based on the contents of a message. Some processors such as filter use conditions for expressing their logic.

Conditions themselves can modify (not) and combine (and, or) other conditions, and can therefore be used to create complex boolean expressions.

The format of a condition is similar to other Benthos types:

condition:
  type: text
  text:
    operator: equals
    part: 0
    arg: hello world

And using boolean condition types we can combine multiple conditions together:

condition:
  type: and
  and:
  - type: text
    text:
      operator: contains
      arg: hello world
  - type: or
    or:
    - type: text
      text:
        operator: contains
        arg: foo
    - type: not
      not:
        type: text
        text:
          operator: contains
          arg: bar

The above example could be summarised as ‘text contains “hello world” and also either contains “foo” or does not contain “bar”’.

Conditions can be extremely useful for creating filters on an output. By using a fan out output broker with ‘filter’ processors on the brokered outputs it is possible to build curated data streams that filter on the content of each message.

Batching and Multipart Messages

All conditions can be applied to a multipart message, which is synonymous with a batch. Some conditions target a specific part of a message batch, and require you specify the target index with the field part.

Some processors such as filter apply its conditions across the whole batch. Whereas other processors such as filter_parts will apply its conditions on each part of a batch individually, in which case the condition acts as if it were referencing a single message batch.

Part indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1. E.g. if part = -1 then the selected part will be the last part of the message, if part = -2 then the part before the last element with be selected, and so on.

Reusing Conditions

Sometimes large chunks of logic are reused across processors, or nested multiple times as branches of a larger condition. It is possible to avoid writing duplicate condition configs by using the resource condition.

Contents

1. and
2. bounds_check
3. check_field
4. count
5. jmespath
6. metadata
7. not
8. or
9. processor_failed
10. resource
11. static
12. text
13. xor

and

type: and
and: []

And is a condition that returns the logical AND of its children conditions.

bounds_check

type: bounds_check
bounds_check:
  max_part_size: 1.073741824e+09
  max_parts: 100
  min_part_size: 1
  min_parts: 1

Checks a message against a set of bounds.

check_field

type: check_field
check_field:
  condition: {}
  parts: []
  path: ""

Extracts the value of a field within messages (currently only JSON format is supported) and then tests the extracted value against a child condition.

count

type: count
count:
  arg: 100

Counts messages starting from one, returning true until the counter reaches its target, at which point it will return false and reset the counter. This condition is useful when paired with the read_until input, as it can be used to cut the input stream off once a certain number of messages have been read.

It is worth noting that each discrete count condition will have its own counter. Parallel processors containing a count condition will therefore count independently. It is, however, possible to share the counter across processor pipelines by defining the count condition as a resource.

jmespath

type: jmespath
jmespath:
  part: 0
  query: ""

Parses a message part as a JSON blob and attempts to apply a JMESPath expression to it, expecting a boolean response. If the response is true the condition passes, otherwise it does not. Please refer to the JMESPath website for information and tutorials regarding the syntax of expressions.

For example, with the following config:

jmespath:
  part: 0
  query: a == 'foo'

If the initial jmespaths of part 0 were:

{
	"a": "foo"
}

Then the condition would pass.

JMESPath is traditionally used for mutating JSON, in order to do this please instead use the jmespath processor.

metadata

type: metadata
metadata:
  arg: ""
  key: ""
  operator: equals_cs
  part: 0

Metadata is a condition that checks metadata keys of a message part against an operator from the following list:

enum

Checks whether the contents of a metadata key matches one of the defined enum values.

type: metadata
metadata:
  operator: enum
  part: 0
  key: foo
  arg:
    - bar
    - baz
    - qux
    - quux

equals

Checks whether the contents of a metadata key matches an argument. This operator is case insensitive.

type: metadata
metadata:
  operator: equals
  part: 0
  key: foo
  arg: bar

equals_cs

Checks whether the contents of a metadata key matches an argument. This operator is case sensitive.

type: metadata
metadata:
  operator: equals_cs
  part: 0
  key: foo
  arg: BAR

exists

Checks whether a metadata key exists.

type: metadata
metadata:
  operator: exists
  part: 0
  key: foo

greater_than

Checks whether the contents of a metadata key, parsed as a floating point number, is greater than an argument. Returns false if the metadata value cannot be parsed into a number.

type: metadata
metadata:
  operator: greater_than
  part: 0
  key: foo
  arg: 3

has_prefix

Checks whether the contents of a metadata key match one of the provided prefixes. The arg field can either be a singular prefix string or a list of prefixes.

type: metadata
metadata:
  operator: has_prefix
  part: 0
  key: foo
  arg:
    - foo
    - bar
    - baz

less_than

Checks whether the contents of a metadata key, parsed as a floating point number, is less than an argument. Returns false if the metadata value cannot be parsed into a number.

type: metadata
metadata:
  operator: less_than
  part: 0
  key: foo
  arg: 3

regexp_partial

Checks whether any section of the contents of a metadata key matches a regular expression (RE2 syntax).

type: metadata
metadata:
  operator: regexp_partial
  part: 0
  key: foo
  arg: "1[a-z]2"

regexp_exact

Checks whether the contents of a metadata key exactly matches a regular expression (RE2 syntax).

type: metadata
metadata:
  operator: regexp_partial
  part: 0
  key: foo
  arg: "1[a-z]2"

not

type: not
not: {}

Not is a condition that returns the opposite (NOT) of its child condition. The body of a not object is the child condition, i.e. in order to express ‘part 0 NOT equal to “foo”’ you could have the following YAML config:

type: not
not:
  type: text
  text:
    operator: equal
    part: 0
    arg: foo

Or, the same example as JSON:

{
	"type": "not",
	"not": {
		"type": "text",
		"text": {
			"operator": "equal",
			"part": 0,
			"arg": "foo"
		}
	}
}

or

type: or
or: []

Or is a condition that returns the logical OR of its children conditions.

processor_failed

type: processor_failed
processor_failed:
  part: 0

Returns true if a processing stage of a message has failed. This condition is useful for dropping failed messages or creating dead letter queues, you can read more about these patterns here.

resource

type: resource
resource: ""

Resource is a condition type that runs a condition resource by its name. This condition allows you to run the same configured condition resource in multiple processors, or as a branch of another condition.

For example, let’s imagine we have two outputs, one of which only receives messages that satisfy a condition and the other receives the logical NOT of that same condition. In this example we can save ourselves the trouble of configuring the same condition twice by referring to it as a resource, like this:

output:
  type: broker
  broker:
    pattern: fan_out
    outputs:
    - type: foo
      foo:
        processors:
        - type: filter
          filter:
            type: resource
            resource: foobar
    - type: bar
      bar:
        processors:
        - type: filter
          filter:
            type: not
            not:
              type: resource
              resource: foobar
resources:
  conditions:
    foobar:
      type: text
      text:
        operator: equals_cs
        part: 1
        arg: filter me please

static

type: static
static: true

Static is a condition that always resolves to the same static boolean value.

text

type: text
text:
  arg: ""
  operator: equals_cs
  part: 0

Text is a condition that checks the contents of a message part as plain text against a logical operator and an argument.

Available logical operators are:

equals_cs

Checks whether the part equals the argument (case sensitive.)

equals

Checks whether the part equals the argument under unicode case-folding (case insensitive.)

contains_cs

Checks whether the part contains the argument (case sensitive.)

contains

Checks whether the part contains the argument under unicode case-folding (case insensitive.)

prefix_cs

Checks whether the part begins with the argument (case sensitive.)

prefix

Checks whether the part begins with the argument under unicode case-folding (case insensitive.)

suffix_cs

Checks whether the part ends with the argument (case sensitive.)

suffix

Checks whether the part ends with the argument under unicode case-folding (case insensitive.)

regexp_partial

Checks whether any section of the message part matches a regular expression (RE2 syntax).

regexp_exact

Checks whether the message part exactly matches a regular expression (RE2 syntax).

xor

type: xor
xor: []

Xor is a condition that returns the logical XOR of its children conditions, meaning it only resolves to true if exactly one of its children conditions resolves to true.

This site is open source. Improve this page.