Skip to content

0.5.0

Released: November 15, 2021

Language Enhancements

Struct data type

Nested data in the event input can now be extracted into a type-safe struct, including support for optional elements and default values:

policy_coverage := $.data.policy_coverage as {
    bodily_injury_per_accident: int?,
    coverage_level as string,
    medical_payment as int ? 0,
}

Structs can also be constructed directly:

appearance := {height: 5.8, eye_color: "blue": bald: true}

You might build a struct to organize features into groups, perhaps to be passed to a model:

model_v1_features := {rate_by_zip: rate_by_zip, account_age: account_age, previous_coverage: previous_coverage}

To do the same in abbreviated syntax:

model_v1_features := {rate_by_zip, account_age, previous_coverage}

Use the dot operator to access items of a struct:

intensity := (col.red + col.green + col.blue) / 3

Map data type

Similar to structs, maps are key-value pairs with string keys. However, in maps, keys are dynamic (determined at run-time). And values must all share the same type.

To extract a map from event data:

prices := $.prices as map{string: float}

While keys must be strings, values can be of any type (including complex types):

locations := $.locations as map{string: {lat: float, lng: float}}

Use the subscript operator with a string key to access items of a map:

intensity := (col['red'] + col['green'] + col['blue']) / 3

Retrieve the keys and values of the map (in key-sorted order) with the corresponding builtin functions:

my_map := map{"b": 7, "a": 2, "c": 5}

Keys(my_map)
-- ["a", "b", "c"]
Values(my_map)
-- [2, 7, 5]

Improved array support

Extracting arrays from the event input now also supports optional elements default values:

contains_nulls := $.path as [float?]
-- [0.0, null, -2.5]
with_defaults $.path as [string ? ""]
-- ["hello", "", "world"]
completely_missing := $.path as [int]?
-- null

Finally, arrays, structs and lists can be arbitrarily nested:

complex := $ as {foo: map{string: [[map{string: {lat: float, lng: float}}]]}}

Multi-Column Latest

To reduce repetition and achieve a similar syntax to SQL, Latest queries can now return multiple values:

policy_coverage := Latest<carrier_details>(
    bodily_injury_per_accident,
    coverage_level,
    medical_payment
by user_id)

The result is a struct. As with all structs, fields can be accessed with the dot operator:

policy_coverage.coverage_level

Latest queries that select only a single column return a scalar value, as before.

Method Call Syntax

In some cases, it feels more natural to call a function as a method with dot notation. So now, all SCOWL functions can optionally be written as method calls on the first argument:

my_map.Keys() = Keys(my_map)
"asdf".Substr(1, 3) = "sd"
GeoHashDecode("8675309").lng.Abs() = 164.485

SSO for Python SDK (Notebooks)

In Sumatra v0.4.x, Python Notebook access came with several shortcomings:

  1. Non-expiring SDK keys ran the risk of being committed to source control or otherwise compromised
  2. A shared SDK key prevented user-level auditing of SDK-originating actions
  3. SDK access required connecting to the VPN, making it difficult for Sumatra to connect and help

To overcome these issues in v0.5.x, we replaced shared SDK keys with token-based Console authentication. To access notebooks now:

  • User logs into Console UI with SSO credentials
  • Under Settings, copies the Python connection code with embedded token
  • Pastes code into Notebook

The token is valid for 24 hours before a new token must be retrieved from the Console.

Note: In a future release, this process will become even easier by launching a browser login directly from the notebook.

Improved Test Event Widget

Under the Live Topology menu, you can still access the Test Event widget, as before, to see responses to particular JSON inputs. We made two significant changes:

  1. The widget is now read-only, meaning that events cannot write values to aggregates or propagate data to the Feed. Send test events to your heart's content.
  2. By switching to using the Console endpoint, the widget is now functional even when your API endpoint is locked down with tight network access controls.

Materialization Error Report

As in v0.4.x, one can fetch a materialized dataframe in Python as follows:

mtr = sumatra.materialize('timeline_name')
mtr.get_enriched('event_type_name')

In v0.5.x, users can now fetch an additional dataframe containing any errors generated during the materialization:

mtr.get_errors('event_type_name')

The resulting dataframe is structured like the feature dataframe, with rows corresponding to events and columns corresponding to features. However, in this dataframe, cells contain error strings and only non-null rows and columns are included.