0.5.0
Released: November 15, 2021
Language Enhancements
Struct data type
Nested data in the event input can now be extracted into a type-safe struct, including support for optional elements and default values:
policy_coverage := $.data.policy_coverage as {
bodily_injury_per_accident: int?,
coverage_level as string,
medical_payment as int ? 0,
}
Structs can also be constructed directly:
appearance := {height: 5.8, eye_color: "blue": bald: true}
You might build a struct to organize features into groups, perhaps to be passed to a model:
model_v1_features := {rate_by_zip: rate_by_zip, account_age: account_age, previous_coverage: previous_coverage}
To do the same in abbreviated syntax:
model_v1_features := {rate_by_zip, account_age, previous_coverage}
Use the dot operator to access items of a struct:
intensity := (col.red + col.green + col.blue) / 3
Map data type
Similar to structs, maps are key-value pairs with string keys. However, in maps, keys are dynamic (determined at run-time). And values must all share the same type.
To extract a map from event data:
prices := $.prices as map{string: float}
While keys must be strings, values can be of any type (including complex types):
locations := $.locations as map{string: {lat: float, lng: float}}
Use the subscript operator with a string key to access items of a map:
intensity := (col['red'] + col['green'] + col['blue']) / 3
Retrieve the keys and values of the map (in key-sorted order) with the corresponding builtin functions:
my_map := map{"b": 7, "a": 2, "c": 5}
Keys(my_map)
-- ["a", "b", "c"]
Values(my_map)
-- [2, 7, 5]
Improved array support
Extracting arrays from the event input now also supports optional elements default values:
contains_nulls := $.path as [float?]
-- [0.0, null, -2.5]
with_defaults $.path as [string ? ""]
-- ["hello", "", "world"]
completely_missing := $.path as [int]?
-- null
Finally, arrays, structs and lists can be arbitrarily nested:
complex := $ as {foo: map{string: [[map{string: {lat: float, lng: float}}]]}}
Multi-Column Latest
To reduce repetition and achieve a similar syntax to SQL, Latest
queries can now return multiple values:
policy_coverage := Latest<carrier_details>(
bodily_injury_per_accident,
coverage_level,
medical_payment
by user_id)
The result is a struct. As with all structs, fields can be accessed with the dot operator:
policy_coverage.coverage_level
Latest
queries that select only a single column return a scalar value, as before.
Method Call Syntax
In some cases, it feels more natural to call a function as a method with dot notation. So now, all SCOWL functions can optionally be written as method calls on the first argument:
my_map.Keys() = Keys(my_map)
"asdf".Substr(1, 3) = "sd"
GeoHashDecode("8675309").lng.Abs() = 164.485
SSO for Python SDK (Notebooks)
In Sumatra v0.4.x, Python Notebook access came with several shortcomings:
- Non-expiring SDK keys ran the risk of being committed to source control or otherwise compromised
- A shared SDK key prevented user-level auditing of SDK-originating actions
- SDK access required connecting to the VPN, making it difficult for Sumatra to connect and help
To overcome these issues in v0.5.x, we replaced shared SDK keys with token-based Console authentication. To access notebooks now:
- User logs into Console UI with SSO credentials
- Under Settings, copies the Python connection code with embedded token
- Pastes code into Notebook
The token is valid for 24 hours before a new token must be retrieved from the Console.
Note: In a future release, this process will become even easier by launching a browser login directly from the notebook.
Improved Test Event Widget
Under the Live Topology menu, you can still access the Test Event widget, as before, to see responses to particular JSON inputs. We made two significant changes:
- The widget is now read-only, meaning that events cannot write values to aggregates or propagate data to the Feed. Send test events to your heart's content.
- By switching to using the Console endpoint, the widget is now functional even when your API endpoint is locked down with tight network access controls.
Materialization Error Report
As in v0.4.x, one can fetch a materialized dataframe in Python as follows:
mtr = sumatra.materialize('timeline_name')
mtr.get_enriched('event_type_name')
In v0.5.x, users can now fetch an additional dataframe containing any errors generated during the materialization:
mtr.get_errors('event_type_name')
The resulting dataframe is structured like the feature dataframe, with rows corresponding to events and columns corresponding to features. However, in this dataframe, cells contain error strings and only non-null rows and columns are included.