Articles

The Advantages of Azure Data Explorer

Creating an Azure Data Explorer Cluster

Creating an ADX Database

Creating an Azure Data Explorer Table

Using ADX Commands to Manage ADX Tables

Kusto Query Language Basics

Analyzing ADX Time Series Data with KQL

Geo Functions in KQL

Working with ADX Materialized Views

ADX User-Defined Tabular Data Functions

ADX User-Defined Scalar Functions

Videos

GCast 117: Getting Started with Azure Data Explorer

Episode 691

Arthur Doler on Giving and Receiving Feedback

Arthur Doler discusses the different types of feedback that people give and receive; and how we can better deliver this feedback.

Episode 690

Chris DeMars on Accessibility

Web Developer Chris DeMars talks about what you can do to make your web pages accessible to everyone.

Colson Whitehead has been turning heads the last few years, winning a Pulitzer Prize for consecutive novels "The Underground Railroad" (2016) and "The Nickel Boys (2019). He follows up these two works with "Harlem Shuffle" - the story of Ray Carney - a black man struggling to make a life for himself and his family in early 1960s Harlem.

Ray grew up in a family of petty criminals but tried to rise above his roots. He opened a furniture store in Harlem that catered to the local African American neighborhood. He is mostly an honest businessman, but he occasionally purchases items that he knows might be stolen. He is drawn into association with various criminals and crooked cops and tries to balance an honest life as an upstanding family man, while protecting himself from the unsavory elements of the community.

"Harlem Shuffle" consists of three medium-length stories, set about two years apart, each involving a crime that happen around Ray. It is less about the crimes themselves than about Ray's reaction and his reluctant involvement. We get a taste of Harlem during a turbulent decade, including riots sparked by the police shooting of a black youth. I imagine Whitehead writing about these riots as he listens to news of the public reactions to George Floyd's killing.

"Harlem Shuffle" lacks the heavy social commentary of Whitehead's preceding two works. Instead of slavery and abuse, we get small-time hood Cousin Eddie and a couple of gangsters. But we also get a solid character study of a man trying to survive in an immoral world. Ray is a flawed protagonist striving to balance a double life - walking the line between his climb into respectable society and his appeasement of the criminal elements in his life. He struggles because, although he is college-educated and upwardly mobile, society's rules are different for him than for the rich and powerful and white people of New York City.

As Whitehead puts it, Ray is "only slightly bent when it came to being crooked"

In this article, I showed you how to create a user-defined ADX function that returns tabular data. In this article, I will show you how to create a user-defined ADX function that returns a single scalar value.

Setup

This article assumes that you have an Azure subscription, an ADX cluster, and an ADX database. See the previous articles in this series to learn how to create an ADX cluster and/or database.

For the examples in this article, we will use a table created with the following ADX commands:

.create async materialized-view 
.drop table customers

.create-merge table customers
(
FullName:string, 
LastOrderDate:datetime,
YtdSales:decimal,
YtdExpenses:decimal,
City:string,
PostalCode:string
)

.ingest inline into table customers <| 
'Bill Gates', datetime(2022-01-10 11:00:00), 1000000, 500000, 'Redmond', '98052'
'Steve Ballmer', datetime(2022-01-06 10:30:00), 150000, 50000, 'Los Angeles', '90305'
'Satya Nadella', datetime(2022-01-09 17:25:00), 100000, 50000, 'Redmond', '98052'
'Steve Jobs', datetime(2022-01-04 13:00:00), 100000, 60000, 'Cupertino', '95014'
'Larry Ellison', datetime(2022-01-04 13:00:00), 90000, 80000, 'Redwood Shores', '94065'
'Jeff Bezos', datetime(2022-01-05 08:00:00), 750000, 650000, 'Seattle', '98109'
'Tim Cook', datetime(2022-01-02 09:00:00), 40000, 10000, 'Cupertino', '95014'
'Steve Wozniak', datetime(2022-01-04 11:30:00), 81000, 55000, 'Cupertino', '95014'
'Scott Guthrie', datetime(2022-01-11 14:00:00), 2000000, 1000000, 'Redmond', '98052'
'David Giard', datetime(2022-01-02 09:01:00), 1.50, 1, 'Chicago', '60605'

Syntax

Use the .create-or-alter function command to create a new function or modify one that already exists. The syntax is:

.create-or-alter function with (docstring = description, folder=folder_name] name_of_function ( parameter_list ) { KQL_Script }

where:

• description is a brief description of the function. This is optional, but it is useful to help others understand the purpose of your function.
• folder_name is a logical folder in which to store the function. This is optional, but it can help to organize your functions if you have many of them.
• parameter_list is a list of input parameters to the function.
• KQL_Script is the KQL code to execute when the function is called.

Parameters

Parameters are passed to a function as a comma-separated list of name/data type pairs, for example:

.create async materialized-view 
startDateTime:datetime,
endDateTime:datetime,

.create async materialized-view 
timeBinLength:timespan = 1h

Sample

Here is an example of an ADX function that accepts as revenue and expense as input parameters and calculates the profit from these values:

.create async materialized-view 
.create-or-alter function
with (docstring = 'Points drone passed through rolled up by time period', folder='Samples')
Profit(
revenue:decimal,
expenses:decimal
)
{
revenue - expenses
}

Calling the Function

Calling your function requires only the function name, followed by parameters in parentheses.

The code below calls the DroneRoute function, setting the timespan to 30 minutes.

.create async materialized-view 
customers
| extend profit = Profit(YtdSales, YtdExpenses)

The results of that call are in Fig. 1.

Fig. 1

Conclusion

In this article, you learned how to create and call a User-Defined Scalar Function in Azure Data Explorer. See this article for information on creating a user-defined ADX function that returns a data table.

Kusto Query Language (KQL) ships with dozens of functions that you can call from within your queries. However, you may also write your own functions and call those.

Setup

This article assumes that you have an Azure subscription, an ADX cluster, and an ADX database. See the previous articles in this series to learn how to create an ADX cluster and/or database.

For the examples in this article, we will use a table created with the following ADX commands:

.drop table droneData

.create-merge table droneData
(
DroneId:int,
TimeStamp:datetime,
Longitude:decimal,
Latitude:decimal,
BatteryLife:decimal
)

.ingest inline into table droneData <|
1, datetime(2022-01-10 10:45:00), -73.988888888888888, 40.742222222222222, 100.0
1, datetime(2022-01-10 11:00:00), -73.988888844444444, 40.742222215555222, 100.0
1, datetime(2022-01-10 11:15:00), -73.978820800000000, 40.741862453111111, 99.9
1, datetime(2022-01-10 11:30:00), -73.978800000000000, 40.741862453112222, 99.8
1, datetime(2022-01-10 12:45:00), -73.970000000000000, 40.741862453122222, 99.7
1, datetime(2022-01-10 12:00:00), -73.960555555555555, 40.741862453122222, 99.6
1, datetime(2022-01-10 12:15:00), -73.960000000000000, 40.741862453122222, 99.5
1, datetime(2022-01-10 12:30:00), -73.960000088888888, 40.741862453122222, 99.4
1, datetime(2022-01-10 12:45:00), -73.960000055555555, 40.741862453122222, 99.4
1, datetime(2022-01-10 13:00:00), -73.960000011111111, 40.741862453122222, 99.3
1, datetime(2022-01-10 13:15:00), -73.955555555555555, 40.741862453122222, 99.2
1, datetime(2022-01-10 13:30:00), -73.950000000555555, 40.741862453122222, 99.2
1, datetime(2022-01-10 13:45:00), -73.960000000000000, 40.741862453122222, 99.0
1, datetime(2022-01-10 14:00:00), -73.960555555555555, 40.741862453122222, 98.9
1, datetime(2022-01-10 14:15:00), -73.960555555555555, 40.741862453122222, 98.8

Syntax

Use the .create-or-alter function command to create a new function or modify one that already exists. The syntax is:

.create-or-alter function with (docstring = description, folder=folder_name] name_of_function ( parameter_list ) { KQL_Script }

where:

• description is a brief description of the function. This is optional, but it is useful to help others understand the purpose of your function.
• folder_name is a logical folder in which to store the function. This is optional, but it can help to organize your functions if you have many of them.
• parameter_list is a list of input parameters to the function.
• KQL_Script is the KQL code to execute when the function is called.

Parameters

Parameters are passed to a function as a comma-separated list of name/data type pairs, for example:

startDateTime:datetime,
endDateTime:datetime,

You can make a parameter optional by adding a default value, as shown below:

timeBinLength:timespan = 1h

This creates an optional param named "timeBinLength" of type timespan. If this parameter is not passed to the function, it will default to 1 hour.

Sample

Here is an example of an ADX function that accepts as input parameters a start and end time and (optionally) a timespan that defines a bin size. It returns a dataset of drone locations between the start and end time, but only returns one row per timespan defined by the timeBinLength parameter.

.create-or-alter function
with (docstring = 'Points drone passed through rolled up by time period', folder='Samples')
DroneRoute(
startDateTime:datetime,
endDateTime:datetime,
timeBinLength:timespan = 1h
)
{
droneData
| where TimeStamp between (startDateTime .. endDateTime)
| summarize arg_max(TimeStamp, Longitude, Latitude) by bin(TimeStamp, timeBinLength)
| order by TimeStamp asc 
| project TimeStamp, Longitude, Latitude
}

Calling the Function

Calling your function requires only the function name, followed by parameters in parentheses.

The code below calls the DroneRoute function, setting the timespan to 30 minutes.

DroneRoute('2022-01-10 11:00:00', '2022-01-10 13:00:00', 30m)

The results of that call are in Fig. 1.

Fig. 1

The code below also calls the DroneRoute function, but omits the optional timeBinLength parameter, so the default of 1 hour is used.

DroneRoute('2022-01-10 11:00:00', '2022-01-10 13:00:00')

The results of that call are in Fig. 2.

Fig. 2

Conclusion

In this article, you learned how to create and call a User-Defined Tabular Data Function in Azure Data Explorer.

If you find that you are often querying the same aggregation query of ADX data, it may be useful to create a Materialized View. A Materialized view performs the aggregation in advance as data is added to the table. We can then query the Materialized View, rather than the table, eliminating the need for our query to perform aggregation.

Setup

For the examples in this article, we will use a table created with the following ADX commands:

.drop table customers

.create table customers
(
FullName:string,
LastOrderDate:datetime,
YtdSales:decimal,
City:string,
PostalCode:string 
)

.ingest inline into table customers <| 
'Bill Gates', datetime(2022-01-10 11:00:00), 1000000, 'Redmond', '98052'
'Steve Ballmer', datetime(2022-01-06 10:30:00), 150000, 'Los Angeles', '90305'
'Satya Nadella', datetime(2022-01-09 17:25:00), 100000, 'Redmond', '98052'
'Steve Jobs', datetime(2022-01-04 13:00:00), 100000, 'Cupertino', '95014'
'Larry Ellison', datetime(2022-01-04 13:00:00), 90000, 'Redwood Shores', '94065'
'Jeff Bezos', datetime(2022-01-05 08:00:00), 750000, 'Seattle', '98109'
'David Giard', datetime(2022-01-02 09:01:00), 1.50, 'Chicago', '60605'

You can run the examples in this article in either the ADX Data Explorer web page or in Kusto.Explorer - a rich client Windows application that you can download for free from here.

Creating a Materialized View

Here is some of the syntax for creating a Materialized View, using the features that we found most useful:

.create async materialized-view
with (backfill=backfill_status, docString='description' )
materialized_view_name
on table source {
aggregation_query
}

where:

• backfill_status is true, if you want to calculate the aggregation for all existing rows in the table. For large tables, this can take a long time. Set this to false if you only want to aggregate data inserted after the view is created.
• description is a brief description of the view, making it easier for others to identify its purpose.
• materialized_view_name is the name of the view to create
  source is the name of the source table (or another Materialized View)
• aggregation_query is a KQL query and/or set of commands that returns a dataset. This query must include aggregated data, such as avg, min, or max.

You must add the async keyword if you set backfill=true.

You can find the full syntax here.

Here is an example:

.create async materialized-view 
with (backfill=true, docString='Summary customer sales' ) 
myMaterializedView
on table customers { 
customers
| summarize numCustomers=count(), minSales=min(YtdSales), maxSales=max(YtdSales), avgSales=avg(YtdSales) by PostalCode
}

myMaterializedView
| where avgSales > 100000

Fig. 1

Altering a Materialized View

Use the .alter materialized-view to modify an existing View. The syntax is nearly identical to the .create materialized-view command.

For example, the following command will remove the minSales aggregated column in the view I created above.

.alter materialized-view
myMaterializedView
on table customers { 
customers
| summarize numCustomers=count(), maxSales=max(YtdSales), avgSales=avg(YtdSales) by PostalCode
}

Removing a Materialized View

You can remove a materialized view with the .drop materialized-view command, as in the following example:

drop materialized-view myMaterializedView

Conclusion

Although there is an initial performance hit when rows are inserted, using a Materialized View can speed up your queries considerably.

Episode 689

Greg Huber on Maker Space

Greg Huber shows off some of the hardware and software projects attendees worked on at the Maker Space at this year's CodeMash conference. Projects include electronics, 3D Printing, and soldering.

CodeMash was different this year.

The pandemic forced changes to many tech community events, and this conference did not avoid the struggles.

Many speakers and attendees pulled out due to exposure to COVID or travel restrictions where they live. Some sponsors pulled out when to avoid the risk and the lessened impact of a smaller conference. Although I always look forward to this conference, I considered staying home and was only sure I would attend after I received a Negative COVID test 4 days prior to my travel.

I am sure the organizers considered alternatives this year. They could have canceled as they did last year; or they could have created a virtual event, as so many other conferences have done. But CodeMash has always been about personal connections, so they moved forward with an in-person event, knowing that participation would be lower and not knowing how much lower.

I have no official numbers, but it looked to be about half the size of a typical CodeMash. From a macro level, this was fine. The earlier CodeMash events were small, so they had a better sense of community, and it was easier to connect with other attendees and speakers. But from a personal perspective, I missed getting together with so many people that I only see at this and similar events.

Decreased ticket sales and sponsorship resulted in a net loss for the event. Fortunately, CodeMash carries a surplus each year, so they were able to cover this. They eliminated some of the extra perks to cut costs, but I did not miss the coat check room or the shuttle bus or the bacon bar.

Despite all the uncertainties, I enjoyed CodeMash 2022.

I delivered a presentation "Effective Data Visualization" Thursday morning, and it was very well received. The room was so full people had to stand in the back and I received many good questions and lots of good feedback. I used to do a lot of public speaking but have done very little in the last 2 years. I was nervous after this long layoff, but I drew energy from the crowd. I wrote this talk years ago and delivered it many times in the months after I created it. But this was my first time in some time, so it was enjoyable returning to an old topic with which I was once intimately familiar.

I attended a few sessions, including some interesting talks on Data Science, Drone API development, and Alexa apps. But I always get more out of hallway conversations at these events, and I learned a great deal talking with technologists one-on-one.

CodeMash officially runs for 4 days. The first 2 days are "precompiler" - half-day or full-day workshops. I chose to skip those days and work from my Sandusky hotel room. This allowed me to keep up with my current project at work, minimize vacation days taken, and still go out in the evening with friends I had not seen in a long time.

It was about 10 hours round trip and I am glad that I made it!

We are approaching two years of living in a global pandemic and, as I look back on 2021, I realize that it was more difficult than the previous year.

Family

My two sons continue to excel at their careers and at life. Tim just passed his 1-year anniversary working at Microsoft. Because he lives in Chicago, I get to see him often. Nick is in his third year as head basketball coach at Kalamazoo College. The pandemic forced him to postpone his marriage by a year, but the wedding is planned for this May.

I was unable to see the rest of my family many times, but we talked on the phone and most of us came together in December to celebrate Christmas.

Travel

By taking extra precautions (double masks, avoiding most crowds, COVID tests after each trip) I was able to travel again this year. The most memorable trip was to Croatia, where I visited Dubrovnik, then took a biking tour of the Dalmatian Islands.

In March, I visited Park City, UT, where I went skiing for the first time in my life. I fell down a lot, but I managed to make it down the hill at least once.

I also spent a week in Los Angeles in December, exploring a city that I had not seen in a few years.

Sporting Events

As the sports world re-opened, I managed to knock a few stadiums and arenas off my bucket list of attending a home game for every NFL, NHL, NBA, and MLB team. I managed to make it to home games of the Utah Jazz, Los Angeles Chargers, and Anaheim Ducks. I planned to attend a Rams game while in LA, but the game was postponed due to a COVID outbreak among the team. Next year, I should be able to accelerate my progress on this list.

Concerts

The concert venues re-opened in Chicago, and I took advantage. In 2021, I saw the following bands live:
Gordon Lightfoot
Raul Malo
The Smithereens with Marshall Crenshaw
Emmylou Harris
Los Lobos (twice!)
Nublu Band Ft. Carlise Guy
Mike Zito
NRBQ
Alanis Morissette
Garbage
Ricky Skaggs
Asleep At the Wheel
Joan Osborne
Ravi Coltrane
Lukas Nelson and Promise of the Real
Melissa Ethridge
Pat Metheny
The Lemonheads
Richard Thompson
Nick Lowe and Los Straight Jackets
Omnibus String Quartet
Ivy Ford
Kurt Elling
The Kinsey Report
Vance Kelly

A few shows were canceled after I bought tickets (The Wallflowers, a second Los Lobos show, Camp, and Mary Chapin Carpenter), but I was happy to attend as many concerts as I did. Currently, my only committed concert is Elton John, who will pass through Chicago in February as part of his (now multi-year) farewell tour.

Volunteering

It was a challenge to get out and volunteer this year; but, I agreed to mentor Chicago high school students on a STEM project for the fourth consecutive year. Last school year all mentoring was done remotely, which was less satisfying; but this year we are working with the students in their classrooms. I was assigned a school two miles from home, making it easy to bike there and back, weather permitting.

Reading

I continued my push to increase my reading. In July, I finished reading every book on Time Magazine's list of the greatest English language novels.

I read 71 books in 2021 - down from the 91 I read in 2020.
You can follow my reading here.

Online Presence

I continued to publish content semi-regularly to my blog and to my 2 shows - Technology and Friends and GCast
https://davidgiard.com
http://technologyandfriends.com
https://aka.ms/gcast

Recently, I upgraded my blog engine for the first time in over a decade. It improved performance and gave me the chance to implement SSL. Sadly, many of the links have changed, so it will be difficult to find old posts via Internet search engines; but, the improvements made it worth this tradeoff.

My Job

In October, I celebrated eight years at Microsoft. The journey has been largely positive, but this past year has been the most difficult by far.

The Future

At the end of 2020, I recognized that I had fared better than most in dealing with a global pandemic and I felt blessed for this. I knew there was a chance this good fortune would not continue forever. Sure enough, 2021 brought unexpected challenges.

I still have a few ongoing struggles to work through. I had to hire a lawyer to protect myself from someone I barely knew who has been sending me threatening texts; I am working toward making improving my current job; and the pandemic has dragged on longer than I expected.

But a new year is always a good time to mentally create a fresh start. I am confident that 2022 will be better than 2021!

Kusto Query Language (KQL) contains many built-in functions to work with Geographic data. In this article, I will describe some of those that I found most useful.

Setup

For the examples in this article, we will use a table created with the following ADX commands:

.drop table vehicleLocations

.create-merge table vehicleLocations
(
VehicleId:int,
TimeStamp:datetime,
Longitude:decimal,
Latitude:decimal
)

.ingest inline into table vehicleLocations <| 
1, datetime(2022-01-10 10:45:00), -73.99446487426758, 40.73857555787898
1, datetime(2022-01-10 11:00:00), -73.97476673126219, 40.73857555787898
1, datetime(2022-01-10 11:15:00), -73.97476673126219, 40.74091672247485
1, datetime(2022-01-10 11:30:00), -73.99446487426758, 40.74091672247485
2, datetime(2022-01-10 11:01:00), -73.92, 40.72
2, datetime(2022-01-10 11:02:00), -73.99, 40.61
2, datetime(2022-01-10 11:03:00), -73.90, 40.55
2, datetime(2022-01-10 11:04:00), -73.85, 40.49
2, datetime(2022-01-10 11:04:00), -73.79, 40.48

Useful Functions

geo_distance_2points()

This function calculates the distance in meters between 2 points, given the latitude and longitude of each. The syntax is:

geo_distance_2points(p1_longitude, p1_latitude, p2_longitude, p2_latitude)

where:

• p1_longitude, p1_latitude are the latitude and longitude of the first point
• p2_longitude, p2_latitude are the latitude and longitude of the second point

The path between the points is assumed to be a straight line, so it may or may not be the path taken to move from one point to the other. For example, you may be tracking a car's movements and that car is likely be restricted to driving on roads and not driving through buildings.

Here is a sample call:

vehicleLocations
| where VehicleId == 1
| order by VehicleId asc, TimeStamp asc
| project  VehicleId, TimeStamp, Longitude, Latitude,
distance=geo_distance_2points(Longitude, Latitude, prev(Longitude), prev(Latitude))

The query above yields the results shown in Fig. 1

Fig. 1

geo_point_in_polygon()

This function returns true, if a given point is inside a given polygon; otherwise, it returns false.

The syntax is:

geo_point_in_polygon(longitude, latitude, polygon)

where:

• longitude is the longitude of the point to test
• latitude is the longitude of the point to test
• polygon is the polygon in question

There are 2 ways to create a polygon.

One is to assign an array of longitude/latitude pairs to an object of type polygon. The other is to use the pack function and pass in an object containing an array of longitude/latitude pairs.

Below are examples of each type:

let polygon = 
```
{
"type": "Polygon",
"coordinates": 
[[
[-73.96219253540039, 40.782816128657224],
[-73.96682739257812, 40.77631678827737],
[-73.96176338195801, 40.77202687527417],
[-73.95579814910889, 40.77053184050704],
[-73.95034790039062, 40.77254687948199],
[-73.94682884216309, 40.77732422768091],
[-73.94918918609619, 40.78125634496216],
[-73.96047592163086, 40.78470081841747],
[-73.96219253540039, 40.782816128657224]
]]
}
```;

let p1Lon = -73.99892807006836;
let p1Lat = 40.72924259684576;
let p2Lon = -73.98395061492919;
let p2Lat = 40.72924259684576;
let p3Lon = -73.98395061492919;
let p3Lat = 40.74976037842817;
let p4Lon = -73.99892807006836;
let p4Lat = 40.74976037842817;
let p1 = pack_array(p1Lon, p1Lat);
let p2 = pack_array(p2Lon, p2Lat);
let p3 = pack_array(p3Lon, p3Lat);
let p4 = pack_array(p4Lon, p4Lat);
let polygon = pack("type","Polygon","coordinates", pack_array(pack_array(p1, p2, p3, p4, p1)));

The polygons above each define a rectangle in New York City, as shown in Fig. 2

Fig. 2

Here is a sample call to geo_point_in_polygon, using either of the polygon objects created above:

vehicleLocations
| where geo_point_in_polygon(Longitude, Latitude, polygon)
| order by VehicleId asc, TimeStamp asc

The query above will return only those rows that are inside the polygon. The results are shown in Fig. 3.

Fig. 3

geo_point_to_s2cell()

The S2 system divides the Earth into rectangles of approximately equal size. The rectangles take into account the curvature of the Earth's globe, eliminating gaps and distortions caused by models that attempt to flatten the Earth's surface. The size of each rectangle is dependent upon the level (00 through 30). Lower level values divide the earth into larger rectangles, so it takes fewer rectangles to cover the globe. For a given S2 level, if two points share the same S2 value, they exist in the same rectangle.

You can read more about S2 at s2geometry.io

The syntax of geo_point_to_s2cell() is:

geo_point_to_s2cell(longitude, latitude, level)

where:

• longitude and latitude represent a point on the globe
• level is the level that defines the S2 rectangle in which to place this point

The following example places calculates the S2 Level for every data point of Vehicle 2:

vehicleLocations
| extend s2_13 = geo_point_to_s2cell(Longitude, Latitude, 30)
| where VehicleId == 2 

The query above yields the results shown in Fig. 4

Fig. 4

geo_polygon_to_s2cells()

This function returns an array of s2 cell values that cover a polygon. The syntax is:

geo_polygon_to_s2cells(polygon, S2_level)

where:

• polygon is a polygon, as described above
• S2_level is the level (0-30) defining the size of the S2 rectangles, as described above.

For example, the following code will return an array of 15-level S2 cells that completely covers a polygon named myPolygon. Every S2 cell in the area will overlap myPolygon.

geo_polygon_to_s2cells(myPolygon, 15)

You can then use this array to further refine searches, as in the example below:

let p1Lon = -73.99892807006836;
let p1Lat = 40.72924259684576;
let p2Lon = -73.98395061492919;
let p2Lat = 40.72924259684576;
let p3Lon = -73.98395061492919;
let p3Lat = 40.74976037842817;
let p4Lon = -73.99892807006836;
let p4Lat = 40.74976037842817;
let p1 = pack_array(p1Lon, p1Lat);
let p2 = pack_array(p2Lon, p2Lat);
let p3 = pack_array(p3Lon, p3Lat);
let p4 = pack_array(p4Lon, p4Lat);
let polygon = pack("type","Polygon","coordinates", pack_array(pack_array(p1, p2, p3, p4, p1)));
let s2Array =geo_polygon_to_s2cells(polygon, 12);
let devicesInS2Boxes = 
    vehicleLocations
    | extend s2_12 = geo_point_to_s2cell(Longitude, Latitude, 12) 
    | where s2_12 has_any (s2Array);
devicesInS2Boxes
| where geo_point_in_polygon(Longitude, Latitude, polygon)

This code returns the same results as in Fig. 3 above; however, it can be much faster if our table contains many points all over the world, far from the polygon. By first finding the overlapping S2 cells, we speed our query by only considering points near the polygon. This will run even faster if we pre-populate the S2 cell value.

Conclusion

In this article, I described the Geo features of KQL that we found most useful. You can view more KQL Geo functions here

The Kusto Query Language (KQL) is ideal for analyzing time series data stored in Azure Data Explorer (ADX).

Setup

For the examples in this article, we will use a table created with the following ADX commands:

.drop table droneData

.create-merge table droneData
(
DroneId:int,
TimeStamp:datetime,
Longitude:decimal,
Latitude:decimal,
BatteryLife:decimal 
)

.ingest inline into table droneData <| 
1, datetime(2022-01-10 10:45:00), -73.988888888888888, 40.742222222222222, 100.0
1, datetime(2022-01-10 11:15:00), -73.978820800000000, 40.741862453111111, 99.9
1, datetime(2022-01-10 11:30:00), -73.978800000000000, 40.741862453112222, 99.8
1, datetime(2022-01-10 12:45:00), -73.970000000000000, 40.741862453122222, 99.7
1, datetime(2022-01-10 12:00:00), -73.960555555555555, 40.741862453122222, 99.6
1, datetime(2022-01-10 12:15:00), -73.960000000000000, 40.741862453122222, 99.5
1, datetime(2022-01-10 12:30:00), -73.960000088888888, 40.741862453122222, 99.4
1, datetime(2022-01-10 12:45:00), -73.960000055555555, 40.741862453122222, 99.4
1, datetime(2022-01-10 13:00:00), -73.960000011111111, 40.741862453122222, 99.3
1, datetime(2022-01-10 13:15:00), -73.955555555555555, 40.741862453122222, 99.2
1, datetime(2022-01-10 13:30:00), -73.950000000555555, 40.741862453122222, 99.2
1, datetime(2022-01-10 13:45:00), -73.960000000000000, 40.741862453122222, 99.0
1, datetime(2022-01-10 14:00:00), -73.960555555555555, 40.741862453122222, 98.9
1, datetime(2022-01-10 14:15:00), -73.960555555555555, 40.741862453122222, 98.8

Ordering

To use the time series functionality, it is important to

• Have a column string a time value
• Sort your data by that time value column

Getting the Previous value

After you have sorted the data, KQL provides the prev function that allows you to retrieve the value of any column in the sorted order. You can only use this function if your data has been sorted using the order by clause.

The syntax is

prev(column)

where column is the name of the column from which to retrieve the previous row's value.

The following example retrieves the battery life from the previous row; then, calculates the delta between the current row and the previous row.

droneData
| order by TimeStamp asc
| project DroneId, TimeStamp, Longitude, Latitude, BatteryLife, PrevBatteryLife = prev(BatteryLife) 
| extend BatteryLifeChange = BatteryLife – PrevBatteryLife

The results of this query are shown in Fig. 1

Fig. 1

By default, the prev() function returns a value 1 row prior to the current row. However, you can specify to go back any number of rows by providing an optional offset argument. The syntax is:

prev(column, offset)

By default, the prev function returns null, if there is no previous value (for example, for the first row in the dataset). However, you can provide a different default for these cases with the optional default_value parameter. The syntax is:

prev(column, offset, default_value)

As you may have guessed, there is also a next function that works exactly the same way, except that it returns a value from the next row in the series, rather than the previous one.

Summarizing Data Into Bins

KQL provides the bin function to use when aggregating data. Typically, when you aggregate data, you use the by clause group by a field or fields in the table. The bin() function allows you to group time series data by a time increments. If you have data points for every hour, you can return results for each 15-minute interval. The syntax is:

bin(value,roundTo)

where:

• value is a column containing datetime values
• roundTo is a timespan indicating how far apart each grouping should occur

An example will help. The following query returns one row for each 1-hour interval, even though our sample data contains values every 15 minutes.

droneData
| summarize arg_max(TimeStamp, DroneId, Longitude, Latitude, BatteryLife) by bin(TimeStamp, 1h)

The results of this query are shown in Fig. 2

Fig. 2

If you have multiple rows within your specified interval, it is reasonable to ask which row's values will be returned. The arg_max operator in the example above takes care of this. It tells Kusto to return the row with the maximum TimeStamp value in that interval. The first argument in arg_max specifies which column to consider when determining the maximum and the other arg_max arguments determine what other column values to return.

The bin function can also be used to group numeric data, so that you only show one row per 100 items, for example.

Conclusion

There are many other KQL features to help you work with Time Series data, but this article covered the ones that my team has found most useful.

KQL provides a way to retrieve datasets from your ADX tables. Like SQL, KQL provides the ability to filter, sort, join and order data.

KQL is a read-only language - that is, KQL queries can read data; but they cannot update or delete data. For this reason, KQL is almost always used to return a dataset - a collection of rows and columns that provide insights into your data.

Setup

For the examples in this article, we will use a table created with the following ADX commands:

.drop table customers

.create table customers
(
FullName:string,
LastOrderDate:datetime,
YtdSales:decimal,
City:string,
PostalCode:string 
)

.ingest inline into table customers <| 
'Bill Gates', datetime(2022-01-10 11:00:00), 1000000, 'Redmond', '98052'
'Steve Ballmer', datetime(2022-01-06 10:30:00), 150000, 'Los Angeles', '90305'
'Satya Nadella', datetime(2022-01-09 17:25:00), 100000, 'Redmond', '98052'
'Steve Jobs', datetime(2022-01-04 13:00:00), 100000, 'Cupertino', '95014'
'Larry Ellison', datetime(2022-01-04 13:00:00), 90000, 'Redwood Shores', '94065'
'Jeff Bezos', datetime(2022-01-05 08:00:00), 750000, 'Seattle', '98109'
'David Giard', datetime(2022-01-02 09:01:00), 1.50, 'Chicago', '60605'

See this article for information on managing tables with ADX commands.

You can run the examples in this article in either the ADX Data Explorer web page or in Kusto.Explorer - a rich client Windows application that you can download for free from here.

Syntax

KQL does not require any terminator, such as a semicolon to indicate the end of a command. A blank line between commands is sufficient.

Each clause of a KQL query is separated by a pipe character ("|"). Reading from left to right, the output of each clause serves as the input of the next clause. So, you can apply a filter before you sort before you take 10 rows. Sometimes, the order is important.

Data Source

The first thing in a KQL query is the source of the data you are querying. This can be a table, a materialized view, or an in-memory dataset. Simply typing the name of a table will return all the rows and columns in that table. So, this is a valid KQL query that returns all rows in the customers table.

customers

The results of this simple query are shown in Fig. 1.

Fig. 1

Filtering

Use the where clause to filter the rows returned by a query. The where keyword, followed by a boolean expression tells Kusto to return only rows for which that expression is true.

The following query returns only the customers with YTDSales of at least $100,000:

customers
| where YtdSales >= 100000

The results of this query are shown in Fig. 2.

Fig. 2

Sorting

The order by clause sorts the output of a query. Add to your query "order by ", followed by a comma-delimited list of columns or expressions. The output will be sorted in the order specified. In case of a tie for the first expression in the order by list, the output will be sorted by the second expression and so on. By default, sorting is in descending order. To explicitly specify in which direction to sort, add "asc" (for ascending order) or "desc" (for descending order) following the column or expression.

The following query sorts the customers by PostalCode in ascending order, then by LastOrderDate in descending order:

customers
| order by PostalCode asc, LastOrderDate

The results of this query are shown in Fig. 3.

Fig. 3

Limit Rows Returned

Use the take keyword to limit the number of rows returned. This can be helpful to reduce the time a query executes or for doing a "TOP-N" query, when combined with the order by clause.

The following query returns the customers with the top 3 YtdSales values:

customers
|order by YtdSales desc
| take 3

The results of this query are shown in Fig. 4.

Fig. 4

Limit Columns Returned

By default, KQL will return every column in the source dataset.

Use the project keyword to select which columns to return. The syntax is "project" followed by a comma-delimited list of column names or expressions.

The following query returns only the FullName and LastOrderDate columns of every row in the customers table:

customers
| project FullName, LastOrderDate

The results of this query are shown in Fig. 5.

Fig. 5

Calculated Columns

You can add expressions to your projections by simply adding to the list and (optionally) assigning a name to the calculated column. Here is an example:

customers
| project FullName, LastOrderDate, NextDay=LastOrderDate + 1d, YtdSales, RemainingQuota=2000000-YtdSales

The results of this query are shown in Fig. 6.

Fig. 6

Another way to add columns is using the extend keyword. This is useful if you want to use a calculated value as input into another expression. Here is an example:

customers
| project FullName, LastOrderDate, NextDay=LastOrderDate + 1d, YtdSales, RemainingQuota=2000000-YtdSales| extend expectedSales = RemainingQuota * 2

The results of this query are shown in Fig. 8.

Fig. 7

The extend keyword is also useful if you want to add an extra output column and keep all the existing columns, without bothering to re-type them.

Here is an example:

customers
| extend NextDay=LastOrderDate + 1d

The results of this query are shown in Fig. 8.

Fig. 8

Aggregation

Use the summarize keyword to aggregate data. The syntax is

| summarize by

where

• is a comma-delimited list of aggregation functions
• is a comma-delimited list of columns on which to group. Only one row will be returned for each unique combination of these column values.

Here are some useful aggregation functions to use in your expressions

You can see a full list of aggregate functions here.

Here is an example, showing the numer of customes and average YTDSales for each Postal Code in our table.

.drop table customers
customers
| summarize count(), avg(YtdSales) by PostalCode

The results of this query are shown in Fig. 9

Fig. 9

Limits

KQL has a hard limit of 500,000 rows and 64MB. In other words, even if your query would return a million rows, Kusto will throw an exception. If you are writing queries that return this much data, it may serve you to rethink your filters and what information you are trying to extract.

A Note About SQL

If you are already familiar with SQL, Kusto provides a quick way to convert SQL queries into Kusto queries.

Simply type “explain”, followed by a SQL query and ADX will output a corresponding KQL query

Conclusion

This article discussed some of the basic concepts and syntax of KQL queries. In a future article, we will cover some advanced functionality and functions.

In the last article, I walked you through a wizard that created an ADX table and populated it with data. you can also create and manage tables from a command line interface.

You can run the examples in this article in either the ADX Data Explorer web page (Fig. 1) or in Kusto.Explorer - a rich client Windows application that you can download for free from here.

Fig. 1

Before you get started, you must create an ADX Cluster and create an ADX database.

Creating a Table

The .create table command allows you to create a new table. The syntax is:

.create table tablename
(
column1:datatype1,
column2:datatype2,
column3:datatype3,
.
.
.
columnN:datatypeN,
)

where columnN is the name of a column and datatypeN is the data type of that column. You will add as many column:datatype entries as columns to add.

For example, use the following to create a table named customers with 3 fields: FullName, YtdSales, and LastOrderDate

.create table customers
(
FullName:string, 
LastOrderDate:datetime,
YtdSales:decimal
)

You can view the structure of this table with the following command:

.show table customers

The output of the command is shown in Fig. 2.

Fig. 2

Ingesting Data

There are multiple ways to ingest data into a table. To ingest values from the command line, use the .ingest command.

The syntax is:

.ingest inline into table tablename <|
list_of_data_for_records

where:

• tablename is the name of the table into which you wish to insert data
• list_of_data_for_records is a comma-delimited list of values to insert into the table. The list of values should be in the same order as the columns in the table. Repeat this list to insert multiple rows.

For example, the following command inserts two rows into the customers table we created above:

.ingest inline into table customers <| 
'David Giard', datetime(2022-01-10 09:00:00), 10000
'Tim Giard', datetime(2022-01-09 10:30:00), 100000

Modifying a Table

If you try to execute the .create table command above twice, it will not succeed the second time. The table already exists and it cannot be created a second time.

However, you can use the .create-merge table to modify an existing table. If the table does not already exist, it will create the table with the structure you specify. If the table exists, it will add any new columns listed.

Here is an example of the .create-merge table command that will add two new columns (“City” and “PostalCode”) to the customers table created above.

.create-merge table customers

(
FullName:string, 
LastOrderDate:datetime,
YtdSales:decimal,
City:string,
PostalCode:string 
)

The .create-merge table command is not destructive, so it will not remove any data (records or values) currently in the table. This command does not allow you to modify existing columns. You can rename a column with the .rename column command. The syntax is:

.rename column tablename.old_columname to new_columname

where:

• tablename is the name of the table in which the column exists
• old_columname is the current name of the column
• new_columname is the name to which you wish to rename the column.

The following command will rename the PostalCode column in the customers table to 'ZipCode'.

.rename column customers.PostalCode to ZipCode

Remove a table

If a table is no longer useful or if you just plain don't like it any more, you can remove it with the .drop table command.  The syntax is

.drop table tablename

where tablename is the name of the table to drop.

Drop the customers table with the following command:

.drop table customers

Be careful. The table is dropped immediately without warning or a confirmation prompt.

Conclusion

I have shown you many of the most useful commands for managing an ADX table. You can learn more at the following links:

Table Management Commands
Column Management Commands

Azure Data Explorer (ADX) is a scalable service that allows you to manage large data sets in near-real-time.

ADX Data is stored in tables, which are stored in a database, which is associated with an ADX Cluster. This article describes how to create a new ADX table and populate it with data. See the previous articles in this series to learn how to create a cluster and/or a database.

This article assumes that you have an Azure subscription, an ADX cluster, and an ADX database. See the previous articles in this series to learn how to create an ADX cluster and/or database.

Navigate to the Azure portal and log in; then, navigate to your ADX cluster, as shown in Fig. 1.

Fig. 1

Click the [Databases] button (Fig.2) in the left navigation menu to open the "Databases" blade, as shown in Fig. 3.

Fig. 2

Fig. 3

All the databases in your cluster are listed. Click on the database in which you wish to create a table. This displays the database details, as shown in Fig. 4.

Fig. 4

Click the [Query] button (Fig. 5) at either the top or the left to open the "Query" blade, as shown in Fig. 6.

Fig. 5

Fig. 6

Right-click the database name to display the context menu, as shown in Fig. 7.

Fig. 7

Select "Create new table" from the context menu to display the "Create new table" dialog. By default, the dialog displays the "Destination" tab, as shown in Fig. 8.

Fig. 8

At the "Cluster" dropdown, select the ADX cluster in which you wish to create the table.

After selecting a cluster, the "Database" dropdown lists all the databases in that cluster. Select the database in which you wish to create the table.

At the "Table name" field, enter a name for the table to create. This name must be less than 261 characters and can only contain letters, numbers, spaces, dots, dashes, and underscores. Of course, it must also not be the same as an existing table in the same database.

Click the [Next:Source] button to advance to the "Source" tab, as shown in Fig. 9.

Fig. 9

At the "Source" dropdown, select from where you plan to import data. You can import from a blob in Azure Blog Storage, from a file on your local hard drive, from all the blobs in a given Azure Blob Storage Container, from an Azure Data Lake, or from an Event Hub. In this example, I will import data from a CSV file on my local disc.

A the "Files" field, click the arrow icon, to display the "File Open" dialog, as shown in Fig. 10.

Fig. 10

Navigate to the folder containing your data file, select it, and click the [Open] button to close the dialog and begin importing the data into your table. This may take some time, depending upon the amount of data you are importing.

When the data has finished importing, click the [Next:Schema] button to advance to the "Schema" tab, as shown in Fig. 11.

Fig. 11

The "Schema" tab displays a preview of the first few rows of your imported data.

The "Compression type" field displays whether or not the data is compressed.

At the "Data format" field, select the format in which the data is stored. My sample file happens to be a comma-separated-values (CSV) file format.

If your data contains a header row, check the "Ignore the first record" checkbox.

Click the [Next:Create table] button to commit the creation of the table with all the imported data. When this is complete, the "Data ingestion completed" dialog displays, as shown in Fig. 12.

Fig. 12

You now have a table pre-populated with data to which you can query or append more data.

Decades ago, the Komarrans invaded the planet Barrayar and dropped nuclear weapons on the region of Vashnoi, rendering it uninhabitable before being driven off by the Barrayarans.

In "The Flowers of Vashnoi", Lois McMaster Bujold tells the story of attempts to cleanse the blighted region by introducing genetically-engineered, radiation-eating insects, developed by scientists Enrique Borgos and Ekaterin Vorkosigan. Ekaterin is the wife of Lord Miles Vorkosigan, the main protagonist of Bujold's generations-spanning Vorkosigan saga; and Enrique, along with Miles's clone/brother Mark first created a version of these insects labeled "Butterbugs" in a failed attempt to create an inexpensive food source and a business venture.

On a trip to Vashnoi, Enrigue and Ekaterin discover that children are living in the radioactive area and that these children are collecting their bugs - attracted by the glowing marks on their backs.

Soon, the scientists are seeking to save not only the environment, but the children who are being slowly poisoned by the radiation of their environment.

"Vashnoi" is a novella of approximately 100 pages, but it is more than a filler story in the science fiction series. The story reveals more of Ekaterin, who was introduced in the novels "Komarr" and "A Civil Campaign". She proves one of the more interesting characters in this collection and her late entry has given the readers little insight into her personality.

As usual, Bujold combines plot twists with character development in creating a believable and entertaining story. In this case, these elements are simplified to accommodate the novella's brief length.

Although "Flowers" stands on its own, it is more enjoyable as an aside to those already familiar with the Vor universe. The story of the Butterbugs takes place in "A Civil Campaign" and the nuclear war was originally described in "Memory".

If you are invested in this universe, this story is well worth your time.

Azure Data Explorer (ADX) is a scalable service that allows you to manage large data sets in near-real-time. Data within ADX is stored within databases, so you will need to create a new database before working with any data in this service.

To create a database, navigate to the Azure Portal and sign in; then, navigate to your Azure Data Explorer Cluster, as shown in Fig. 1. If you have not yet created an ADX cluster, follow the steps in this article.

Fig. 1

Click the [Databases] button (Fig. 2) in the left menu to display the "Databases" blade, as shown in Fig. 3.

Fig. 2

Fig. 3

Click the [Add database] button (Fig. 4) to display the "Create New Database" dialog, as shown in Fig. 5.

Fig. 4

Fig. 5

At the "Database name" field, enter a name by which you will identify this database. This name should be descriptive of the data you intend to store herein. It must be unique within this cluster, it must be no more than 260 characters and must only contain letters, numbers, spaces, dashes, and dots.

At the "Retention period" field, enter the number of days you wish to retain data in this database before it is purged. If you never want data purged, check the "Unlimited days for retention period" checkbox.

At the "Cache period" field, enter the number of days you wish to keep data in cache after it is added to a cache. Caching data makes queries against that data much faster. If you wish to keep data in cache forever, check the "Unlimited days for cache period" checkbox.

Click the [Create] button to create the database.

After a few seconds, the database is created and the "Databases" blade displays with the new database listed, as shown in Fig. 6.

Fig. 6

Click on your database row to display details about your database, as shown in Fig. 7.

Fig. 7

From this details blade, you can edit and manage the database.

In this article, you learned how to add a new database to an Azure Data Explorer Cluster.

Azure Data Explorer (ADX) is a scalable service that allows you to manage large data sets in near-real time.

The first step in working with ADX is to create an ADX Cluster. A cluster is a collection of servers that work together to ensure scalability and redundancy in data storage and processing.

This document walks you through the process of creating an ADX Cluster.

Navigate to the Azure Portal and sign in.

Click the [Create a resource] button (Fig. 1); then, search for and select "Azure Data Explorer", as shown in Fig. 2.

Fig. 1

Fig. 2

The "Azure Data Explorer" summary page displays, as shown in Fig. 3. This page contains information about ADX as well as links to help you learn more.

Fig. 3

Click the [Create] button to open the "Create an Azure Data Explorer Cluster" blade, as shown in Fig. 4.

Fig. 4

At the "Subscription" dropdown, select the subscription that should contain this cluster.

At the "Resource group" field, select or create a resource group for this cluster.

At the "Cluster name" field, enter a unique name for this cluster.

At the "Region" field, select a region in which to create the cluster. Consider the location of users and any external data you may need to access when deciding on a region.

The "Workload" dropdown displays a list of the types of workloads you may want to run, as shown in Fig. 5.

Fig. 5

This list is divided into 3 categories: Development, Standard, and Specialized.

Development contains only one option ("Dev/Test") which creates a cluster with only one node. This will not be very scalable, so it is likely not appropriate for a production environment. However, it is an inexpensive way for you to learn the features of ADX.

Standard contains three options: "Storage optimized", "Compute optimized", and "Heavy compute". Select "Storage optimized" if you don't plan to do a lot of computational work on your data. Select one of the other two if you expect to perform many calculations on your data. The "Heavy compute" option provides the most computer power.

After you select the "Workload", select the appropriate size of the machines in your cluster, as shown in Fig. 6.

Fig. 6

Machines with more cores are more powerful, but cost more.  Options in this dropdown are dependent on the workload you select.

The "Compute specifications" dropdown (Fig. 7) will suggest a specific VM that is appropriate for your chosen workload and size. You can accept this or choose a different machine.

Fig. 7

At the "Availability zones" dropdown (Fig. 8), you have the option to deploy your cluster to multiple availability zones. Availability zones are data centers within the same region that are separated geographically and do not have a single point of failure between them. This option costs extra and is not required.

Fig. 8

Click the [Review + create] button (Fig. 9) to display the "Review+Create" tab, as shown in Fig. 10.

Fig. 9

Fig. 10

This tab displays the options you selected for your cluster and lists any problems, such as missing required fields.

Click the [Create] button to begin creating this cluster. This process will take several minutes. When the cluster is created, a message displays, as shown in Fig. 11.

Fig. 11

Click the [Go to resource] button to navigate to the Azure Data Explorer Cluster "Overview" blade, as shown in Fig. 12.

Fig. 12

From here, you can add and manage data in your cluster.

In this article, you learned how to create a new ADX cluster. In the next article, I will show you how to add a database to the cluster.

Azure Data is Explorer (ADX) is a service designed to analyze large amounts of structured and semi-structured data in near real-time. It was developed in 2015 for use by internal Microsoft teams but was used in external products and generally released in 2019.

ADX has several advantages:

• Because ADX is designed for insertions and not for updates, inserting data is very fast, making it ideal for processes, such as IoT applications or Logging that need to absorb data frequently. An absence of database constraints, such as unique primary keys and parent-child relationships make insertions very fast.
• ADX can store massive amounts of data. Internally, Microsoft uses ADX to store huge sets of data, such as Microsoft Office Client telemetry data and Azure Activity Logs.
• More and/or larger servers can be added to an ADX cluster to meet increased workloads by scaling up or out. Clusters can be scaled back down when workloads decrease to reduce costs. Servers in an ADX Cluster are automatically synchronized. There are even autoscale options to automatically add and remove servers based on the workload.
• Because ADX is a managed service, the low-level tasks of managing the servers and other infrastructure are abstracted away from you, so you can focus on the data you are collecting and analyzing.

ADX ships with the Kusto Query Language (KQL) - a declarative language that allows you to not only sort, filter, and return columns, but is ideal for querying time series data. KQL also includes a number of useful functions to perform tasks, such as analyzing geographic data and visualizing data. The Azure Portal allows an interface to manage clusters and execute queries, but KQL queries can also be integrated into reporting tools, such as Power BI and Tableau.

KQL is a read-only query language, so there is no risk of data corruption through injection attacks.

ADX is capable of quickly returning results from very large data sets.

This article provides informatio on ADX and how it is able to achieve its speed and scalability.

In the next few articles, I will show you how to get up and running using Azure Data Explorer.

12/6
Today I am grateful for an early morning latte.

12/7
Today I am grateful to attend a class yesterday, where I learned to make my first gingerbread house.

12/8
Today I am grateful for the taste of Häagen-Dazs ice cream.

12/9
Today I am grateful for good dental care.

12/10
Today I am grateful to attend Happy Hour with the Lincoln Park Ski Club last night.

12/11
Today I am grateful to attend a cocktail mixing class yesterday.

12/12
Today I am grateful for lunch with Nick and Adriana yesterday.

12/13
Today I am grateful to hear the Omnibus String Quartet perform the music of Haydn, Beethoven, and Rachmaninoff in Highland Park last night.

12/14
Today I am grateful for a fresh hair cut.

12/15
Today I am grateful for Day 1 of my vacation.

12/16
Today I am grateful to see a Ducks-Kraken NHL game in Anaheim last night.

12/17
Today I am grateful to attend an exciting Chiefs-Chargers game in Los Angeles last night. Mahones hit Kelce for a 35-yard TD to win in overtime!

12/18
Today I am grateful for:
-a tour of Warner Brothers Studio
-an amazing concert featuring Béla Fleck, Sam Bush, and others
-my first visit to the Getty Museum
-the panoramic views along Mulholland Drive

12/19
Today I am grateful for:
-Live music at the Viper Room last night
-Margaritas on the Venice Beach boardwalk in the afternoon

12/20
Today I am grateful for:
-a bike ride along the ocean from Marina del Rey to Pacific Palisades
-a walk along the Santa Monica pier
-seeing "Spider-Man: No Way Home" in IMax at Grauman's Chinese Theatre

12/21
Today I am grateful for a walk around Los Angeles yesterday, including visits to Little Tokyo, Grand Central Market, Angels Flight, and Union Station.

12/22
Today I am grateful for a drive along the Pacific Coast Highway yesterday.

12/23
Today I am grateful for a lovely week in Los Angeles with no responsibilities.

12/24
Today I am grateful to meet my friend's daughter yesterday.

12/25
Today I am grateful for dinner with Diane and her family last night.

12/26
Today I am grateful to spend Christmas with my family.

12/27
Today I am grateful for a weekend in Michigan.

12/28
Today I am grateful to stream movies at home.

12/29
Today I am grateful for the first snowfall of the winter.

12/30
Today I am grateful to spend the day scheduling and attending doctor appointments

12/31
Today I am grateful for:
- a long conversation with Randy yesterday.
- an exciting come-from-behind victory last night by the Spartans in an NY6 Bowl Game

1/1
Today I am grateful for a New Year's Eve dinner with my sons in Wicker Park last night.

1/2
Today I am grateful for the hope of a better year in 2022!

Captain Vorpatril's Alliance continues Lois McMaster Bujold's Vorkosigan Saga but includes very little about the series protagonist Miles Vorkosigan. Instead, this book focuses on Miles's cousin Ivan Vorpatril. In previous novels, Ivan was presented as an irresponsible womanizer, sometimes referred to by the insulting moniker "Ivan-You-Idiot". But he is the hero of this story.

When rival houses conspire to overthrow one of the galaxy's royal families, assassins target the family's beautiful daughter Tej and her genetically engineered half-sister, Rish. Fortunately for them, Ivan is ordered by Byerly Vorrutyer of Imperial Security ("ImpSec") to protect them. Ivan rescues the two ladies and hides them from the bounty hunters on their trail; then, thinking quickly during a raid by immigration officials, he marries Tej as a temporary ploy to prevent her arrest, deportation, and likely assassination. The real action starts when they return to Ivan's home planet of Barrayar, where they attempt to annul their marriage of convenience and become involved in clandestine operations that cause conflict between the Barrayaran Empire and others.

This is a romantic comedy that mostly features Ivan and Tej. But we also get Byerly and Rish as a couple, along with Ivan's mother Alys and retired ImpSec chief Simon Ilyan. But it is more than that, as Bujold twists the plot multiple times.

Tej is similar to Ivan in that she is a stabilizing force in a manic royal family. Like Ivan, she lacks the ambition and chaos that drive her relatives.

Everyone assumes that Tej's family is dead until they show up on Ivan's home planet with a secret plan to regain their fortune and their titles.

This was a delightful story that is more lighthearted than most books in the Saga. In the past, Ivan was a minor character, presented mostly as comic relief or as a contrast to Miles's driven nature. Here, we see him fleshed out and struggling with his duties and his feelings. We see his loyalty and his resourcefulness. And we understand why Ivan has always stayed under the radar and appeared to lack ambition.

Bujold's wit takes this story beyond a simple space opera and makes it exciting and fun.

I needed a break from work, and I desired a break from the Chicago winter. So, I settled on Los Angeles as a vacation destination. A week in southern California seemed the right recipe for recovery.

Wednesday

I arrived Wednesday evening and drove straight to Anaheim to attend an NHL hockey game. The Ducks played the newly-formed Seattle Kraken in a matchup that was fun if one-sided (Anaheim won 4-1). The Ducks scored a rare shorthanded goal and each team scored on the power play. I had hoped to meet my friend Hattan for a drink after the game, but an illness in his family forced him to cancel.

Thursday

Thursday afternoon, I met my cousin Bob and his partner Christine for lunch. It has been over 3 years since we have been together, and it was a delight to reconnect. The manager stopped by our table, looking for the owner of an illegally parked car that was preventing anyone from leaving the restaurant parking lot. I was not the naughty parker he was seeking, but I earned his gratitude and a free appetizer by moving my car, allowing others to exit.

Thursday evening, I attended an exciting Chargers-Chiefs game at SoFi Stadium. I had excellent seats at this state-of-the-art venue, and I witnessed one of the most exciting games I have seen in years. It was close throughout and the Chiefs won in overtime on a 34-yard touchdown pass from Patrick Mahomes to Travis Kelce.

On the way home, I stopped at Randy's Donuts, a small stand topped with a giant donut statue that has been featured in numerous movies, TV shows, and music videos.

Friday

Friday was my busiest day of the week. It began at the Getty Museum. Much of the museum was closed for renovation, so we did not stay long and headed north for a meandering drive along Mulholland Drive to take in its scenic vistas overlooking the city. The afternoon included a tour of Warner Brothers Studio. It was interesting, but much of the tour focused on current TV shows with which I was unfamiliar. I would have preferred hearing about a broader time period among the studio's history, which spans over 100 years. In the evening, we attended a concert at the Ace Hotel Theatre featuring bluegrass musician Béla Fleck and a set of very accomplished musicians. It was an excellent show, which you can read about here.

Saturday

After such an eventful Friday, we decided to take it easy on Saturday, walking along Venice Beach and stopping for margaritas and nachos at a seaside restaurant. By the evening, we were ready to go, so we visited The Viper Room to hear 3 different local bands perform. The music was good, and we sat with some interesting strangers - one was a former on-air personality at the local NPR radio station; another did set designs for some famous movies; and another was a sales manager for Salesforce.

Sunday

I planned to attend the Rams-Seahawks game Sunday, but a COVID outbreak among the team forced the NFL to postpone the game to Tuesday evening. Plan B was to rent bikes and ride from Venice Beach to the Pacific Palisades and walk along the Santa Monica boardwalk.

In the evening, we went to Hollywood to watch "Spider-Man: No Way Home" on the IMAX screen at the famous Grauman’s Chinese Theatre. The movie was great, and we were able to take in the touristy side of Hollywood, including handprints in concrete and stars on the Boulevard.

Monday

Monday afternoon was spent walking around the area near downtown Los Angeles, including the following:

• A ride up Angels Flight, a cable car traversing a steep city block. It claims to be the world's shortest railroad
• Little Tokyo - a Japanese neighborhood with many shops and cafes
• Union Station - the largest train station in the west. Beautiful architecture.
• El Pueblo de Los Angeles - a nice looking plaza with some history of the era when Mexico ruled California
• Grand Central Market - a bunch of food stands under one roof. I had a taco.
• Bradbury Building - a beautiful old building that has appeared in numerous movies. Sadly, it was closed, but we walked around the outside, admiring the façade.

Tuesday

The flight home was scheduled for Tuesday evening, so we took a drive along the Pacific Coast Highway. As you might guess, this is a highway that hugs the coast of the Pacific Ocean, yielding some spectacular views. We drove up to Malibu and tried to get to Pepperdine, but the campus was closed, so we sat on the beach sipping margaritas and enjoying the view before driving to the airport.

In a surprise move, my ticket was the last one upgraded to First Class, marking a luxurious end to a wonderful vacation. In a more surprising move, I gave the upgraded ticket to my travel companion.

Overall

The hotel in Marina Del Rey was nice and featured a tasty - if overpriced - breakfast each morning.

I was happy to get away from the stress and the cold of my day-to-day life for a few days and sleep in and relax and enjoy the sights. It was good to see parts of Los Angeles that I had never visited; it was good to visit my cousin; and it was good to escape.

It has been a few years since my last visit to LA and I hope to return. Maybe to finally see that Rams game; maybe during baseball season.

The Theatre at the Ace Hotel in downtown Los Angeles is decorated with ornate trimmings that look like stalactites and stalagmite protruding from a cave; but it was easy to forget the unusual decor when Béla Fleck and his friends stepped onto the stage Friday night and began to play.

Fleck was joined in concert by  Sam Bush (mandolin and fiddle), Jerry Douglas (lap steel guitar), Stuart Duncan (fiddle), Edgar Meyer (bass), and Bryan Sutton (guitar). This group has for years been the house band at the annual Telluride Bluegrass Festival and recently collaborated on Fleck's excellent album "My Bluegrass Heart", from which the band drew over half of this evening's songs.

It was a joy to watch and hear six outstanding musicians who played so well together. Sutton especially stood out with his amazing fingerwork and his velvet-smooth vocals; Meyer was creative in his playing, alternating with and without a bow; and Bush projects a joy in playing music as a little boy projects the joys of Christmas morning presents. Although he is one of the world's great banjo players, Béla Fleck was content to step back and let the spotlight shine on his bandmates and their solos.

I first began following Fleck during his days with the New Grass Revival in the 1970s and 1980s, but this was the first time I experienced him in concert. Sam Bush, who appeared onstage Saturday with Fleck, was also a part of TNGR. I saw him perform in northern Kentucky almost 20 years ago and was surprised how little he had aged in the years between.

Most of the evening's songs were written by Fleck, but they graced us with a few well-done covers, including John Hartford's "Up on the Hill Where They Do the Boogie", Bill Monroe's "White House Blues", and Gordon Lightfoot's "Cold on the Shoulder" (although the latter was sung and played with a bluegrass arrangement as a tribute to Tony Rice, who passed away a year ago).

The band's 2 sets totaled more than 2 hours, even before they returned for an encore, which began with the New Grass Revival Song "When the Storm is Over".

It was a remarkable evening of musicianship.

Episode 688

Tudor Damian on Pandemics and Security

Security expert Tudor Damian describes the increased cybersecurity risks brought about by the recent global pandemic.

In the universe of Neil Gaiman's "American Gods", pagan gods exist and walk the Earth in material form. They are called into existence when someone begins worshipping them and their power is proportional to the number of worshippers and the strength of the devotion of those worshippers. They cease to exist only when the world forgets them completely.

America is a nation of immigrants, so it is not surprising that many newcomers brought their gods with them from the old country. However, as those immigrants and their descendants began to assimilate into American society, their belief in the old gods diminished and often disappeared entirely. The old gods were left to wander the continent - immortal yet powerless. In their place, Americans worshipped the new gods of television, industry, and the Internet.

"American Gods" tells the story of Shadow Moon, who became a pawn in the war between the old gods and the new gods. Shadow's wife died just before he was released from prison, so he accepted a job protecting Mr. Wednesday, a former deity now making his way as a con man. As they journey across America, Shadow and Wednesday meet gods, goddesses, mythical creatures, pseudo-government agents, and a walking corpse.

The looming war involves hundreds of gods from dozens of pantheons, each with their own agenda. Like the deities of the ancient world, these gods are full of human weaknesses, such as pride, anger, lust, and pettiness. They fight because they fear that there is not enough faith in the world to sustain them all. They seek a clash of titans to escape their current hand-to-mouth existence.

Gaiman weaves a complex story involving many characters and places and he breathes life into each. He layers his own mythology on top of the legends passed down from ancient Greeks, Scandinavians, Egyptians, Africans, Native Americans, and others. The story may not be true, but it is plausible. Gaiman creates rules for his universe, and he sticks to them.

Some readers will not like the tangential turns that the book sometimes takes - the story of African American slaves invoking their ancestral gods to cope with the horrors of plantation life; the tale of a jinn reduced to driving a taxicab and dreaming of restarting his life; and the tragedy of a child raised alone in darkness until he is sacrificed by his tribe. But these tales add color to the legends of the main story and sometimes they tie in directly with the plot.

"American Gods" is a darkly entertaining story of a con man and an ex-con showcasing the underbelly of America as written by an English immigrant. It is an adventure story and a travelogue and a statement about the fickleness of society.

Even by the standards set by Neil Gaiman's other novels, "American Gods" is a great story!

In the fall of 1987, I took a break from grad school to work as a stockbroker. Between the time I passed my certification exams and the time I started working, the Dow Jones Industrial Average fell 22.6% in a single day - the largest daily percentage drop in history. My career as a stockbroker was short-lived, but I gained a lesson in the risks of working in the investment world and it sparked my interest in the inner workings of Wall Street.

Former investment banker William Cohan has followed this world much more closely than I have and he used his knowledge to write his 2009 book "House of Cards", which tells the story of the collapse of Bear Stearns - a once-successful investment firm that failed and was sold cheaply to JP Morgan Chase in 2008.

Part 1 details the final days of Bear Stearns. BS was once the fifth-largest firm on Wall Street and its stock price closed at $171.51 per share in January 2007; but it was sold to JPMC for just $2 per share 14 months later.

The next section recounts the history of the firm and the significant people contributing to its growth and fighting for power within the organization.

The final section brings together the first two parts by focusing on the actions of individuals in the months leading up to Bear's collapse.

There were many reasons for Bear's downfall and there is plenty of blame to go around. The restrictions on lending had recently been relaxed and the US government encouraged banks to lend money to lower-income families - families that previously could not qualify for a mortgage. The greed of bankers took over as they began making high-risk loans, then selling those loans to investors. As property values fell, securities backed by mortgages on those properties also lost value. Many people defaulted on their mortgages causing further strain.

It did not help that a manager of one of Bear's funds repeatedly lied to his investors - understating the fund's investments in high-risk securities by an order of magnitude.

The book could have been shorter, but Cohan chose to go into detail about the histories, personalities, and eccentricities of many of the Bear executives. It helps to understand the egos (and sometimes the hubris) of the decision-makers who lead the company down this path.

Cohan shines a light on the fragility of the entire industry. Because so much of the securities industry is based on confidence, companies will often overstate their optimism and hide bad news in order to maintain higher prices. Conversely, once investors lose faith, they begin to abandon an investment or fund or firm and the collapse can accelerate rapidly.

Of course, Bear Stearns was not the only casualty of the subprime mortgage crisis. 2008 saw a major upheaval in the entire financial industry, resulting in the failure or weakening of several firms. In his epilogue, Cohan gives a brief account of the fall and bankruptcy of Lehman Brothers, which occurred a few months after Bear Stearns's collapse.

2008 was a disastrous year for Wall Street and contributed to a national recession. The question remains: can we avoid a similar disaster in the future? The book does not answer this question.

Nov 8
Today I am grateful for unseasonably warm Chicago weather.

Nov 9
Today I am grateful to work with my trainer yesterday for the first time in over 3 weeks.

Nov 10
Today I am grateful to see Kalamazoo College play at Elmhurst University last night.

Nov 11
Today I am grateful to all the veterans who have served our country.

Nov 12
Today I am grateful to see the Lemonheads in concert last night.

Nov 13
Today I am grateful for a productive work week.

Nov 14
Today I am grateful to see a home basketball game at Kalamazoo College for the first time in 2 season.

Nov 15
Today I am grateful for brunch yesterday in downtown Chicago with Andra and her husband and friends.

Nov 16
Today I am grateful for new jumper cables.

Nov 17
Today I am grateful for a new belt.

Nov 18
Today I am grateful to see Richard Thompson in concert last night.

Nov 19
Today I am grateful to speak at the Memphis .NET User Group last night.

Nov 20
Today I am grateful for the works of William Shakespeare.

Nov 21
Today I am grateful to see the great Nick Lowe in concert last night with Los Straitjackets.

Nov 22
Today I am grateful to complete my first project in Adobe Premiere Pro yesterday.

Nov 23
Today I am grateful that my city parking sticker finally arrived.

Nov 24
Today I am grateful to hang out with Peter, Shannon, and Michael last night at the Chicago Athletic Association.

Nov 25
Today I am grateful to Tim who treated Natale and me to an excellent dinner at Giant in Logan Square last night.

Nov 26
Today I am grateful for the opportunity to volunteer to make Thanksgiving dinners yesterday for seniors, shut-ins, homeless, and others.

Nov 27
Today I am grateful to celebrate Thanksgiving in Kalamazoo with my boys a day after Thanksgiving.

Nov 28
Today I am grateful:
-to attend an exciting MSU football game in the snow yesterday in East Lansing
-for dinner with Nick last night in Lansing

Nov 29
Today I am grateful for a 4-day weekend.

Nov 30
Today I am grateful for a chiropractor office across the street.

Dec 1
Today I am grateful for the first meeting of the school year mentoring students at Chicago Tech Academy.

Dec 2
Today I am grateful for easy returns and exchanges of online purchases.

Dec 3
Today I am grateful to see a production of "As You Like It" featuring the music of The Beatles on my first visit to the Chicago Shakespeare Theatre last night.

Dec 4
Today I am grateful to find good recipes online.

Dec 5
Today I am grateful to attend a high school basketball game for the first time in years.

William Shakespeare's "As You Like It" is a silly, farcical play about Rosalind - a young girl in exile who dresses as a man and advises young Orlando - also in exile - on how to woo and win the woman he loves. The twist is that Rosalind is the woman with whom he is in love. Among the many subplots: Rosalind’s father Duke Senior has been overthrown and exiled by his brother Duke Frederick; the shepherd boy Silvius is in love with Phoebe, who falls in love with Rosalind, whom she believes is a man; and Duke Frederick’s daughter Celia chooses exile with her cousin/best friend Rosalind against the wishes of her father. It is hard to take it seriously: No one recognizes Rosalind simply because she changes her outfit; multiple couples fall in love at first sight; and the story is wrapped up very quickly with a series of coincidences and changes of heart.

But it is a fun story and the Chicago Shakespeare Theatre had fun with it. They removed some of the dialogue and replaced it with music from The Beatles. They also dressed the characters in 1960s attire and added a pro wrestling announcer.

Some songs fit really well into the storyline: the melancholy Jaques sings "Fool on the Hill" after meeting Touchstone the jester; Audrey and Touchstone - an older couple courting one another sing "When I'm 64"; Rosalind sings "You've Got to Hide Your Love Away" as she hides her identity from and love for Orlando. But mostly, we hear the many love songs written by the Fab Four, or the audience is treated to a song that fits the mood ("While My Guitar Gently Weeps" entertains Duke Senior's exiled court).

The cast was excellent, highlighted by Kevin Guhdal as both Dukes and Kayvon Khoshkam as Touchstone. They were not averse to breaking the fourth wall ("Why are we hiding?" “Because it’s Shakespeare!")

The production breathed new life into an old play. It was a pleasure to see an old story in a new light.

How did I not know about this book before? A Walking Tour of the Shambles was written by two of my favourite authors: Neil Gaiman and Gene Wolfe; and it is about Chicago - the city in which I have lived the past 7+ years.

I travel a lot and I buy a lot of guidebooks that contain suggested walking tours, pointing out sites along the walker's path and this looks very much like a real guidebook. It even has the number "16" in the corner, suggesting that there are others in this same series.

But there are no other guidebooks in this series. It is false. Nearly everything in this book is false. The Shambles is purported to be a neighborhood in Chicago, but no such neighborhood exists. And if it did, it would probably not contain a house full of murderous grandfather clocks; or man-eating crocodiles; or Cereal House, where you must provide the name of your next of kin before daring to spend the night.

Still, Gaiman and Wolfe did their homework. They recommend that tourists visit the home of H.H. Holmes; Holmes was a real person. He was a notorious serial killer who murdered dozens of Chicagoans in the late nineteenth century. But his house of horrors was in Chicago's Englewood neighborhood and has long since been demolished. Chicago does have a Canal Street, but it is clearly not the dangerous Canal Street described in this book.

The House of Clocks does not exist, but it does have its own website.
http://www.preserveusfromthehouseofclocks.com/

While not among their best works, this slim volume is fun and funny, and scary, and it made me smile. Those of us who are fans of Gaiman and Wolfe will enjoy it.

In early 2019, the COVID pandemic interrupted the tour of Nick Lowe and Los Straitjackets, or as Lowe described it: "It was a forced layoff." They recently returned to the road, but only for two weeks, concluding the brief tour Saturday night at Park West.

Other than a brief flash in the late 1970s, Lowe never achieved broad commercial success. But he is known in the music industry as an outstanding songwriter and producer and at 72, his baritone vocals sound as good as ever.

He delighted the packed auditorium, opening with three upbeat rockers ("So It Goes", "Ragin' Eyes", and "Without Love") before slowing things down with the melancholy "Lately I've Let Things Slide", a song that seemed appropriate for those of us struggling with isolation these past two years.

Los Straitjackets were an excellent backing band, staying mostly out of Lowe's spotlight, despite sporting Luchador masks, making them look like Mexican wrestlers in dark suits. Lowe stepped offstage for about 15 minutes mid-concert to allow Los Straitjackets to perform some of the surfer-rock- influenced instrumentals for which they are best known.

Lowe returned to the stage to complete the set, finishing with a rousing version of his classic "I Knew the Bride (When She Used to Rock 'n' Roll)".

He returned for an encore - a moving version of "(What's So Funny 'Bout) Peace, Love and Understanding performed" - the excellent song he wrote that was made famous by Elvis Costello. Surprisingly, he returned for a second encore - a beautiful cover of Elvis Costello's "Alison".

It was an outstanding show. Nick Lowe has not missed a step in the years he has been touring. I last saw him in a solo performance 14 years ago at a small club in Michigan. He was great then; he was great when I first discovered him in my high school years; and he was great Saturday night at Park West.

Episode 687

Nick Kwiatkowski on Dialogflow CX

Dialogflow CX is an AI cloud-based agent from Google. Nick Kwiatkowski of Michigan State University talks about how his team used this tool to build a conversational chatbot to help nurses connect more efficiently with their patients, as well as other projects built on Dialogflow CX.

Smoke and Mirrors collects a few dozen early short stories and poems by Neil Gaiman. Most were published previously in other anthologies.

I enjoyed Gaiman's preface, in which he described the inspiration for each of the selections and I returned to this section repeatedly.

I always enjoy reading Gaiman's poems, but I rarely remember any of them 24 hours after reading them. Not so, with his short stories - many of these stay with me for years. A favourite theme of Gaiman's is to take an existing story and twist its perspective or to mimic the style of an existing author. He does both in these stories, paying homage to H.P. Lovecraft, Michael Moorcock, the Brothers Grimm.

Although I did not love every story in this volume, I cannot say that I hated any of them. All were worth reading. A few of the stories appeared in Gaiman's "M is for Magic" collection, which was published later, but which I read previously.

Below are my favourite stories from this collection.

Chivalry

A story about an old lady, who finds the Holy Grail in a thrift shop. Shortly after, Sir Galahad approaches to attempt to purchase the relic.

Shoggoth's Old Peculiar

A humorous H.P. Lovecraft mystery

We Can Get Them for You Wholesale

A terrifying story about a man, who seeks volume discounts when hiring an assassin

Murder Mysteries

An angel is murdered in Heaven and another angel is charged with finding his murderer and avenging his death.

Snow, Glass, Apples

A version of Snow White in which the stepmother is the victim and Snow is the villain.

The Goldfish Pool and Other Stories

A hopeful English screenwriter tries to navigate Hollywood's facade.

I have been listening to Richard Thompson for decades, but I was in no way prepared for seeing him in concert. But see him I did, as a few tickets popped up hours before a Saturday night show that had been sold out for months. It was good luck for me, as I had bought tickets to see The Wallflowers that same night and an illness in the band forced that show's cancellation.

But sitting in the second row at the Old Town School of Folk Music, I was able to get close to the singer/songwriter/guitarist, who made every effort to relate to his audience. He chatted between songs and joked with the crowd. Thompson published a memoir - "Beeswing: Losing my Way and Finding my Voice, 1967-1975" last year and he read a few passages from it, each time leading into a song.

Hearing the richness of Thompson's baritone voice was a pleasure and watching as he accompanied himself on guitar was impressive. His live performance far exceeds what is captured on his recordings. At age 72, he has lost none of his vocal prowess nor his playing dexterity.

For most of the show, Richard was alone on stage, accompanying himself on guitar; but for a few songs, he invited singer Zara Phillips to sing harmony on the chorus. Phillips's pleasant voice enhanced each song in which she participated.

Thompson sang songs from throughout his career. He included songs from Fairport Convention ("Who Knows Where the Time Goes?", "Poor Will and the Jolly Hangman"), from albums recorded with his ex-wife Linda ("Walking on a Wire", "Down Where the Drunkards Roll"), and songs from his solo career, most notably the crowd favourite "1952 Vincent Black Lightning".

Richard Thompson rose to prominence in the 1960s when his band Fairport Convention helped to launch the folk music scene. He left FC in 1971 and has been recording and touring in the years since, but he has never sought nor achieved the commercial success that his talent deserves.

The couple sitting next to me told me had seen this singer dozens of times and still enjoyed his shows.

Saturday night at Old Town, everyone left satisfied.

More Photos

Episode 686

Allison Hartnett on Microsoft TEALS and Computer Science Education

Allison Hartnett and the Microsoft Technology Education and Literacy in Schools (TEALS) program works to bring more access to Computer Science to students in Grades K-12. She talks about what the program, its partners, and its volunteers are doing to promote CS education among underrepresented groups.

https://microsoft.com/teals
https://hourofcode.org/
https://twitter.com/tealsprogram

Some artists let the music do the talking for them. This was the case with Evan Dando, as he and his band the Lemonheads performed at Thalia Hall Thursday night. The band moved quickly from one song to the next with at most a quick "Thank you" between numbers.

On the one hand, this allowed them to pack as many of their 3–4-minute songs into as short a time as possible. On the other hand, it made it difficult to achieve a connection between the audience and the performers.

Don't get me wrong. The trio (Dando, along with bassist Farley Gavin and drummer Michael Jones) sounded really good. All are accomplished musicians and Dando's voice sounds as powerful now as it did when he was cranking out hits almost 30 years ago. And it's not that the band didn't have energy or that they ignored the audience. They just didn't talk to them. And the concert ended with Dando mumbling "See ya later" less than 90 minutes into his set and walking offstage with no return for an encore.

Everything was abrupt. A half hour into the show, Gavin and Jones walked off stage and Dando picked up a guitar and performed a 20-minute acoustic set, highlighted by the " I Lied About Being the Outdoor Type", "Being Around", and "Frank Mills" from the musical "Hair". Then the band returned and continued as a trio.

The band's songs are typical of the indie bands that major labels signed in droves in the late 1980s and 1990s; their songs feature catchy melodies and clever lyrics layered over driving rhythms and frantic guitars.

Even though the Lemonheads' last two albums featured only cover songs, this evening was mostly about original material. However, the trio did delight with the John Prine song "Speed of the Sound of Loneliness" and finished with "Different Drum", the  Michael Nesmith song made famous by Linda Ronstadt's Stone Poneys.

Overall, it was a good show by a quality band that tried to sound very much like they did on their albums. I enjoyed it, but it wasn't much different from listening to their albums.

Photos

Diplomatic Immunity continues Lois McMaster Bujold's saga of Miles Vorkosigan, the diminutive galactic lord in the distant future. It is Miles's first mission since his marriage to Lady Ekaterin.

The couple are called away from their honeymoon to investigate a violent incident on the edge of the empire. The sector is controlled by "quaddies" - genetically engineered humans with an extra set of arms where one would expect legs.

On a remote space station, Miles is reunited with his old friend Bel Thorne and caught in the middle of a plot in which the attacker is unknown and which potentially drives rival empires on a path toward interstellar war. To make matters worse, Miles is attacked by a bioweapon that infects him with a disease from which he may not recover.

This is a fun adventure, wrapped up in a whodunit that keeps the reader guessing throughout. It is also one of the few stories to feature the Quaddies, a race I grew to enjoy after reading "Falling Free".

GCast 117:

Getting Started with Azure Data Explorer

Azure Data Explorer (ADX) is a service designed to analyze large amounts of structured and semi-structured data in near real time. This video walks you through creating a cluster, creating a database within that cluster, and adding a table with data to that database.

10/4
Today I am grateful to see Melissa Etheridge in concert last night in Waukegan.

10/5
Today I am grateful for kind words from our customer as we wrapped up this project.

10/6
Today I am grateful to download and install Photoshop and Lightroom yesterday and edit some photos with these tools.

10/7
Today I am grateful to finally edit and share photos from the concerts I have attended the past 2 months.

10/8
Today I am grateful for my personal trainer.

10/9
Today I am grateful to see an excellent live performance of "RENT" last night in Chicago.

10/10
Today I am grateful to watch a James Bond movie at an IMAX theatre yesterday.

10/11
Today I am grateful to visit the upper floors of the Harold Washington Library for the first time in almost 2 years.

10/12
Today I am grateful to receive the COVID booster shot yesterday.

10/13
Today I am grateful to finally donate to charity the enormous pile of stuff that has been sitting next to my front door for weeks.

10/14
Today I am grateful that I began working at Microsoft 8 years ago today.

10/15
Today I am grateful that a much-needed vacation begins this evening.

10/16
Today I am grateful to see Pat Metheny put on an excellent performance last night at Thalia Hall in Pilsen.

10/17
Today I am grateful to arrive safely in Dubrovnik after my first international flight in almost 2 years.

10/18
Today I am grateful to fall asleep last night to the sounds of the Adriatic Sea outside my window.

10/19
Today I am grateful to read a book and sip espresso on a mountain overlooking Dubrovnik yesterday.

10/20
Today I am grateful for two walking tours of Dubrovnik yesterday.

10/21
Today I am grateful for
-cave diving in the islands of the Adriatic Sea
-4 days in Dubrovnik

10/22
Today I am grateful to explore the old town of Trogir, Croatia yesterday afternoon.

10/24
Today I am grateful for:
-a tour of an olive oil museum
-a tour of a stone cutting school
-dinner at the home of a local Croatian family last night.

10/25
Today I am grateful for 3 days on the island of Brač.

10/26
Today I am grateful for a cooking lesson and a great meal last night in Hvar Town.

10/27
Today I am grateful for 3 days on the island of Hvar.

10/28
Today I am grateful for a walking tour around Split, Croatia, learning about the history of this city.

10/29
Today I am grateful for 2 weeks in Croatia.

10/30
Today I am grateful for plenty of sleep yesterday.

10/31
Today I am grateful for an exciting victory over um in football yesterday and an 8-0 record heading into November.

11/1
Today I am grateful for the Trick-or-Treaters who came to my door yesterday.

11/2
Today I am grateful for leftover Halloween candy.

11/3
Today I am grateful for my chiropractor.

11/4
Today I am grateful to attend a Roosevelt University home basketball game last night for the first time.

11/5
Today I am grateful that the freeway exits closest to my home are finally open again.

11/6
Today I am grateful for tacos in Wicker Park last night.

11/7
Today I am grateful for one extra hour in the weekend.

There was nothing special about Ana. She was a typical 10-year-old girl growing up in Zagreb, Yugoslavia when the war broke out. But shortly after her baby sister was sent to America for medical treatment, Ana's world was turned upside down. Serbs attacked her family and she found herself alone and thrust into the middle of a civil war.

Years later, Ana is living as an American college student, attempting to reconcile the tragedies of her past with her current life. She returns to Croatia seeking closure and those with whom she was once close.

Sara Nović's debut novel Girl at War tells Ana's story. Nović takes the reader forward and backward in time as we learn of Ana's search for peace. It is a moving story of a young girl denied a normal childhood.

Although this is a novel, it often reads like an autobiography. If the prose sometimes seems a bit awkward, this made it more authentic for me; if the narration is detached, it emphasized Ana's attempts to suppress the memories of her past.

I read this novel while visiting Croatia and I had the privilege of hearing stories of the civil war from the locals and from a museum above Dubrovnik. As a result, the story felt even more real for me.

These days, Microsoft tends to announce products as they begin development; so they have fewer big announcements at their large events like Microsoft Ignite. Nevertheless, yesterday's 2.5 hour keynote did include some interesting information and announcements.

The two things that interested me the most were:

Azure OpenAI Service - a new addition to Microsoft's powerful Cognitive Services suite that allows you to create models that understand the context of text. The demo showed a transcript of a basketball game in which OpenAI was able to summarize the game's action into a brief paragraph. You can watch that demo here.

Microsoft Teams Connect should make it easier to collaborate in Teams with members from different organizations. This has been a pain point on some of my recent projects, so I look forward to seeing how this affects my work.

Microsoft Loop appeared in my build of Teams a few days ago, so my teammates and I have been playing with its collaborative features, which allow us to quickly add things like tables and to-do lists and edit them simultaneously.

While watching videos, I sometimes learn about things that are not new, but are new to me. Copilot is a plug-in for Visual Studio Code that automatically generates code, based on comments you add. I was unfamiliar with it, but I will be trying it out this week. It looks very cool!

Below are my raw notes, taken while watching the keynote:

New: Azure OpenAI Service

Customize models. Add context

Summarize text

Summary of summaries

Co-Pilot VS Code plug-in

Creates code based on comments

Teams Connect

Collaboration: Chat, call, meet, etc.

Metaverse

Digital representation of the world

Dynamic 365 Connected Spaces

Microsoft Mesh

Mesh for Microsoft Teams

Avatars with facial expression

Shared whiteboard

Judson A: Sustainability case study

Teams

Loop

Pages with flexible canvases

Loop components: Pull in content

Keep in sync

Multiple users can edit in Teams, Outlook, Word, or Teams mobile

Outlook

@ mentions

Automatically create meeting

Clip Champ

All your video needs in one place | Clipchamp

Teams Connect

External shared channels alongside internal channels

Collaborate with those outside company

Public Preview 2022Q1

Webinar

Virtual Green Room for presenters to prep

Curated attendee: Attendees see only shared content and guests invited to speak

ISV

Bring your apps into Teams

Bring Teams into your org

Viva Insights

Meeting suggestions provided to organizer

Create a meeting plan

"Send praise" feature

Dynamics 365

Supply Chain Insights

provide risk scores

Projected delivery dates

Storm surge risk

Power Automate

Process step map

Add comments to apps

Azure Arc

Hybrid and multicloud management

Virtual desktop in Azure Stack

SQL Server 2022 Preview announced

Azure Container Apps

Fully-managed and serverless platform

Does not use Kubernetes. Focus on apps

Upgrade path to Kubernetes

GitHub Codespaces

Create development machines in the cloud in <15 seconds

Access via web browser

New: Power Apps pay-as-you-go pricing model

Distribute Power Apps as native iOS and Android apps

Episode 685

Dave Rael on Git

Dave Rael discusses how git version control can make your development easier.

https://tinyurl.com/tpw-git-parable
https://www.linkedin.com/in/daverael
https://amzn.to/3aGsZUd

I had never heard of a bicycling vacation before it was suggested to me a few months ago. But it sounded amazing - spending a week on the Dalmatian Islands off the coast of Croatia - biking about 30 miles each day while eating local food, learning about the local culture, and seeing the sights of a country I had never visited.

So, two weeks ago, off I flew to the southern coast of Croatia.

The organized tour started Thursday. But, I left Chicago Saturday evening and arrived in Croatia Sunday morning.

The trip began in Dubrovnik, an ancient city nestled between towering cliffs and the Adriatic Sea. We planned these days on our own before the tour company bussed us four and a half hours northwest to Trogir - an island city further up the coast. After a day in Trogir, we took a ferry to the island of Brač, where our biking adventure began. We spent 4 days on Brač and 3 on Hvar, bicycling different paths around the island each day. On the final day, we ferried back to the mainland and spent a day exploring the large coastal city of Split before flying home.

Dubrovnik was amazing. I have been wanting to visit since someone recommended it to me decades ago. The oldest part of the city is enclosed by walls built between 1100 and 1700 AD after the departure of the Venetian Empire. The walls were strong enough to repel most invaders and survive a devastating 1667 earthquake. The walled city is a UNESCO heritage site, but that did not prevent the Yugoslavian army from bombarding the city during the 1991 Homeland War, damaging over 60% of the buildings. Happily, the Croatians were able to rebuild the damaged city after gaining independence. We enhanced our trip with a few activities: We scheduled two guided walking tours: one focused on the history of the city; the other on food and wine; and we took a boat ride out to some of the islands off the coast, where we went snorkeling into some natural caves. We spent a couple hours exploring one of the islands. I tried to take some video with my drone, but I broke a propellor by accidentally crashing it into a wall.

I don't know if it was the boat ride, the cave diving, or something else that caused an old back injury to flare up, but it affected me the rest of the trip. I was fine while riding a bike but coughing or standing up too quickly sometimes caused intense pain. Luckily, it did not significantly impair our activities that week.

Trigor and Split are also old walled cities along the coast with rich histories of their own and we spent a day exploring each - hiring a guide to show us Split.

I traveled with a group of 4 people, but our guided tour included 10 other tourists - all of whom knew one another and most of whom had traveled together previously. Our tour guides - Marco and Mario - provided excellent leadership. Each had local connections (Mario was born and raised on Hvar and Marco's wife was from Hvar), so they provided great perspectives about the history, culture, food, and locations. They also took care of many of our logistical issues - making dinner, boat, and hotel reservations and planning our bike routes each day. Many of the Hvar inhabitants are related to Mario, so we received special treatment. The boat driver was his uncle, and the van driver was his brother.

The islands were gorgeous. Mountains jut up from the Adriatic and you can see the sea from nearly every point, with beautiful vistas in each direction. The land was covered with rocks, which farmers have dug up over the centuries and stacked into houses, fences, kilns, or just piles.

This is the end of the tourist season, which meant that many of the restaurants were closed; but the advantage is that there were a lot fewer people now than in the summer. Fewer cruise ships stopping also meant smaller crowds. Most people are not wearing masks here, but that was less distressing because we did not run into a lot of people. Everyone we met was very friendly. The locals were familiar with the VBT tours and waved as we rode past.

The roads were in surprisingly good condition. We did not encounter a lot of traffic, but there were few bike lanes, so we had to be cognizant of cars on the road. We mostly stayed off the main roads but had to cross them from time to time.

The American TV show "The Bachelor" was filming in front of our hotel, so we were able to see and hear a bit of next season's show.

I have never seen a place with so many cats. They roam free in the streets and it is unclear if they belong to any human families.

This was my first time riding an electronic bike (eBike) and it was ideal for this trip. Although it has a motor, it is different from a motorized scooter. An eBike motor assists with pedaling, rather than replacing it. You must continue to pedal, but you can select up to 4 speeds to boost your velocity. This was particularly helpful when biking up a steep hill. About half the trip was level, downhill, or a slight upgrade. I turned off the motor during these times and turned it on when confronted with one of the many uphill grades. The weight of an eBike makes it more difficult when the motor is turned off, but the extra boost came in handy on the steep climbs.

I rode nearly 150 miles over the course of 6 days. Some of the routes were optional, but there was no way I was going to skip any of them.

Fans of the popular TV show "Game of Thrones" would recognize many sites in Croatia. Dubrovnik was the setting for King's Landing, while Split played the part of Meereen. They offer tours that focus on filming locations for the show, but we were not interested.

This was the longest vacation trip I have taken since 1988 and it came at a time when I desperately needed a break from work. I was impressed with VBT - the trip organizers and I am already considering a similar trip next year to a different part of the world. I took a lot of photos, which you can see here.

Here is how I spent each day:

OCTOBER 17
Arrived in Dubrovnik
Watched the sunset over the Adriatic Sea from a nice restaurant just outside the city walls
Walked around the old city at night

OCTOBER 18
Took a cable car to the cliffs above Dubrovnik. Sipped coffee, read, and watched the sunset from a cafe at the top of the cliff
Visited a museum dedicated to the 1991 Homeland War which broke up Yugoslavian and granted independence to Croatia

OCTOBER 19
Attended an organized walking tour of the old city, learning the history of Dubrovnik and Dalmatia
Circumnavigated the old city atop the surrounding walls
Attended a second walking tour - this one visiting old restaurants and focusing on the food and wine of the region.

OCTOBER 20
Took a boat ride out to the islands off Dubrovnik's coast.
Dove into two natural caves on the islands.

OCTOBER 21
Bus ride to Trigor, which passed briefly through Bosnia.
Walked around the old city of Trigor, including a trip to the top of the St. Lawrence Cathedral tower, from which one can see most of the city.

OCTOBER 22
Ferry to Brač.
6-mile warmup bike ride to and from Dol
Return to Dol for dinner in a thousand-year-old building. Rakija/grappa tasting (walnut, herb, cherry, olive, etc.)

OCTOBER 23
32-mile bike ride. Stops in:
Skrip: Tour olive oil museum. Tasting olive oil, fig jam, olive spread
Puciska: Harbor town. Tour of stone cutter school.
Dinner with a local family - a brother and sister who served homemade rakija.

OCTOBER 24
34-mile bike ride.
Started at the highest point in Croatia. Rode down to the coast; then along the coast back to the hotel.
Lunch + Turkish coffee in Milna by the harbor
Gelato in Supetar - the largest city on Brač

OCTOBER 25
11-mile bike ride in the morning, beginning at the highest point in Croatia
Hike along the shore
Boat ride to Hvar
Checked into a beautiful old hotel, overlooking the harbor
Cooking class

OCTOBER 26
29-mile bike ride, crossing the entire island - west to east and back again.
Stopped at a beach to swim

OCTOBER 27
35-mile bike ride, again across the entire island, but also including an optional ride over the mountain back to the hotel.

OCTOBER 28
Took a morning ferry to Split
Walked around the old part of the city and enjoyed a late lunch at an outdoor cafe.
Hired a tour guide in the evening to show us around the old city, which was formerly the palace of the third-century Roman Emperor Diocletian, who abdicated his throne in 305AD.

OCTOBER 29
Woke up at 5 AM to catch a 7:30 flight to Munich, then change planes to head home.

OCTOBER 30
Slept 10 hours in my own bed.

More Photos

An Acceptable Time concludes the "Time Quintet" that began with Madeleine L'Engle's classic novel "A Wrinkle in Time".

The series follows the Murrys - a family that are frequently thrust into adventures that span multiple planets, dimensions, and eras.

In this story, Meg's daughter Polly is staying with her grandparents in New England when their neighbor retired Bishop Colubra discovers a "time gate" - a doorway to three thousand years in the past. Polly and friends find themselves trapped in the past, caught between two Native American tribes at war.

As in so many of her books, L'Engle spices the story with religious references and philosophies. She never overwhelms the reader with preaching or dogma; rather, she raises questions and shows different relationships between men and God. Some worship a single creator, while others see gods in their ancestors or in each force of nature; some believe in a loving, parental God, while others fear a vindictive god, who demands blood sacrifices. Unlike stories in the Old Testament, the snake featured in this story is a force for good. I found it interesting that the Christian Bishop was accepting of the beliefs of druids. They had fundamental theological differences, but they chose to focus on their common beliefs.

There are a few weaknesses. The skepticism of Polly's grandparents is inconsistent with their previous encounters with time and space travel; and conflicts are resolved a little too conveniently at the end. Coincidences occur that border on a well-timed miracle. It stretches belief, but that is probably ok in a book about time travel.

An Acceptable Time is a story of selfishness and betrayal. But it is also about forgiveness and redemption and the different aspects of love. It is a story of hope and a fitting end to this series.

Episode 684

Guy Royse on Probabilistic Data Structures

Guy Royse discusses Probabilistic Data Structures - a pattern that results in high performance, but with a tradeoff in accuracy. He describes when such a tradeoff makes sense and how to implement this pattern in tools like Redis.

http://guyroyse.com/

Miles Vorkosigan is in love with the recently-widowed Ekaterin Vorsoisson. Several obstacles inhibit his hopes at courtship: She is still in mourning for her late husband; her husband was abusive toward her; and Vorkosigan enemies are spreading rumors that Miles was implicit in the husband's death.

In her science fiction / romantic comedy "A Civil Campaign", Lois McMaster Bujold takes the reader through this adventure. She intertwines the main story with several others - a noblewoman who undergoes gender reassignment surgery in order to claim an open Countship (only males are eligible for this honor); a business started by Miles's clone/brother Mark to produce and market dairy products from genetically-engineered insects; and the upcoming wedding of the Galactic Emperor. Bujold cleverly ties together all these stories in the end.

This is mostly a lighthearted romantic comedy; but it also tackles some serious issues around gender roles. Bujold provides us with numerous examples of strong, independent women trying to navigate a patriarchal society. And there is moral philosophy: "Reputation is what other people know about you. Honor is what you know about yourself."

It has been a couple years since I set aside Bujold's series of books on the future universe inhabited by Miles Vorkosigan. I am glad I did. Reading this book felt like a reunion with old friends.

Sharp Objects by Gillian Flynn is the story of how a tragic murder can affect a small town. But it is also the story of a young woman's struggles to confront the past she tried to escape.

Camille Preaker was a Chicago reporter sent to cover the story of a child killer in her hometown of Wind Gap in southern Missouri. Returning after years away, Camille reopens wounds caused by her relationship with a mother who never loved her, memories of a dead sister, and regret of the person she used to be. The longer she stays in this town, the more she reverts to her teenage self - repeating the mistakes of her youth.

Camille is isolated and self-destructive - both emotionally and physically, which makes it difficult to investigate the case, challenging to live a normal life, and impossible to form meaningful relationships. Flynn takes the reader inside the head of the character, making us feel for her, even if we cannot identify with her or like her. Camille grew up with a twisted sense of morality. At one point, she rationalizes statutory rape - a gang rape, at that.

The narrative is often uncomfortable. Wind Gap is filled with narrow-minded people a stereotype often depicted for those in a small town.

This was Ms. Flynn's debut novel - written well before she gained fame with "Gone Girl". It is not for everyone. The topic is disturbing, and the characters are deeply flawed. But I liked her depiction of the damaged prodigal daughter struggling against her past.

I found it entertaining.

Episode 683

Richard Campbell on the Future of Space Travel

Richard Campbell discusses the history of private space travel and where this industry is headed.

Some musicians have extremely high technical proficiency; others bring a great deal of emotion to their performance. Pat Metheny is the rare one who has achieved both.

He proved it Friday evening at Pilsen's Thalia Hall. It was my first visit to Thalia Hall and my first time seeing Metheny, who I discovered decades ago during my college years.

Mr. Metheny switched between ballads and electric fusion and funk and straight-ahead jazz with the same effortlessness with which he switched guitars. His arsenal of guitars was impressive, and each brought a unique sound. He played acoustic guitar and electric guitars - both solid and hollow-bodied; but he also boasts some unusual instruments in the guitar family. His synthesizer guitar sounds more like a horn section than a stringed instrument; and he made beautiful melodies with his custom-made 42-string "Pikasso" guitar.

And for this music, the melody often takes center stage. Between the improvisations, there was always a beautiful tune to enjoy.

Accompanying Metheny on stage tonight were James Francies on keyboard and Joe Dyson on drums. Collectively, the trio calls themselves Side Eye and their synergy was great. Francies and Dyson are accomplished musicians in their own right and gave us some impressive solos; but their real strength was in backing the master guitarist.

It is hard to believe that Pat Metheny is now 67 years old and not just because he still retains a full head of his signature wild hair. His energy, enthusiasm, and dexterity are those of a much younger man.

Photos

The musical "RENT" debuted 25 years ago in New York City. At the time, it must have caused some people to be uncomfortable. It  included homosexuals and drag queens and drug addicts and interracial relationships and fighting back against capitalist greed and AIDS.

Set in Manhattan's East Village, it paints a bleak picture of life for struggling bohemians of that area.

But it also presents a message of hope. Despite the tragedies he suffers, Tom Collins remains optimistic. Despite the pain of a dead girlfriend, Mark attempts to love again. Despite losing his girlfriend to another woman, Mark becomes friends with the lesbian couple. And the cross-dressing Angel inspires us all, despite dying of AIDS.

And, of course, there is the music. RENT is filled with rock songs and ballads and show tunes to lift you up and to pull at your heart.

It took me 25 years to see this classic. But I finally made it to the CIBC Theatre in Chicago last week. I am glad I did.

GCast 116:

Azure Active Directory B2C Token Generation With No User Interaction

Learn how to configure Azure Active Directory B2C, so you can request an Authentication Token without any user interaction. This is ideal for running automated processes that require a token, such as an integration test that calls a secure API.