How do we extract Serilog .ForContext() objects logged to Elasticsearch, using ElasticClient (NEST)?

Refresh

March 2019

Views

3 time

0

We are looking to better centralize, structure and search our logs using the ELK stack. We are currently logging to a database table, and we have an EventLog POCO that captures the fields required to populate those rows. Our first stab at getting this data into Elasticsearch was to simply log the events via Serilog like this:

EventLog eventLog = ...;
log.ForContext("EventLog", eventLog, true).Write(eventLog.Message);

This results in a Serilog "logevent"-type document, with a "field" that contains our custom structure, and gets us most of what we want:

{
  "_index": "logstash-2019.03.14",
  "_type": "logevent",
  "_id": "p_amfGkBaphWeJXGjjSW",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-03-14T09:41:21.6924251-05:00",
    "level": "Information",
    "messageTemplate": "blah",
    "message": "blah",
    "fields": {
      "EventLog": {
        "_typeTag": "EventLog",
...

Kibana allows us to do ad-hoc searches of our logs on fields in our custom structure, and we're fairly happy already.

Now we want to search our logs programmatically, and we're looking at NEST for that, but we can't quite figure out the right incantations. ElasticClient.Search<T> seems to be what we need, we're not sure what T needs to be, nor how NEST interprets the search criteria or the returned data.

What we've tried:

client.Search<EventLog>();   // returns zero results
client.Search<Serilog.Events.LogEvent>();   // throws: Error converting string to Serilog.Events.MessageTemplate
client.Search<EventLog>(s => s.Type("logevent")); // returns results!, but the EventLog objects are all empty (properties all have default values)

The only call that seems to get us anywhere close is:

client.Search<Object>(s => s.Type("logevent")); // returns the raw JSON source of each log entry

So, we feel like we're missing something simple and fundamental here. The NEST docs are all predicated on a Project type that is indexed directly into Elasticsearch, but what to we need to do to be able to extract and query over our custom structures that we add to the log using Serilog's ForContext?

0 answers