xbuild and “the PCL reference assemblies not installed” error message

I was trying to setup a build script with FAKE to build a Xamarin solution via a bash script which I could then use in TeamCity.
In Xamarin Studio I could build my solution without any errors but when trying to run my script in a terminal, I received this error: “PCL reference assemblies not installed”.

I searched for several solutions untill I uninstalled Xamarin and Mono.
When reinstalling Xamarin I suddenly noticed that it was installing Mono version 4.0.2.

The error message, however, showed version 4.0.1.

After noticing this difference, I remembered I installed Mono via Homebrew to run ASP.NET vNext on Mac OS X.
After uninstalling it with

brew uninstall mono

, my script runned without any errors.

Sigh…

Centralize your logs with the ELK stack

At my current customer I recently had the opportunity to play around with logstash.
There are several log files spread across different servers which makes it difficult to easily identify the most critical errors.

logstash aids in this by centralizing all these log files in one place.
We use a combination of Elasticsearch, logstash and Kibana or more better known as the ELK stack.

logstash will collect all the log files of our servers, parse them, send them to Elasticsearch in a uniform format and then use Kibana to visualize these logs.
I just want to cover how we’ve set things up and give some useful tips along the way.

I’ll skip the part of how to install logstash because this is quite straightforward.
Just take a look at the documentation of logstash.

Input configuration

logstash is all about configuration which is accomplished by using plugins of which several are included by default.
The file plugin is the first plugin we need; here we will define where logstash needs to search for our log files.

input {
    file {
        path => "/Logs/**/general.log"
        start_position => "beginning"
    }
}

start_position => “beginning” instructs logstash to read from the beginning of the file. The default behavior is to only read the new log lines since logstash started.
However it’s important to know that logstash will keep track which lines are already processed. If logstash is restarted, it won’t start again from the beginning of the file.

start_position only has influence on new log files.
This is of course useful in production environments but while testing your setup this can be quite annoying.
By adding sincedb_path => “/dev/null” to the file plugin configuration you force logstash to read the complete file again.

File locks

The documentation isn’t very clear about how many times a file is read but based on the source code in GitHub, you need to take two parameters into account.

input {
    file {
        stat_interval => 1
        discover_interval => 5
    }
}

The values in the example above are the default values and can be omitted.
stat_interval: if all files are processed (defined by the file plugin), how many seconds does logstash needs to wait before processing all files again?
discover_interval: how many seconds logstash will wait before processing another file in the list

If the wildcard in the path property would match against 3 files, it will process the first file, wait for 5 seconds and then process the second file and wait another 5 seconds and so on…
If the three files are processed, it will wait 1 second and process the list again.

 

Filters configuration

First plugin covered, on to the next… filters!
Filters are used to define how you want to parse the log files.

filter {
    multiline {
        pattern => "^%{TIMESTAMP_ISO8601}"
        negate => true
        what => previous
    }

    grok {
        match => ["message", "(?m)%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{GREEDYDATA:information}"]
        tag_on_failure => ["error_message_not_parsed"]
        remove_field => ["message"]
        break_on_match => false
    }

    grok {
        match => [ "path", "/Logs/(?<server>[^/]+)/(.*).*" ]
        tag_on_failure => ["path_not_parsed"]
    }
}

The first filter is the multiline filter.
By default logstash will create a new record for each line in the log file.
In our case (and most cases I presume) stacktraces are included in the logs so we would rather group log lines together as one message.

The multiline filter is configured as follows: group all lines of the log together in one message untill you reach another timestamp.

Now that we’ve grouped the necessary log lines together, we can start splitting up the message into different fields with Grok.

Grok is a plugin with a predefined set of regular expressions.
It’s possible that each log file is in a different format especially when your log files are read from different sources (log4net error logs, IIS logs, Apache logs, …).

By using Grok patterns you can target each type of log file seperately.

In the Grok filter for the message you will notice that the expression starts with (?m).

grok {
    match => ["message", "(?m)%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{GREEDYDATA:information}"]
    tag_on_failure => ["error_message_not_parsed"]
    remove_field => ["message"]
    break_on_match => false
}

This is required because without it the multiline filter defined previously is overridden. The Grok pattern GREEDYDATA stops at a newline by default (see https://logstash.jira.com/browse/LOGSTASH-509)

The other patterns are quite straightforward, it will first put the timestamp in a seperate field, then the log level and it finally puts what’s left of the message in another field.

Let me show you an example.
This is the error message:

2014-12-12 03:21:49,285 ERROR http://www.mywebsite.be The request was aborted: The request was canceled.
 System.Net.WebException: The request was aborted: The request was canceled. ---> System.IO.IOException: Cannot close stream until all bytes are written.
 at System.Net.ConnectStream.CloseInternal(Boolean internalCall, Boolean aborting)
 --- End of inner exception stack trace ---
 at System.Net.ConnectStream.CloseInternal(Boolean internalCall, Boolean aborting)
 at System.Net.ConnectStream.System.Net.ICloseEx.CloseEx(CloseExState closeState)
 at System.Net.ConnectStream.Dispose(Boolean disposing)
 at System.IO.Stream.Close()

This is the outputted result:

 timestamp: 2014-12-12 03:21:49,285
 log-level: ERROR
 information: http://www.mywebsite.be The request was aborted: The request was canceled.
 System.Net.WebException: The request was aborted: The request was canceled. ---> System.IO.IOException: Cannot close stream until all bytes are written.
 at System.Net.ConnectStream.CloseInternal(Boolean internalCall, Boolean aborting)
 --- End of inner exception stack trace ---
 at System.Net.ConnectStream.CloseInternal(Boolean internalCall, Boolean aborting)
 at System.Net.ConnectStream.System.Net.ICloseEx.CloseEx(CloseExState closeState)
 at System.Net.ConnectStream.Dispose(Boolean disposing)
 at System.IO.Stream.Close()

The last line of the Grok filter states that it shouldn’t stop after a successful match (which it does by default) but that it should also execute the next Grok filter.
That filter will extract some information from the file path to better identify the original source of the log file.

grok {
    match => [ "path", "/Logs/(?<server>[^/]+)/(.*).*" ]
    tag_on_failure => ["path_not_parsed"]
}

At GitHub you can find an overview of the available Grok patterns.
It’s possible to create your own Grok pattern and a helpful tool to debug your Grok pattern is http://grokdebug.herokuapp.com

Next step is to configure where we want to store all the information logstash just processed.
We’ve choosen Elasticsearch because we already used this for other purposes, it didn’t need much setup anymore and because it’s blazing fast!

I won’t cover Elasticsearch here, there is a lot of documentation available on the web.

output {
    elasticsearch {
        host => "your-elasticsearch-server"
        index => "name-of-the-index-for-your-logs"
        protocol => node
        node_name => name_of_your_node
        cluster => elasticsearch_cluster_name
        template => "/etc/logstash/mapping/es-template-logstash-with-ttl.json"
        template_overwrite => true
    }
}

The host is the fully qualified domain name of your Elasticsearch server.
The protocol is set to node so logstash will create a non-data node which is responsible for the gathering of the logs but not for indexing the data.

When setting the protocol to node, you can also use the option node_name.
If you don’t configure this value, Elasticsearch will give the node a random name.

Another option for the protocol setting is the value http which will use the HTTP API of Elasticsearch.

By default logstash comes with a default Elasticsearch template file.
We’ve extended this template file to also include a TTL (time-to-live) value for the Elasticsearch documents.

This allows us to keep the size of the Elasticsearch index under control.
Each log message in the index will automagically get cleaned up if it’s older then the defined TTL (30 days in our setup).

    "mappings": {
        "_default_": {
            "_all": {
                "enabled": true
            },
            "_ttl" : { "enabled" : true, "default": "30d" },
            [...]

If everything went according to plan, your logs are now available in one place and in one format.
If you have still some issues, you can start the logstash service via command-line with the option –verbose or –debug.

The debug flag however generates so much noise it’s not easy to identify the error.
In most cases the verbose flag should quickly point you in the right direction.

Kibana

Now we can setup Kibana to easily read through the logs and do some visualisation magic!

To install Kibana, you just need a webserver.
Drop the files of the Kibana package in the webserver folder and you’re up and running!

The only file you need to edit is the config.js in the Kibana directory.
Search for elasticsearch in the configuration file and replace it with the FQDN of the Elasticsearch server.

You can also change the default_route property and point it to ‘/dashboard/file/logstash.json’ which provides a default setup for logstash.
Fire up the URL of your webserver and you should see a default dashboard.

These are some examples of dashboards we’ve made.

kibana-bar-graph

kibana-line-graph

I will cover Kibana in more depth in another blog post.

To summarize everything, here is a sample configuration of all the settings I discussed:

input {
    file {
        path => "/Logs/**/general.log"
        start_position => "beginning" #read from the beginning of the files
    }
}

filter {
    multiline {
        pattern => "^%{TIMESTAMP_ISO8601}" #group all lines of the log together in one message untill you reach a timestamp
        negate => true
        what => previous
    }

    grok {
        match => [ "message", "(?m)%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{GREEDYDATA:information}" ] #add the timestamp, log level and rest of the message in seperate fields
    }

    grok { 
        match => [ "path", "/Logs/(?<server>[^/]+)/(.*).*" ] #add parts of the file path in seperate fields
    }
}

output { 
    elasticsearch {
        host => "FQDN" #the FQDN of the elasticsearch server
        protocol => node #create a non-data node which is responsible for the gathering of the logs but not for indexing the data
        cluster => "elasticsearch_cluster" #cluster name of your elasticsearch; aids in the discovery of the nodes
        index => "log-servers" #write the logs to this index in ES (if the index doesn't exist, it will be created automagically)
        template => "path to ES template"
        template_overwrite => true #make sure to use the template defined in this config and not to use one defined in elasticsearch
    }
}

If you have remarks or questions, just leave a message!

Soundcheck!

Ik gooi het even over een andere boeg voor de blog want de laatste tijd kwam ik totaal niet toe aan het schrijven van posts, vooral omdat het enkel gericht was naar de nerdy topics.

Daarom een verse start en zal dit dus eerder een samenvatting worden van allerlei verschillende topics. Zo hoop ik wat meer dingen te schrijven en te delen…
Laten we eens kijken waar dat ons brengt!

We gaan onmiddellijk de muzikale toer op.
Sinds december ben ik overgestapt van Spotify naar Rdio. De reden waarom en wat ik momenteel van Rdio vind, leg ik wel eens uitgebreid uit op een ander moment.

Via de stations van Rdio heb ik vandaag Frightened Rabbit leren kennen.

Rdio stations

Rdio stations

Beluister zeker eens het album “Pedestrian Verse”; ik was vrij snel verkocht!

Albums van Frightened Rabbit via Rdio

Albums van Frightened Rabbit

Ik heb ze leren kennen via de voorstellen van Rdio op basis van The National dus als dat jouw wel ligt, zou ik het zeker eens beluisteren!

Andere gerelateerde artiesten volgens Rdio? The Shins, Band of Horses, Arcade Fire, …