What do you like best?
Scalyr offers support for a variety of protocols and data sources to be aggregated together quite easily. The agent is quick and offers solid performance even under a lot of load. It also has a lot of plugins and can even accept custom-developed plugins.
The UI allows for the flexibility of searching on whatever fields are being input, which is convenient for mixed streams. The service handles a large amount of data very well and is very configurable on parsing, with a syntax that makes sense and is easy to use. It even has a built-in parsing tester that is extremely convenient. Everything in the UI also feels very snappy and quick compared to other services, even when generating massive graphs.
When there's issues with the service, they are well communicated and usually brief.
What do you dislike?
Scalyr definitely feels like a new product, with rough edges and some limitations that I hope are going to be worked around soon.
Graph creation is difficult and a little limited, the syntax is strange. Alarms are similarly a little awkward to put together at first.
I've had trouble with the Docker integration when used with Swarm. It doesn't seem to work unless I tie it together manually using Syslog.
A few areas have a little bit of a confusing GUI flow, between JSON files and normal UI.
The aggregation of fields and their "most common occurrences" can be sometimes frustrating when I want to search for entries that have occurred very few times, but this can usually be worked around.
Support has sometimes been slow to respond or keep up on support requests.
Graphs are sometimes "spiky", due to a mismatch between log pickup periods and log display periods. If you see this, set your graphs to a bar with a 5-minute period or more.
Recommendations to others considering the product
Make sure that whoever you have administrating Scalyr is familiar with system administration and is comfortable working with JSON config files and regular expressions. I recommend something like Terraform to manage the configurations for CloudFront, S3 buckets, and the ensuing SQS queues that Scalyr uses to pick up logs.
What business problems are you solving with the product? What benefits have you realized?
We're pushing all of the logs for every load balancer, S3 bucket, and CloudFront in our whole AWS cloud into Scalyr, without having to configure any awkward Lambdas or ElasticSearch clusters. We also have pushed all of our application logs into it using the agent's Syslog protocol, which has made our error searching much easier.