Allow system admin to see what is collected

[djoreilly]

Sometimes system admins need to know what is being collected by Velociraptor on their endpoints. They would like to be able to verify that no sensitive data is being sent to the server. Maybe a client side option that would write the VQL responses to JSON files.

[scudette]

You can set up a server monitoring artifact to view all completed collections - you can see an example of such a thing here https://docs.velociraptor.app/artifact_references/pages/elastic.flows.upload/ which actually uploads the data.

You can also see all the initiated collections and hunt in the audit log (and forward it to another system or maybe slack or discord in real time). All audit events are sent on this artifact https://docs.velociraptor.app/artifact_references/pages/server.audit.logs/ so you can monitor it with SELECT * FROM watch_monitoring(artifact="Server.Audit.Logs")

[djoreilly]

So I mean the OS system administrators of the endpoint machines running Velociraptor clients - not the Velociraptor administrators. Even if forwarding to Elastic was set up, there would need some way to ensure they only have visibility to the data from their endpoints.

[djoreilly]

For some more context here is an example of a concern sysadmins have. There may be something running on their endpoint that shells out and puts sensitive information on the command line and Linux.Events.ProcessExecutions will send it to the Velociraptor server. A feature like this allows them to monitor what is being gathered for a while until they are satifised that nothing sensitive is leaking out.

I have done a quick implementation djoreilly@2ede58e

[scudette]

Yes this is correct - you are writing the output of the logs which are the same as shown with the --verbose flag.

It is not reasonable IMHO to log the actual output from the queries (some queries generate gb of results). The logs show what kind of data was collected though. What is the purpose of the logs? to see if the collection was reasonable or misused?

[djoreilly]

It is not reasonable IMHO to log the actual output from the queries (some queries generate gb of results). The logs show what kind of data was collected though. What is the purpose of the logs? to see if the collection was reasonable or misused?

We have a policy that all Linux servers on the company network must have the Velociraptor client installed. This makes some system owners a bit anxious and they wish to know exactly what is collected. Sure we can point them to the source code for all the artifacts we use, but most people don't know VQL and would have difficultly understanding them. If there was an option like this, then they could enable it for as long as they like and just look at the raw data that is being sent to the server. Of course this option would not be set in the default client config pushed out to all clients. I have updated the diff so the option comes with a warning djoreilly@2f1751c

[scudette]

The problem with the PR is that it is sending potentially very large results into the logs. Velociraptor keeps the last 1000 log messages in memory so they can be accessed remotely during profile collection or debugging. So this data will stay around for quite a while and may cause high memory usage. Typically logs should not be handling very large messages.

[djoreilly]

Ah okay, I didn't know the last 1000 log messages where kept in memory. I'll get back to the people asking for the feature and explain the problems. Thanks again.

[scudette]

You can see where that is done here

velociraptor/logging/logging.go

Line 542 in f753569

if len(memory_logs) > 1000 {

You can just write the data out yourself (i.e. not through the log)