07 - Extending VQL and API in Velociraptor.pptx

Extending VQL and API's in Velociraptor

1. Velociraptor Digging Deeper! ‹#›

2. Extending VQL and the Velociraptor API ‹#›

3. Module overview VQL is really a glue language - we rely on VQL plugins and functions to do all the heavy lifting. To take full advantage of the power of VQL, we need to be able to easily extend its functionality. This module illustrates how VQL can be extended by including powershell scripts, external binaries and extending VQL in Golang. For the ultimate level of control and automation, the Velociraptor API can be used to interface directly to the Velociraptor server utilizing many supported languages (like Java, C++, C#, Python). ‹#›

4. Extending VQL - Artifacts ‹#› The most obvious tool for extending VQL is simply writing additional artifacts. We have seen this done extensively in previous modules Artifacts serve to encapsulate VQL queries: Allows us to reuse a complex VQL query without worrying too much about the implementation. Allows the implementation to evolve with time - perhaps offering additional functionality or better algorithms Ultimately we are limited with the basic capabilities of the VQL engine.

5. Extending artifacts - PowerShell ‹#› Powershell is a powerful systems automation language mainly used on Windows systems where is comes built in and almost always available. Many complex software products contain powershell modules around automation and system administration. It does not make sense for Velociraptor to directly support complex software packages like Office365, Azure etc. But it is critical to be able to recover forensically relevant data from these package Therefore it makes sense to wrap powershell scripts in VQL artifacts.

6. Exercise: PowerShell based pslist ‹#› This is not a PowerShell course! So for this example we will use the following very simple snippet of PowerShell Get-Process | Select Name, Id, Path

7. Exercise: PowerShell based pslist ‹#› The execve() plugin takes a list of args and builds a correctly escaped command line. In many cases we dont need to encode the command line

8. Alternative - encode scripts ‹#› If you get encoding errors then you can still encode the script.

9. Alternative - tempfile scripts ‹#› Tempfiles will be automatically cleaned up at the end of the query

10. Dealing with output Using the execve() plugin we can see the output in Stdout It would be better to be able to deal with structured output though. We can use powershell's ConvertTo-Json to convert output to JSON and Velociraptor's parse_json() to obtain structured output. This allows VQL or operate on the result set as if it was natively generated by a VQL plugin! ‹#›

11. ‹#›

12. ‹#› Parsing JSON output PowerShell outputs a single JSON object which is an array all the rows. We need to parse it in one operation - so we need to buffer all the Stdout into memory (set length=1000000).

13. ‹#›

14. ‹#› Reusing powershell artifacts Since our powershell script is now encapsulated, we can use it inside other artifacts and plain VQL. Users of this artifact dont care what the PowerShell Script is or what it does - we have encapsulation!

15. Remediation ‹#›

16. Velociraptor Remediation Remediation means to restore the network from a compromised state - Usually remove persistence. Create a scheduled task to run something daily SCHTASKS /CREATE /SC DAILY /TN "EvilTask" /TR "dir c:" /ST 20:01 ‹#›

17. Exercise ‹#› Find your "evil" scheduled task and remediate using the Windows.Remediation.ScheduledTasks artifact

18. Using external tools ‹#› © 2020 Velocidex Enterprises

19. Why use external tools? Velociraptor has a lot of built in functionality, but we can not cover all use cases! Velociraptor can automatically use external tools: Velociraptor will ensure the tool is delivered to the endpoint The tool can be called from within VQL VQL can parse the output of the tool - thereby presenting the output in a structured way VQL can then further process the data ‹#›

20. Velociraptor Tools Tools are cached locally on the endpoint and are only re-downloaded when hashes change. Admin can control Where the tool is served from - Serve Locally or from Upstream Admin can override the tool with their own Artifact writers can specify The usual download location for the tool. The name of the tool Velociraptor is essentially an orchestration agent ‹#›

21. Autoruns Autoruns is a Sysinternals tool Searches for many different types of persistence mechanisms We could reimplement all its logic OR We could just use the tool ‹#›

22. ‹#›

23. Let's add a persistence sc.exe create owned binpath="%COMSPEC% /c powershell.exe -nop -w hidden -command New-Item -ItemType File C:\art-marker.txt" https://github.com/redcanaryco/atomic-red-team/blob/master/atomics/T1569.002/T1569.002.yaml ‹#›

24. ‹#›

25. ‹#›

26. ‹#›

27. ‹#› Artifact encapsulation Use autoruns to specifically hunt for services that use %COMSPEC% Artifact writers just reuse other artifacts without needing to worry about tools.

28. ‹#›

29. ‹#› Exercise - Use Sysinternal DU Write an artifact to implement Sysinternal's du64.exe to calculate storage used by directories recursively.

30. ‹#› Third party binaries summary Generic.Utils.FetchBinary on the client side delivers files to the client on demand. Automatically maintains a local cache of binaries. Declaring a new Tool is easy Admins can override tool behaviour Same artifact can be used online and offline

31. Automating the Velociraptor Server ‹#›

32. Server artifacts Server automation is performed by exporting server controlling functions as VQL plugins and functions This allows the server to be controlled and automated using VQL queries Server artifacts encapsulate VQL queries to performs certain actions Server monitoring artifacts watch for events on the server and respond. ‹#›

33. ‹#›

34. Client version distribution 30 Day active client count grouped by version 1 Day active client count ‹#›

35. Server concepts Client : A Velociraptor instance running on an endpoint. This is denoted by client_id and indexed in the client index. Flow : A single artifact collection instance. Can contain multiple artifacts with many sources each uploading multiple files. Hunt : A collection of flows from different clients. Hunt results consist of the results from all the hunt's flows ‹#›

36. Exercise - label clients Label all windows machines with a certain local username. Launch a hunt to gather all usernames from all endpoints Write VQL to label all the clients with user "mike" This can be used to label hosts based on any property of grouping that makes sense. Now we can focus our hunts on only these machines. ‹#›

37. Exercise - label clients with event query The previous method requires frequent hunts to update the labels - what if a new machine is provisioned? Label all windows machines with a certain local username using an event query. ‹#›

38. Server management with VQL We just ran a hunt which downloaded 300Gb of data (oops!) now our server is full! We need to delete the results from this hunt because we don't need them. ‹#›

39. Develop “Server.Admin.DeleteHunt” ‹#› Step 1: Find all the flows launched within the hunt Step 2: Figure out all the files that make up the flow: enumerate_flow() Step 3: Remove these files: file_store_delete()

40. ‹#› Start cancelling flows If your hunts are too heavy and overload the server simply start cancelling flows with VQL.

41. The Velociraptor API ‹#›

42. Why an API? Velociraptor needs to plug into a much wider ecosystem Velociraptor can itself control other systems Can already be done by the execve() and http_client() VQL plugins. Velociraptor can be controlled by external tools Allows external tools to enrich and automate Velociraptor This is what the API is for! ‹#›

43. API Server GUI Client program (Python) X509 Cert X509 Cert TLS with mutual certificate verification. ‹#› TLS authentication occurs through pinned certificates - both client and server are mutually authenticated and must have certificates issued by Velociraptor's trusted CA. Execute arbitrary VQL

44. The Velociraptor API The API is extremely powerful so it must be protected! The point of an API is to allow a client program (written in any language) to interact with Velociraptor. The server mints a certificate for the client program to use. This allows it to authenticate and establish a TLS connection with the API server. By default the API server only listens on 127.0.0.1 - you need to reconfigure it to open it up.

45. Create a client API certificate velociraptor --config server.config.yaml --config server.config.yaml config api_client --name Mike --role administrator api.config.yaml ‹#›

46. Grant access to API key Currently two levels of access: velociraptor --config /etc/velociraptor/server.config.yaml config api_client --name Mike --role reader,api api_client.yaml velociraptor --config /etc/velociraptor/server.config.yaml acl grant Mike --role administrator,api Access to push events to an artifact queue: velociraptor --config /etc/velociraptor/server.config.yaml acl grant Mike '{"publish_queues": ["EventArtifact1", "EventArtifact2"]}' ‹#›

47. Export access to your API Normally Velociraptor is listening on the loopback interface only If you want to use the API from external machines, enable binding to all interfaces ‹#›

48. Using Jupyter to analyze hunts ‹#›

49. Jupyter Jupyter is an interactive notebook app Runs python internally but we can use it to call the server via APIs ‹#›

50. Install the python bindings For python we always recommend a virtual environment and Python 3 then: pip install pyvelociraptor jupyter pandas ‹#›

51. ‹#› Download the api_client.yaml file to your workstation and edit the api_connection_string to your external domain name

52. Run a simple query Set the api file in the environment variable Use the sample app in the PyVelociraptor bindings to test your connection to the server pyvelociraptor "SELECT * FROM info()" ‹#›

53. ‹#›

54. ‹#›

55. Running VQL on the server Write a simple script by copying the sample app and editing it creds = grpc.ssl_channel_credentials( root_certificates=config["ca_certificate"].encode("utf8"), private_key=config["client_private_key"].encode("utf8"), certificate_chain=config["client_cert"].encode("utf8")) options = (('grpc.ssl_target_name_override', "VelociraptorServer",),) with grpc.secure_channel(config["api_connection_string"], creds, options) as channel: stub = api_pb2_grpc.APIStub(channel) request = api_pb2.VQLCollectorArgs( max_wait=1, Query=[api_pb2.VQLRequest( VQL=" SELECT * from clients() ", )]) for response in stub.Query(request): package = json.loads(response.Response) print (package) ‹#›

56. Schedule an artifact collection You can use the API to schedule an artifact collection LET collection <= collect_client(client_id='C.cdbd59efbda14627', artifacts='Generic.Client.Info', args=dict() ) ‹#›

57. Wait for the client to finish When a collection is done, the server will deliver an event to the System.Flow.Completion event artifact You can watch this to be notified of flows completing SELECT * FROM watch_monitoring(artifact='System.Flow.Completion') WHERE FlowId = collection.flow_id LIMIT 1 ‹#›

58. Reading the results You can use the source() plugin to read the results from the collection. SELECT * FROM source(client_id=collection.ClientId, flow_id=collection.flow_id, artifact='Generic.Client.Info/BasicInformation') ‹#›

59. Exercise: Put it all together Write VQL to call via the API to collect an artifact from an endpoint and read all the results in one query. ‹#›

60. Event Queries and Server Monitoring ‹#›