Demonstrating Monitoring Solutions for CF and Lucee

Building on his 2021 ITB presentation, "Monitoring Solutions for CF and Lucee," Charlie now focuses on practical demonstrations of these tools. Discover key observations and metrics for troubleshooting, tuning, and receiving alerts. Gain insights into the evolution of these tools since the last talk, drawn from Charlie's extensive experience assisting users with server, container, and CommandBox environments.

1. INTO THE BOX 2024

2. BLUE ROOM
PRESENTED BY
Charlie Arehart
Demonstrating Monitoring Solutions
for CF and Lucee (and BoxLang!)

3. Charlie Arehart (CArehart.org, @carehart)
“Wha’ happened?”
oNearly all of us face challenges at some time with CF, Lucee, etc. whether on:
oOur dev environment (whether running on commandbox or not), or
oa server, or a cluster of servers, or
ovm’s, or
ocontainers, etc.
oRegardless of platform or architecture, this can be challenging
oIs the problem at the OS level? Or some component (cpu, memory, disk, network)
oIs it in CF or Lucee? Some setting? Our code? A bug?
oIs it about our traffic? Whether expected or not, unusual or not?

4. Charlie Arehart (CArehart.org, @carehart)
We need SOMETHING to observe
oThere are a range of alternatives to watch what’s going on
oAt each of those levels
oWhether built-in or as can be added (some free, some paid)
oMy 2021 ITB preso covered those in more detail
oWhat the options are, which are included/how to add them, considerations
oCommon causes of problems, more
oIn this talk, I just want to demonstrate many of the alternatives
oYou can go view the other presofor the details (nearly all are unchanged)
ocarehart.org/presentations/#mon_consider

5. Charlie Arehart (CArehart.org, @carehart)
Me.about()
oI focus on CF/Lucee server troubleshooting, as an independent consultant
oAssist organizations of all sizes, experience levels
oWork remotely: safe, secure, easy via shared desktop(zoom, meet, teams, etc.)
oSolve most problems in less than an hour, teaching you also as we go
oSatisfaction guaranteed
oMore on rates, approach, online calendar, etcat carehart.org/consulting
oBut to be clear, I’m not selling anything in this session! 
oJust sharing my experience, and that of others
oPresentation online at carehart.org/presentations

6. Charlie Arehart (CArehart.org, @carehart)
Topics
oCF/Lucee monitoring features
oSimple things built-in
oAlternative monitoring tools
oJava monitoring features
oJava command line tools, Ui’s, and more (built-in or freely added)
oAlternative monitoring tools (APM’s)
oOS monitoring features
oSimple—and more capable—built -in tools
oAlternative monitoring tools
oOther monitoring tools to consider

7. Charlie Arehart (CArehart.org, @carehart)
A comment…
oDifferent people may view some tools/features here as being for:
oTroubleshooting
oDebugging
oObservability (o11y)
oKeep this point in mind when searching for info on them
oOne person’s “debugging” is another’s “troubleshooting”
oAnd one person’s “monitoring” is another’s “observability”
oIn this talk, I’m not going to dwell on the distinctions
oAnd not covering step debugging at all (deserves its own talk)

8. Charlie Arehart (CArehart.org, @carehart)
I’ll default showing most of my demos…
o…in Windows
oI know the Mac and Linux folks will grumble
oBut you’re used to translating or finding things
oIndeed many things you find are WRITTEN more for *nix folks
oWindows folks often are left feeling like this:
ohttps://www.youtube.com/embed/TSse5b3wDe4

9. Charlie Arehart (CArehart.org, @carehart)
Questions we should be able to answer
oIs CF/Lucee/boxlangrunning?
oWhat are key resource usage metrics (cpu, memory, etc.)?
oHow many requests are running/have recently run?
oWhat are their key characteristics?
oAre any hung? Slow? WHY?
oSame with respect to queries, cfhttp calls, etc.
oAre errors happening? At what rate? Why happening?
oAnd more

10. Charlie Arehart (CArehart.org, @carehart)
Simple things built- into CF/Lucee
oLucee and CF both offer logs of course
oCF offers a metrics.log, if enabled
oAnd both offer request debug output (as enabled in CF or Lucee Admin)
oBetter than nothing, but not suited to production (or viewing requests of others)
oLucee Admin shows a bit of monitoring
oHeap use within Lucee instance
oCpuwithin Lucee instance
oCount of requests (running and queued), threads
oCounts of memory scopes used
oCount of datasource connections
oDemo…

11. Charlie Arehart (CArehart.org, @carehart)
Simple things built- into CF/Lucee (cont.)
oConsider also:
ogetmetricdata() function (CF and Lucee)
ocfstatcli script in CF (cfusion/bin directory)
oLucee getmemoryusage() function, about JVM heap memory
oAlso Lucee get*memory()functions, about system memory
oCF outputs metrics to Windows Performance Monitor
oOf course, these work with CF/Lucee however they are deployed

12. Charlie Arehart (CArehart.org, @carehart)
Extending CF/Lucee for monitoring
oLucee Performance Analyzer extension (github.com/zspitzer/lucee-
performance-analyzer)
oRequires debugging be enabled in Lucee Admin
oDemo…
oCF or Lucee:
ogithub.com/foundeo/ cfmetrics
oColdBoxDebugger (forgebox.io/view/cbdebugger)
oAnd there are more- complete monitoring solutions for CF/Lucee…

13. Charlie Arehart (CArehart.org, @carehart)
FusionReactor
oPowerful tool with many capabilities
oCan work entirely within JVM (“on-prem”)
oor also pushing data to cloud (“FR Cloud”), separately priced
oEasily added into FusionReactor: forgebox.io/view/commandbox-fusionreactor
oCould do entire talk on FR, and indeed I have done many
oJust last month: carehart.org/presentations/#cf_fr_opt
oMy series: youtube.com/playlist?list=PLG2EHzEbhy09al6Lx0JlIUcpbA8UrwPuo
oWorks with CF or Lucee (or BoxLang!)
oCommercial, with available developer license (fusion-reactor.com)
oQuick demo…

14. Charlie Arehart (CArehart.org, @carehart)
SeeFusion
oCame out about same time as FR mid- 2000’s
oOffers many basic/key monitoring features
oNot as capable as FR, but has a new advantage…
oIs no longer commercial: now open source, github.com/seefusion/seefusion
oWorks with both CF and Lucee
oMust implement either as Tomcat valve or servlet filter (vs javaagent)
oMust “wrap” datasources to monitor SQL/JDBC
oCFMeetupsession on SeeFusion: youtube.com/watch?v=QKpTtpCc0qE

15. Charlie Arehart (CArehart.org, @carehart)
PMT
oAvailable free in CF2018 and above
oAdobe’s replacement for the CF Server Monitor in CF8-2016
oOffers many capabilities like FR (and each has things the other doesn’t)
o20-min intro video: youtube.com/watch?v=ycydtKHi_AI
oWhile the tool is powerful, the take up in the community has been slow
oDue partly to its architecture: PMT and PMT Datastore are apps separate from CF
oCan be run on same machine as CF or another (even separate from each other)
oTip: Can use FR to monitor PMT and Datastore, as they are java apps!
oQuick demo of PMT…
o(One thing that PMT and FR can do—uniquely --is track cfthreadprocessing)
oBut sometimes CF/Lucee tools can’t be added or aren’t sufficient
oOf course, CF and Lucee (and BoxLang) run on Java
oAnd there are several Java tools we can look to…

16. Charlie Arehart (CArehart.org, @carehart)
Java/JVM command line tools
oSeveral commands, each having many subfeatures , in JDK versions of Java
oJava 8 and earlier offered option of JRE or JDK, later java versions are JDK only
oTools located in Java’s bin directory (such as the Java used to run CF/Lucee/BoxLang)
oFor local development instances, tools should just work with them (even running as service)
oAs long as user running jvm tool is same as user running CF/Lucee/BoxLang
oOn Windows, if CF/Lucee runs as service under “LocalSystem ” account, run cmdlineas admin
oWhat if instance is not local, or user running tools is not same as user running CF/Lucee/etc?
oThen you need to specify JMXremoteargsin startup of CF/Lucee/BoxLang
oPlenty of resources online cover doing that, for CF and Lucee (and java in general)
oSee also optional related tool called jstatd
oThese jvm tools work with CF/Lucee/BoxLangregardless of how deployed

17. Charlie Arehart (CArehart.org, @carehart)
Java command line tools (cont.)
oThey’re run by pointing to the OS process id of the running java/CF/Lucee/etc
oCan try to find with jcmd(one of those java commands), with no args
oThat will list those java processes YOU started
o(If CF/Lucee/etcis running as a service (Windows or *nix), won’t show those)
oCan use OS commands to find pid of running processes
oWindows: task manager (look for coldfusion.exe, not coldfusionsvc.exe), or
otasklist| findstrcoldfusion(CF typical install)
otasklist| findstrtomcat (Lucee typical install)
otasklist| findstrjava (Commandboxrunning CF/Lucee/BoxLang)
oLinux: pgrepjava(CF and Lucee always show in Linux as “java” processes)

18. Charlie Arehart (CArehart.org, @carehart)
Java jcmd
oCan do many things with this tool
oCan get list of features by using jcmd<processid>
oSome examples (assuming CF/Lucee/etcis process id 123)
ojcmd123 VM.version(argsare case-sensitive)
ojcmd123 VM.uptime(how long jvm/CF/Lucee/etchas been up)
ojcmd123 GC.heap_info(memory space info)
ojcmd123 GC.run(request garbage collection)
oAnd more …

19. Charlie Arehart (CArehart.org, @carehart)
Obtaining thread dumps/stack traces
oThread dumps identify what running requests (and other threads) are doing
oFusionReactor, PMT, and SeeFusion offer means to create them
ojcmdoffers another way to obtain them
ojcmd<pid> Thread.print(request thread dump)
oAs does:
ojstack<pid>
oAnd we’ll see other visual jvm tools that can create/view thread dumps
oDemos…
oFor now, on to other java cli tools…

20. Charlie Arehart (CArehart.org, @carehart)
Other Java CLI monitoring tools
ojdb, jinfo, jmap, jps, jstack, jstat
oI’ll leave you to explore those as may interest you
ojhatand hprofwere removed with Java 9
oCapabilities now in other tools already mentioned
oAgain,jstatdallows allowing remote calls into a host running a JVM
oIncluding setting it up via ssh
oWill leave that for you to explore if interested
oFor now, let’s wrap up with one powerful (and rather new) jvm CLI tool...

21. Charlie Arehart (CArehart.org, @carehart)
JFR (Java flight recorder)
oJFR is a feature that’s been around for a while, can be very powerful
oWas for a time an Oracle commercial feature, but is now free
oCan create jfrrecordings two ways
oUse jfrtool to launch a JVM/app or use jcmd<pid> JFR.start
oCan pass it argsto control WHAT it records, which is where power comes in
oAvailable argsalso to control saving/viewing JFR info at consle
oWill see GUI tools to create/view JFR recordings soon
oIndeed, while CLI tools can be powerful, many prefer GUI tools for many monitoring tasks
oAnd there are built-in or easily obtained GUI jvm tools…

22. Charlie Arehart (CArehart.org, @carehart)
jconsole
oFirst of 3 available java GUI tools (least capable of the 3)
oBut better than nothing, since it’s built-in to Java
oLike CLI tools, found in JDK’s binfolder
oDemo…
oIf on headless Linux or debugging a remote machine, use JMXRemoting
oMore about JMX: Java management extensions
oBoth a way the JVM exposes metric data to tools, and an API to dig into more
oViewing JMX info supported by this tool and next two, and also by FusionReactor

23. Charlie Arehart (CArehart.org, @carehart)
visualvm, formerly jvisualvm
oHas also gone through transition of licensing over years, now free/open source
ovisualvm.github.io
oCan do many things in its UI, while watching a JVM process (CF/Lucee/etc.)
oobtain thread dumps, heap dumps/analyze them
oprofile memory and cpu use (across all threads in the process)
oview JMX info
oread and graph JFR recordings(!)
oand more
oHas community of plugins offering still more
oDemo…

24. Charlie Arehart (CArehart.org, @carehart)
Java Mission Control (JMC)
oHas also gone through transition of licensing over years, now free/open source
owiki.openjdk.org/display/jmc/Main
odocs.oracle.com/en/java/java-components/jdk-mission -control/
oCan do many of same things as visualvm
oIncluding taking thread dumps, processing JFR recordings, showing JMX info
oSeems to REQUIRE enabling of JMX (port being exposed by CF/Lucee/etc)
oDownload:
oJMC 8: jdk.java.net/jmc/
oJMC 9: oracle.com/java/technologies/jdk-mission -control.html
oJMC 9 requires Java 17, JMC 8.1 requires Java 11

25. Charlie Arehart (CArehart.org, @carehart)
Related Java monitoring tools
oI’ve mentioned how JVM (and some CF tools) can create thread dumps
oCheck out helpful free online tool to analyze them:
ofastthread.io
oAnd free sister online tools:
oheaphero.io
ogceasy.io
oycrash.io
oStill more java monitoring tools
oSee my list at cf411.com/ javamon
oBut such built-in Java monitoring tools have more capable cousins…

26. Charlie Arehart (CArehart.org, @carehart)
Java APMs
oThere’s a whole world of Java Application Performance Monitoring (and
“observability”) tools, which can be added to CF/Lucee/etc
oAcross all platforms
oSome free, some commercial
oLike other Java tools, often these APMs don’t easily track page requests
oAnd they often present low-level java info, hard to relate back to CF/Lucee/etc
oNo time to demo them in this talk
oSee my list at cf411.com/ apm
oNote that some Java tools and APMs include “system” monitoring features
oIndeed, often problems in our apps really reflect larger system problems
oSeveral kinds of tools focus on that…

27. Charlie Arehart (CArehart.org, @carehart)
Simple OS- level tools
oWindows
oTask Manager (see ProcessesandDetails, sortable; and Performance)
otasklist(command)
oBonus: systeminfo| findstr"Boot time"
oLinux
otop(can sort using “f” key)
oConsider installable variants: htop, atop, vtop, gtop, etc.
ops-aux(try also -auxf)
oBonus: uptime
oMacOS
oActivity Monitor (GUI)
oAlsoLinux (Unix) command line tools above

28. Charlie Arehart (CArehart.org, @carehart)
More capable OS tools, built- in
oWindows
oResource Monitor
oPerformance Monitor (tip: switch “graph type” to “report”)
oLinux
osar(collect system activity over time)
onetstatand ss -tua(network stats)
oConsider adding
oiostat(cpuload, disk activity), iotop(still more disk i/o detail)
oiftop(bandwidth usage on an interface by host)

29. Charlie Arehart (CArehart.org, @carehart)
System monitoring tools, as add- ons
oAs with APMs, a whole world of OS monitoring tools available
oAcross all platforms
oAgain, some free, some commercial
oSee my list at cf411.com/ sysmon
oFinally, some problems are specific to other aspects of processing…

30. Charlie Arehart (CArehart.org, @carehart)
Other monitoring tools
oThere are also tools for monitoring your:
oWeb server
oDatabase
oNetwork
oIO
oAnd more
oI track tools like these and still more at cf411.com/ dbmon
oAnd cloud platforms (AWS, Azure, GCP) offer monitoring for their components

31. Charlie Arehart (CArehart.org, @carehart)
Finding more
oI’ve held off offering links to more on most of these things
oMost can be found readily via web searching
oDo beware that some resources are old (and may no longer be accurate)
oAgain, the PDF of this talk is already online on my site
oAnd the recordings of these ITB sessions will be offered online soon
oI’ve also pointed to related presentations I’ve done (usually also recorded)
oCertainly feel free to reach out to me if you have trouble finding a resource

32. Charlie Arehart (CArehart.org, @carehart)
Summary
oProblems will happen with your CF/Lucee instances
oDon’t regard them as a black box, or throw darts trying to “fix” things
oWe’ve seen there are built-in tools, or tools easily added, to better understand:
oCF/Lucee monitoring capabilities and options
oJava monitoring features and options
oOS monitoring features and options
oMonitoring other components/aspects of architecture
oI hope you feel more empowered to dig into such problems on your own
oIf you hit snags/want guiding hand, am available for remote, screenshare consulting
oCan work in as short as 15-min intervals; solve most problems in less than an hour

33. INTO THE BOX 2024
INTO THE BOX 2024