Custom Logging System for Plumber API and Shiny Applications A futuristic dashboard showing a central server connected to multiple API endpoints. Lines representing data flow between endpoints show user actions and responses, with real-time monitoring screens and storage for log files. On the left, warning icons indicate error tracking, while on the right, a clock symbolizes automated log management. The background has code snippets and data logs, creating a technical, blue-toned atmosphere that reflects system performance monitoring and troubleshooting.

Custom Logging Plumber API

Picture of Lovekumar Patel

Introduction

Our client, a leading investment consulting firm, aimed to improve system monitoring, troubleshooting, and reliability by implementing a logging system that tracks user actions and errors in both Shiny applications and Plumber APIs. The existing system lacked a comprehensive logging solution, making it difficult to pinpoint issues, monitor system performance, and ensure application reliability.

 

Challenges and Pain Points

Before the implementation, the client faced several challenges:

  1. Limited Visibility: The absence of a structured logging mechanism led to difficulties in understanding the flow of user interactions and API requests. Troubleshooting issues were often time-consuming and required extensive manual investigation.
  2. Error Tracking: Errors in the system were not consistently captured, leading to sporadic failure reports and an inability to proactively address potential issues.
  3. Maintenance Overhead: Without automated log management, the client had to manually manage log files, resulting in cluttered storage and inefficient use of server resources.
  4. Delayed Response to Critical Issues: The client was unable to promptly respond to critical failures or system errors due to the lack of real-time alerts.

 

ProCogia’s Implementation

This project aimed to implement a robust logging system for both APIs and Shiny applications using the Logger package in R. Additionally, we will create shell scripts to manage the storage and periodic cleanup of logs using cron jobs (cron tabs). The integration of comprehensive logging will enhance monitoring, troubleshooting, and overall system reliability.

 

Key Features

The Plumber R package is a powerful tool for exposing R functions as API endpoints, offering developers significant flexibility in how they design their APIs. Among the critical considerations when building an API is how to effectively log information about API requests and responses. Such logs are invaluable for monitoring API performance and utilization.

While Plumber provides an example of logging API requests using filters, this approach logs incoming requests before a response is generated. While valid, it limits the log’s detail, as the response information is not yet available. In this post, we’ll explore an approach to logging using preroute and postroute hooks, allowing us to log both the request and its associated response.

 

Setting Up Logging

For this example, we’ll use the logger package to handle logging, although other logging frameworks could also be used. The setup involves two files: plumber.R and entrypoint.R.

 

Defining API Endpoints in plumber.R

The plumber.R file defines a simple API with two endpoints for demonstration purposes:

				
					# plumber.R
# A basic API for logging demonstration

library(plumber)

#* @apiTitle API Logging Demo

#* @apiDescription Demonstration of custom logging in Plumber API

# Example endpoint that uses user and session information
#* @get /hello
#* @param name The name to greet
function(req, res, name = "World") {
    paste("Hello,", name, "!")
}

#* Generate a random histogram
#* @serializer png
#* @get /histogram
function() {
    random_data <- rnorm(100)
    hist(random_data)
}

Configuring Logging in entrypoint.R

Next, we’ll configure logging in the entrypoint.R file, utilizing the logger package.

				
					# entrypoint.R
library(plumber)
library(logger)

# Set up the logging directory and appender
log_directory <- "api_logs"
if (!fs::dir_exists(log_directory)) fs::dir_create(log_directory)
log_appender(appender_tee(tempfile("api_log_", log_directory, ".log")))

Here, we specify that logs should be written both to stdout and to a uniquely named file in the api_logs/ directory. The tempfile() function ensures that each log file has a unique name, preventing conflicts when multiple processes are logging simultaneously.

 

A Helper Function for Log Entries

We’ll define a helper funcion to handle cases where certain log values might be empty:

				
					# helper.R
# Convert empty strings to a placeholder
handle_empty <- function(string) {
  if (string == "") "-" else string
}

This function ensures that empty log fields are represented as a dash (“-”), making the logs easier to read.

 

Creating a hooks for pre and post route

Typically, users define APIs using the “annotations,” or special comments in their API source code. It is possible to define a Plumber API programmatically using the same underlying R6 objects that get created automatically when you define your API using annotations. Interacting with Plumber at this level can offer more control and specificity about how you want your API to behave.

In this setup, the preroute hook starts a timer, and the postroute hook stops the timer and logs the relevant request and response details, including:

				
					# entrypoint.R
# Create Plumber router
root <- plumb("plumber.R")

# Register hooks for logging
api <- root %>% 
    pr_hook("preroute", function(req, res) {
        tictoc::tic()
    }) %>%
    pr_hook("postroute", function(req, res) {
        elapsed_time <- tictoc::toc(quiet = TRUE)$toc
        # Log request and response details
        log_info(paste(
            "Client:", handle_empty(req$REMOTE_ADDR), 
            "| User-Agent:", shQuote(handle_empty(req$HTTP_USER_AGENT)), 
            "| Host:", handle_empty(req$HTTP_HOST), 
            "| Method:", handle_empty(req$REQUEST_METHOD), 
            "| Endpoint:", handle_empty(req$PATH_INFO), 
            "| Status:", handle_empty(res$status), 
            "| Duration:", round(elapsed_time, digits = getOption("digits", 5)), "seconds"
        ))        
    })

api$run(port = 8000)
  • Client IP: The IP address of the client making the request.
  • User Agent: The client’s user agent string.
  • Host: The host receiving the request.
  • Method: The HTTP method used for the request (GET, POST, etc.).
  • Endpoint: The specific API endpoint being accessed.
  • Status: The HTTP status code of the response.
  • Duration: The time taken to process the request.

 

This format is designed to give a clear overview of each API request and response, similar to the NCSA Common log format but tailored for our use case.

For production purpose, it may be important to add usernames, session and request if for auditing purposes.

 

Testing the API

To test the setup, start the API by running entrypoint.R. If you’re using RStudio, you can click the “Run API” button in the IDE.

Once the API is running, you can test the logging by making a request. For instance, open your web browser and navigate to the /histogram endpoint:

				
					http://127.0.0.1:8000/histogram

You should see a histogram displayed in your browser. Meanwhile, the RStudio console and the log file will show an entry like this:

				
					INFO [2024-08-09 14:30:23] Client: 127.0.0.1 | User-Agent: "Mozilla/5.0" | Host: localhost:8000 | Method: GET | Endpoint: /histogram | Status: 200 | Duration: 0.15 seconds

Advanced Error Handling and Logging in Plumber API

Error handling and logging are critical components of any robust API. By properly logging errors, you can monitor issues, debug more effectively, and ensure that your API is reliable. In this section, we’ll walk through how to set up a custom error handler in a Plumber API and log errors consistently, ensuring that all relevant information is captured.

 

Implementing a Custom Error Handler

To handle errors in a structured and reusable way, we define a custom error handler function,

custom_error_handler

, which logs detailed information about the request and the error into helper file.

				
					r
# helper.R
custom_error_handler <- function(req, res, err) {
    elapsed_time <- tictoc::toc(quiet = TRUE)$toc
    
    # Log the error with detailed information
    log_error(
        paste(
            "Client:", handle_empty(req$REMOTE_ADDR), 
            "| User-Agent:", shQuote(handle_empty(req$HTTP_USER_AGENT)), 
            "| Host:", handle_empty(req$HTTP_HOST), 
            "| Method:", handle_empty(req$REQUEST_METHOD), 
            "| Endpoint:", handle_empty(req$PATH_INFO), 
            "| Status:", handle_empty(res$status), 
            "| Duration:", round(elapsed_time, digits = getOption("digits", 5)), "seconds",
            "| Error:", conditionMessage(err)
        )
    )
}

This error handler captures:Client IP: The IP address of the client making the request.User Agent: The user agent string from the client.Host: The host receiving the request.HTTP Method: The HTTP method used (GET, POST, etc.).Endpoint: The specific API endpoint that was accessed.Status: The HTTP status code returned.Duration: The time taken to process the request.Error Message: The error message generated during processing.

This consistent logging format ensures that all error logs have the same structure, making it easier to analyze and troubleshoot issues.

 

Triggering and Handling Errors

To demonstrate this error handling, we create an endpoint /cause_error that intentionally triggers an error:

				
					r
# plumber.R
#* @get /cause_error
#* Endpoint to trigger an error
function(req, res) {
    stop("This is a deliberate error!")
}

We then set up Plumber to use our custom error handler by registering it with pr_set_error:

				
					r
pr_set_error(function(req, res, err) {
    custom_error_handler(req, res, err)  
})

With this setup, whenever an error occurs in the API, the custom_error_handler function is invoked, logging the error and related details.

 

Extending Error Handling with rlang’s abort

For more advanced error handling, you can extend this approach using rlang’s abort functions. rlang::abort() allows you to create custom error classes and attach additional metadata to errors, making it easier to handle different types of errors in a standardized way.

Here’s an example of how you could integrate

rlang::abort:
				
					r
# helper.R
# Define a reusable error function with abort
abort_custom_error <- function(message, class = NULL, ...) {
    rlang::abort(message, class = class, ...)
}
# plumber.R
# Use the custom error function in an endpoint
#* @get /trigger_custom_error
function(req, res) {
    abort_custom_error("A custom error occurred!", class = "custom_error")
}

With this setup, errors are raised with custom classes, which can be caught and handled differently depending on the class. This adds a layer of flexibility and reusability to your error handling, especially in larger applications where different error types might need different responses or logging.

 

Conclusion

By implementing a custom error handler and consistently logging all errors in your Plumber API, you can significantly improve your ability to monitor and debug your API. This setup demonstrates how to globally handle errors in a Plumber API using pr_set_error() and log the errors with user-specific information.

By customizing the error response and logging the details, you gain better control over error management and improve your ability to diagnose issues. This approach is particularly useful in production environments where monitoring and debugging are critical. Additionally, leveraging rlang’s abort functions allows for more advanced and reusable error handling, giving you fine-grained control over how errors are managed and reported. This setup not only enhances the robustness of your API but also makes it easier to maintain and scale as your application grows.

The flexibility of Plumber makes it an excellent tool for building APIs, but with flexibility comes the responsibility of implementing best practices such as logging. By using preroute and postroute hooks, we can create detailed logs that capture both request and response information, offering valuable insights into API performance and utilization.

For more details on using Plumber, logging, and other advanced features, visit the Posit Community.

Stay tuned for our next article, where we’ll explore advanced log analysis, integration with monitoring tools, and building a Shiny dashboard to visualize API metrics in real-time. In the meantime, discover more about our work in financial services by exploring our case studies.

Explore more stories

Dig deeper into data development by browsing our blogs…

Custom Logging System for Plumber API and Shiny Applications A futuristic dashboard showing a central server connected to multiple API endpoints. Lines representing data flow between endpoints show user actions and responses, with real-time monitoring screens and storage for log files. On the left, warning icons indicate error tracking, while on the right, a clock symbolizes automated log management. The background has code snippets and data logs, creating a technical, blue-toned atmosphere that reflects system performance monitoring and troubleshooting.

Custom Logging Plumber API

Introduction Our client, a leading investment consulting firm, aimed to improve system monitoring, troubleshooting, and reliability by implementing a logging system that tracks user actions

Watch now ->
Explore all articles

Get in Touch

Let us leverage your data so that you can make smarter decisions. Talk to our team of data experts today or fill in this form and we’ll be in touch.

Transforming Data, Empowering Decisions

Capabilities

Industries

Blog

Cases Studies

About Us

Partners

Legal

© 2024 Procogia, Inc. All rights reserved.