Skip to content

OBPIH-7888 part 2: Integrating ObjectMapper#5985

Open
ewaterman wants to merge 12 commits into
feature/receiving-redesignfrom
ft/OBPIH-7888-2-simpler-serialize
Open

OBPIH-7888 part 2: Integrating ObjectMapper#5985
ewaterman wants to merge 12 commits into
feature/receiving-redesignfrom
ft/OBPIH-7888-2-simpler-serialize

Conversation

@ewaterman

Copy link
Copy Markdown
Member

✨ Description of Change

Link to GitHub issue or Jira ticket: https://pihemr.atlassian.net/browse/OBPIH-7888

Description: Integrates JSON serialization with Jackson ObjectMapper.

This gives us two things:

  1. Allows us to use Jackson annotations on our DTOs
  2. Unless you're doing some special mapping of DTO fields to JSON, you are no longer required to define toJson or asResponseBody methods on our DTOs.

I'll try to record a video explaining the changes but I'll also leave comments throughout

@ewaterman ewaterman self-assigned this Jun 18, 2026
@github-actions github-actions Bot added type: feature A new piece of functionality for the app domain: backend Changes or discussions relating to the backend server flag: config change Hilights a pull request that contains a change to the app config labels Jun 18, 2026

// We don't know how to serialize the object so we will rely on the framework to do it for us.
return toSerialize
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic was moved to ObjectMapperConfigurer. The JsonSerializer now just wraps the data in a standard structure then calls ObjectMapper

* any controller that implements {@link org.pih.warehouse.core.BaseController}.
*/
@Component
class ObjectMapperConfigurer {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file has the meaningful change. I tried to add verbose docstrings explaining the behaviour and why everything here is needed.

* as long as there is a Mapper defined between the two.
*/
@Component
class SmartMapper {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't actually end up using this component. I can remove it (and the changes to MapperComponentResolver) if that would make things clearer, but I figure this will be useful at some point so I left it in. It lets us do stuff like:

X xEntity = new X(...)
XDto xDto = smartMapper.map(xEntity, XDto)

which will automatically resolve the mapper to use for converting from X to XDto and call it.

@Override
Map<String, Object> asResponseBody() {
return [
product: product?.asResponseBody(),

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is one of the main advantages of the new implementation. Not only does it automatically handle serializing the object, but it automatically serializes each field as well. So the ProductSimpleDto will also get serialized automatically (and if ProductSimpleDto was still implementing ResponesBodyFormattable, it would have used that method to serialize it).

* @return The total quantity received for the shipment item across all receipt, including not submitted receipts.
*/
@JsonProperty("totalQuantityReceived")
int getTotalQuantityReceived() {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an example of how we can include transient getter methods as fields in the JSON response using @JsonProperty

@ewaterman ewaterman changed the title OBPIH-7888 part 2; Integrating ObjectMapper OBPIH-7888 part 2: Integrating ObjectMapper Jun 18, 2026
if (stack.size() > MAX_JSON_DEPTH) {
gen.writeNull()
return
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these checks on ALREADY_SEEN_OBJECTS and MAX_JSON_DEPTH weren't required to get this working, but as I was developing this I hit a few infinite loops/stack overflow errors due to some misconfigurations I had. I fixed those cases but I figured I'd leave these checks in for protection.

@jmiranda jmiranda Jul 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These checks are exactly what scare me about this solution.

  1. I think I mentioned this somewhere else, but the default max depth seems super deep unless there's more to this than I'm understanding. And I know it's warding off the worst-case scenario (infinite loop), but what are the performance implications for letting the serializer do a kinda infinite loop here e.g. an expensive getter somewhere in the nested object graph that gets invoked on every nested objected. Naive example, but it could happen.

  2. In general, I actually want serialization to fail pretty loudly (e.g. StackOverflowError) if it means we're doing something wrong (serializing deeply nested domain objects unnecessarily) or being lazy (not implementing a DTO).

But if we do need to keep the handle circular reference logic, we should log.warn/error'ing those so we can watch those pile up and make that the signal to map a particular domain to a DTO.

With that said, I would prefer to disable the Hibernate module and make us intentionally map from domain to DTO.

Important

This does not mean we want a DTO for every domain class now.

I think that's an important point, so I want to pause to let that sink in. We don't need to flesh out and implement the entire DTO class hierarchy right now. We just need to implement the DTOs that we want to expose the resources via the API right now. I understand there are core classes like (Location, Person, User) that require us to flesh out more than just the cycle count classes, but I think we could get away with most of those being "short" references. In fact, (although I'm not advocating for this), but it seems you could use @JsonIdentify (discussed below) to create simple short DTO versions of all the core domain classes just to get that out of the way.

Note

In fact, I would actually love to spend some time on the following before we flesh out too much of the DTO layer.

  • Defining conventions (replace Dto suffix with more useful, meaningful name like Details, Summary, Payload, etc)
  • Design the DTO layer upfront so we limit the number of mistakes to avoid API deprecation notices. Again we don't need to implement all of the classes upfront, we just need to show what we're expecting.

A really good example of where we might go wrong is with InventoryLevel. I don't want to expose InventoryLevel as a resource. I want it to expose CycleCountConfig, ReplenishmentRules, ForecastingConfig, etc.
Another good example is Location and how we should expose Supplier vs Facility vs Location (internal bin, receiving bay)

From my perspective, if we go the DTO-always route instead of the DTO-plus-fallback-to-wrapper route, we don't need this wrapper class at all. In fact, I don't think we would want to handle circular references ourselves anway. Instead we'd want the DTO layer to be mapped properly so that we explicitly prevent circular references and we want Jackson to be loud when we do it wrong. This should be possible by either referencing the "shorter" DTOs that explicitly short-circuit the nested objects or through Jackson config.


Aside: I did some cursory research into the Jackson mechanisms that are used for the circular references and found the following.

Honorable Mentions for generally useful features we need to look into and document as we move forward:

  • @JsonAutoDetect - actually, it seems all of our DTO classes should use this annotation to opt-in (so this probably needs to be moved
  • @JsonIdentityReference - just the id, ma'am. nothing but the id. this one should actually move to the list above.
  • @JsonView - tag fields with view classes (Summary, Detail) and serialize with a chosen view. i think this is probably close to what i was think re: different representations of the same object (OBPIH-2649)
  • JsonFilter + FilterProvider - holy shit, runtime include/exclude. that's dope.
  • JsonSerialize - point to a custom serializier

Holy shit, there's so much more in Jackson than I expected!!! I'm starting to really lean into the idea of building this DTO layer and using Jackson. **I know some of the comments here are critical, bordering on harsh, but I want to make it clear that I think this is the right direction. ** @ewaterman

None of the Jackson top three mechanisms help us with the Hibernate serialization problem since our domains do not (and will not) be configured to serialize via Jackson config placed on the domain class (at least that's my opinion today). I could change my mind on that but I think the map domain to DTO is the safer and more correct route.

Note: The will not is emphatic for the same reasons we're building the DTO layer in the first place:

  • property visibilty - we should control what properties we expose
  • Hibernate - we don't want to deal with lazy loading,
  • circular references (
  • security (we haven't even discussed the cases where certain roles/users should not see a given property)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, there's been a set of PRs and tickets I had on my mind since I first started to review the original object mapper PR. I found one ticket that mentioned the DTO layer and a bunch of PRs. There are probably other OBPIH tickets that include constraints, requirements, conventions but I'll have to do that look up later.

@kchelstowski created a ticket to advocate for this back in 2023 (https://pihemr.atlassian.net/browse/OBPIH-5993). Back then I was expecting us to move to the Grails @resource mechanism to map domains to REST resources, so I was ok with using domain and command classes to define and expose properties for the API resource. But that wasn't necessarily going to solve some of the issues we had around representing and serializing objects via the API.

  • different representations of the same domain class
  • hiding properties based on role
  • hiding properties that should not be exposed publicly
  • avoid circular references

So I want to acknowledge that I think @kchelstowski was right about introducing the DTO layer and this PR by @ewaterman is a sensible first step towards implementing that vision (particularly the adoption of Jackson and it's configurability and extensibility).


Aside: For years, I have been thinking about creating an Architecture Decision Record to keep track of the decisions we're making here (and elsewhere), scoped very narrowly to what we're actually deciding. So I'm going to create that repository now.

In our case here there would likely be multiple ADRs

  • ADR-0001 - Canonical API serialization mechanism (with decision to adopt Jackson as our default serialization framework, but name the existing approach as well as other approaches i.e. MapStruct, @Resource, etc)
  • ADR-0002 - Create a DTO layer to act as the default representation of resources exposed via API, webhook payloads, file exports, etc. Need to discuss all of the constraints and requirements for this. Don't need to solve them all right now.
  • There are probably a dozen more ADRs to draft here, so I'll try to

To do that I want to make sure we capture all of the aforementioned (and unnamed) requirements, constraints, conventions from PRs and tickets over the years. It doesn't have to be exhaustive, but we should make sure that the adoption of Jackson will either solve the problem or not get in our way when we try to tackle the problem in the future. A good future problem to solve would be the "hide properties based on role", so doing a discovery on that might be a next step of the ADR.

Note

Aside: We'll need to something similar when we look into Spring Security as our default security mechanism. I feel like this is going to be a more difficult decision based on the complexity and opiniated nature of Spring Security.

I had Claude pull up all of the discussions we've had in pull requests re: the "DTO layer" over the years. Here are those discussions, grouped by PR, each link with a one-line summary. I want to extract all of the "requirements" and "conventions" that were brought up so we can create a

PR #4435 — OBPIH-6015 (Dec 2023 – Jan 2024) · your foundational DTO-layer position
r1437926214 (2023-12-29) — Don't add a DTO layer without a requirement; revert to a default toJson() on ProductSupplier unless there's a compelling reason.
r1437927243 (2023-12-29) — Follow existing conventions or invest in "v2"; your recommended convention: Map toJson() on the domain, marshaller in Bootstrap, BaseDomainApiController, and separate controller actions for list/search.
r1449813011 (2024-01-12) — Prefer toJson() returning a map; don't litter the code with DTOs if you might refactor to JSON views later; offered the JSON-views path as a "give it a shot" option.
r1449817169 (2024-01-12) — On LazyInitializationException: suggested testing by having toJson() reach into an association not loaded by the criteria query — i.e., you flagged the lazy-loading serialization risk two years ago.
PR #5976 — OBLS-781 Bulk APIs (June 2026) · recent, a specific DTO
r3418092959 (2026-06-16) — Suggested returning the saved/validated objects, or a DTO that wraps them and renders to JSON.
r3422157231 (2026-06-16) — Disliked returning "a list of maps with an arbitrary shape" and magic strings ("ok"/"error"); proposed a concrete UpsertResult DTO with UpsertStatus/UpsertAction enums.
r3422171203 (2026-06-16) — Sketched the method signatures returning UpsertResult / List.
r3422183403 (2026-06-16) — Allowed keeping the current shape for now, but noted you may want a separate DTO class to wrap the list of UpsertResult.
(The later #5976 comments — r3455901632, r3464270929, r3464282636 — are about the two-session/transaction-proxy and product-availability-refresh issue, not the DTO layer.)

PR #5968 — OBPIH-7867 Start a receipt (June 2026) · tangential, API response shape
r3396565077 (2026-06-11) — On 201 responses: you favor returning a Location header (HTTP-spec compliant) and having the client fetch, but noted you were overruled on the stock-movement APIs since the object is almost always needed immediately.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to reiterate that the serializer wrapper is not for supporting hibernate. We support hibernate via the config line:

objectMapper.registerModule(new Hibernate5Module())

So if we want to not support hibernate, all we do is remove that line.

The serializer wrapper is what allows ResponseBodyFormattable and ResponseMapper to work. Even if we remove hibernate serialization support, I still expect both of those to be used in combination with a DTO if we need to customize the serialization logic beyond what Jackson offers. They are enhancements to the built in serialization logic.

So even if we disable hibernate, we still need the serializer wrapper.

The serializer wrapper also adds the circular dependency and max depth checks. I like having these checks because by default the ObjectMapper will keep serializing until you stack overflow, which I do not want to happen on prod. I can definitely change the logic there to complain loudly with a helpful error message if we hit some invalid max depth (I reduced it to 25), but I want to be extra sure that we don't bring down prod with some misconfigured DTO serialization that we didn't catch during testing for whatever reason.

I do think we'd benefit from standardizing our approach to serialization (what annotations to use and when, when to introduce ResponseBodyFormattable or a ResponseMapper) and having consideration for things like dynamically hiding fields based on user role. That would be a good follow up discovery. I suspect the latter can be supported via an additional hook in on this serialization wrapper (another reason to keep it around).

@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 35.62500% with 103 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (feature/receiving-redesign@7c2739f). Learn more about missing BASE report.

Files with missing lines Patch % Lines
.../warehouse/core/http/ObjectMapperConfigurer.groovy 49.41% 37 Missing and 6 partials ⚠️
...y/org/pih/warehouse/core/mapper/SmartMapper.groovy 11.76% 15 Missing ⚠️
...rehouse/core/mapper/MapperComponentResolver.groovy 18.75% 12 Missing and 1 partial ⚠️
...g/pih/warehouse/inventory/CycleCountItemDto.groovy 0.00% 5 Missing ⚠️
...groovy/org/pih/warehouse/core/mapper/Mapper.groovy 0.00% 4 Missing ⚠️
...y/org/pih/warehouse/location/BinLocationDto.groovy 0.00% 3 Missing ⚠️
...oovy/org/pih/warehouse/location/FacilityDto.groovy 0.00% 3 Missing ⚠️
.../org/pih/warehouse/location/LocationTypeDto.groovy 0.00% 3 Missing ⚠️
...n/groovy/org/pih/warehouse/location/ZoneDto.groovy 0.00% 3 Missing ⚠️
...org/pih/warehouse/product/ProductCatalogDto.groovy 0.00% 3 Missing ⚠️
... and 6 more
Additional details and impacted files
@@                      Coverage Diff                      @@
##             feature/receiving-redesign    #5985   +/-   ##
=============================================================
  Coverage                              ?   11.15%           
  Complexity                            ?     1771           
=============================================================
  Files                                 ?      872           
  Lines                                 ?    48600           
  Branches                              ?    11381           
=============================================================
  Hits                                  ?     5420           
  Misses                                ?    42303           
  Partials                              ?      877           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Integrates Jackson ObjectMapper-based JSON serialization into the API response pipeline, enabling Jackson annotations on DTOs and reducing the need for per-DTO asResponseBody() / toJson() methods.

Changes:

  • Introduces ObjectMapperConfigurer and a wrapping serializer to apply OpenBoxes-specific mapping rules during Jackson serialization.
  • Refactors many DTOs to rely on Jackson field serialization (and selective @JsonProperty on computed getters) instead of ResponseBodyFormattable.
  • Updates controller/test infrastructure (JsonSerializer, BaseController, and controller specs) to render JSON via Jackson.

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/test/groovy/testutil/ObjectMapperTestInitializer.groovy Adds a test-only ObjectMapper initializer (incl. ignoring Spock $* fields).
src/test/groovy/testutil/ApiControllerSpec.groovy Updates controller test setup to use Jackson-backed JsonSerializer.
src/main/groovy/org/pih/warehouse/shipping/ShipmentItemDto.groovy Removes ResponseBodyFormattable serialization in favor of Jackson field serialization.
src/main/groovy/org/pih/warehouse/shipping/ContainerSimpleDto.groovy Removes ResponseBodyFormattable serialization in favor of Jackson field serialization.
src/main/groovy/org/pih/warehouse/receiving/ShipmentReceivingSummaryDto.groovy Removes ResponseBodyFormattable serialization in favor of Jackson field serialization.
src/main/groovy/org/pih/warehouse/receiving/ShipmentItemReceivingSummaryDto.groovy Replaces asResponseBody() with Jackson annotations for computed getters.
src/main/groovy/org/pih/warehouse/receiving/ReceiptSaveResponseDto.groovy Removes ResponseBodyFormattable serialization in favor of Jackson field serialization.
src/main/groovy/org/pih/warehouse/receiving/ReceiptItemSaveDto.groovy Removes asResponseBody() override and relies on Jackson to include subclass fields.
src/main/groovy/org/pih/warehouse/receiving/ReceiptItemDto.groovy Removes ResponseBodyFormattable serialization; DTO will now serialize by field names.
src/main/groovy/org/pih/warehouse/receiving/ReceiptDto.groovy Removes ResponseBodyFormattable serialization in favor of Jackson field serialization.
src/main/groovy/org/pih/warehouse/product/ProductSimpleDto.groovy Removes ResponseBodyFormattable serialization in favor of Jackson field serialization.
src/main/groovy/org/pih/warehouse/product/lot/ProductLotDto.groovy Removes ResponseBodyFormattable serialization in favor of Jackson field serialization.
src/main/groovy/org/pih/warehouse/location/LocationSimpleDto.groovy Removes ResponseBodyFormattable serialization in favor of Jackson field serialization.
src/main/groovy/org/pih/warehouse/inventory/CycleCountCandidateMapper.groovy Updates mapper class to implement the new ResponseMapper trait form.
src/main/groovy/org/pih/warehouse/core/PersonDto.groovy Removes ResponseBodyFormattable serialization in favor of Jackson field serialization.
src/main/groovy/org/pih/warehouse/core/OrderedDataGroup.groovy Removes ResponseBodyFormattable serialization in favor of Jackson field serialization.
src/main/groovy/org/pih/warehouse/core/mapper/SmartMapper.groovy Adds a convenience service for mapping via Mapper components.
src/main/groovy/org/pih/warehouse/core/mapper/ResponseMapper.groovy Converts ResponseMapper from abstract class to trait.
src/main/groovy/org/pih/warehouse/core/mapper/MapperComponentResolver.groovy Extends resolver to support Mapper components and source+target lookup.
src/main/groovy/org/pih/warehouse/core/mapper/Mapper.groovy Converts Mapper from interface to trait with generic source/target type helpers.
src/main/groovy/org/pih/warehouse/core/mapper/BidirectionalMapper.groovy Converts BidirectionalMapper from abstract class to trait.
src/main/groovy/org/pih/warehouse/core/http/ResponseBodyFormattable.groovy Removes the static collection helper from ResponseBodyFormattable.
src/main/groovy/org/pih/warehouse/core/http/ObjectMapperConfigurer.groovy Adds Jackson configuration and a wrapping serializer for OpenBoxes custom mapping rules.
src/main/groovy/org/pih/warehouse/core/http/JsonSerializer.groovy Switches JSON serialization from Grails JSON to Jackson ObjectMapper.
grails-app/controllers/org/pih/warehouse/core/BaseController.groovy Updates JSON rendering to render Jackson-produced JSON strings with explicit content type/encoding.
build.gradle Adds jackson-datatype-hibernate5 to support Hibernate entity serialization.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/main/groovy/org/pih/warehouse/core/mapper/MapperComponentResolver.groovy Outdated
Comment thread src/main/groovy/org/pih/warehouse/receiving/ReceiptItemDto.groovy
Comment thread src/main/groovy/org/pih/warehouse/core/mapper/SmartMapper.groovy Outdated

@jmiranda jmiranda left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I like the idea of Jackson + DTOs, I don't think we've actually made a decision or even had a discussion about whether it makes sense to use over existing solutions (JSON views, object marshaller). Grails handles serialization fairly well, we've just been using a solution that worked best for us in Grails 1.3.x and didn't migrate to JSON views when we moved to Grails 3.

Where I would argue that Jackson probably gets my vote is when we're starting to talk about serializing objects for other non-web payloads like webhook events OR we need to deserialize an object as easy as we serialize. But we're not really taking either of those into account right now so the move to Jackson feels premature.

Rolling our a serialization layer (yes, we're using Jackson but we're also adding in our own wrapper) feels like a decent bridge to get to a more consistent state with serialization, but it deserves a deep technical device to make sure it's the right decision since it's going to take a lot of work. However, as long as this doesn't break the existing render object as JSON convention then I can live with it as we learn and migrate towards Jackson as our serialization solution.

Comment thread src/main/groovy/org/pih/warehouse/receiving/ReceiptItemDto.groovy

// If an object being serialized has no fields to serialize, don't throw an error,
// just ignore the object. This gracefully resolves Grails/GORM AST transformation weirdness.
objectMapper.disable(SerializationFeature.FAIL_ON_EMPTY_BEANS)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But wouldn't keeping this enabled be a good indication that we didn't flesh out our API model well enough? This error is a signal to developers to get their shit together, no? It also makes sure our API is completely backed by DTOs, which is what I think we're trying to do here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't remember exactly why I needed this. I figured that it'd be pretty difficult to unintentionally create a DTO that has no fields, and even if we did we'd likely notice right away that the API response is empty


// Add support for serializing Hibernate domain entities (though we should typically
// prefer converting an entity to a DTO and serializing the DTO instead).
objectMapper.registerModule(new Hibernate5Module())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah if we're going to commit to Jackson and DTO for some APIs, I think we need to commit for all. The existing APIs with toJson() marshallers should continue to work (correct?) therefore we don't really need backwards compatibility, we just need to slowly migrate to Jackson + DTO for everything over time.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes this is only going to affect new APIs that extend BaseController so we could say that we ALWAYS require a DTO. As I mention in a previous comment though, it might make using this logic a headache (at least at first) because we'd need to create a bunch of DTO classes for our entities just to have it be able to properly serialize something

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this would only be applicable for the controllers that would extend the BaseController, I would be restrictive and force us not to serialize domains if possible.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried disabling serializing Hibernate entities but it's already a mess for cycle count since it doesn't use DTOs very often. I can keep this restriction if we really want it, but it's going to balloon this PR somewhat because I'll need to add a bunch of new dtos (or at least have the domains implement asResponseBody). Honestly I'd rather just allow Hibernate entities for now but make a goal for us to move away from it. Not allowing domains will make it harder to integrate existing APIs with the BaseController

thoughts?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry just getting back to this tonight. As I mentioned above in the ObjectMapperConfig comments, I would love to disable Hibernate because "make a goal for us to move away from it" is usually something we're usually good at. Once it's in place, we can use the "fixme later" excuse. If we take a hardline here, I think it'll be better for us in the future, even if it adds cost upfront. We're pretty good dealing with upfront cost because we have inertia in the moment.

So I think I would prefer the asResponseBody approach (which I assume could just call the toJson on the domain class right)? So it's not really add much more.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another issue that I've had trouble articulating has to do with asResponseBody() and ResponseBodyFormattable. I haven't quite figured out how to solve it, but response body is very specific to web requests.

However, what I think we're trying to do is add a transport-agnostic method of serializing/deserializing objects, whether we render to or ingest from json, csv, xml, etc. At least that's my hope for what we're trying to accomplish. Whereas "repsonse body" makes this 💯 about the lifecycle of an API request. I want this serialization solution to handle rendering an API response, preparing a webhook payload, exporting a file, or producing an integration message that can be sent to a Kafka queue or SFTP server.

On top of that this same framework should be cabaple of handling deserialization without any changes to the object mapper. In other words, it should work bi-directionally. That might just be something we deal with later, but I don't want it to be too much of afterthought.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with having a single, transport-agnostic serialization method is that different formats need to be structured differently. JSON and XML support complex nested data structures but CSV and XLS do not. The data must be flat for CSVs and XLS because it is just rows and columns.

Take a look at ResponseMapper. I was trying to split out the cases in a way that is agnostic of the underlying format. I ended up with:

  • asResponseBody for JSON and XML -> maybe "asHierarchicalData"?
  • asExportRow for CSV and XLS -> maybe "asBulkDataRow" or "asTabularRow"?


// Groovy makes the fields themselves "private" (because it auto-generates getters) so
// we need to set field visibility to ANY to be able to auto-serialize fields.
objectMapper.setVisibility(PropertyAccessor.FIELD, JsonAutoDetect.Visibility.ANY)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the consequence of restricting this to none? And what would we need to do in order to counter it? Adding the JsonProperty annotation to all fields? In our toJson we're explicitly including / excluding properties. Here it seems like we'll be including any property in the DTO. I'm not saying we shouldn't include them all, just want to know the behavior so we can make the appropriate choice. For me, I'd rather have the implicit behavior of including everything. But there's definitely maybe going to be a case where we need to include a property in the DTO for context, but don't want to include it in the JSON response.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes by default all fields are included and you'd annotate a field with @JsonIgnore if you want to exclude it.

If we did the opposite and by default exclude all fields, you'd need @JsonProperty on all fields you want to include

Comment on lines +38 to +39
objectMapper.setVisibility(PropertyAccessor.GETTER, JsonAutoDetect.Visibility.NONE)
objectMapper.setVisibility(PropertyAccessor.IS_GETTER, JsonAutoDetect.Visibility.NONE)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the idea and respect the nauanced reason for the different between this policy and the one on fields. Playing devil's advocate I think if we're including the fields by default, then at least getters fit into that implictly included set. At least for me, every getter is a transient property. Booleans also fall into that for me. The nuance is that including getters is a potential performance issue so we need to be careful and thoughtful about what to include. I don't necessarily security plays into this at all, but I could be wrong.

Again, not advocating for one over the other. As much as I like opinionated solutions (i think implicit property inclusion falls into that category), I would be ok taking the opposite (explicit) approach for API response bodies.

One additional caveat (might be instructive for the policy below) is that the explicit getter / property inclusion potentially avoids breaking changes because we inadvertantly exposed a property we didn't mean to.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a case for restricting access to everything and and applying annotations in the DTO layer.

objectMapper.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.NONE)

I assume this would allow us to use @JsonAutoDetect for when we really do want to go the implicit route. I know that breaks for the domains but I would argue we should not try to serialize the Hibernate domains at all. Which is a vote for getting rid of the Hibernate module and enabling fail on empty beans.

That'll force us to create DTOs for everything that needs to be exposed via the API and make good decisions on what properties get exposed.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

including getters is a potential performance issue so we need to be careful and thoughtful about what to include

this is the strongest argument to not include them IMO. It's very easy to make a db query in a getter method and so I want us to be explicit about the getter methods that we actually want to include

we should not try to serialize the Hibernate domains at all

I agree but we already do this a lot in the app. I didn't want to force a dev to make a bunch of new DTOs for existing domains just to get their API serializing properly. I imagine it'd disincentivize using this flow since the existing serialization does support entities. I figured this was the path of least resistence

There's a case for restricting access to everything and and applying annotations in the DTO layer.

I'd be potentially open to this idea

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does the mechanism of PropertyAccessor work to determine if something is a getter? Does it check if a method name (getter) starts with get*? If it doesn't, I'm worried that it can treat all the class methods as getters and try to include any methods without arguments in the serialization?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah it checks if the method name starts with get* or is*. But the way I have it configured now is that it always ignores methods and only serializes fields

Comment on lines +244 to +253
ResponseMapper responseMapper = mapperComponentResolver.getResponseMapper(valueClass)
if (responseMapper) {
return responseMapper.asResponseBody(value)
}
if (value instanceof ResponseBodyFormattable) {
return value.asResponseBody()
}
if (value.metaClass.respondsTo(value, "toJson")) {
return value.toJson()
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoa. This is both awesome and terrifying. I think this deserves a tech huddle.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah we can discuss. For the record, we're already doing this. I just moved the logic from JsonSerializer.serializeObject to here.

New classes shouldn't use toJson. ResponseBodyFormattable or a response mapper is better. I only included toJson to support preexisting dtos that do it that way.

* How many layers deep into a JSON object we allow before short circuiting and returning null.
* TODO: Make this an application property once we're confident in this flow.
*/
private static final int MAX_JSON_DEPTH = 50

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of a config property, but this is potentially a bad idea. If you're going 50 levels deep, something went wrong. I would argue that giving someone the ability to configure the app to go any deeper would potentially cause more harm than good.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it prevent the circular dependency? E.G. receipt has receiptItems returned, but receiptItem returns receipt in its DTO, so we end up with an infinite loop? I think Grails' converter did a great job in that, and it potentially cut the dependency to include only id for a child, that would potentially return the parent again and again.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember there were some jackson's annotations that needed to be added both sides, to determine which one is the child and which one is the parent, to avoid that problem. Probably @JsonManagedReference and @JsonBackReference

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with @JsonManagedReference and @JsonBackReference is that the @JsonBackReference is essentially always excluded when serializing.

What this should do is as you describe. See handleCircularReference. It detects that we've already seen the object and so sticks only its id in (or if it doesn't have an id it calls toString on it). So it should behave like the Grails converter

* This works at any level in the object hierarchy. If a DTO contains a field that implements one of the above
* custom serialization methods, that field will be serialized via that method.
*/
class OpenBoxesWrappingSerializer extends JsonSerializer<Object> implements ResolvableSerializer, ContextualSerializer {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll be honest, I would be in favor of getting rid of this JSON serializer layer if the cost was "we need to create more DTOs".

Image

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not about having more DTOs. It's about hooking our DTO/entity -> Map conversion logic (which is the toJson() or asResponseBody() methods) into the ObjectMapper.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My point was that this is trying to solve the problem of we don't have enough DTOs to cover all of the APIs, so we need something to bridge the gap with the existing domain classes. I am arguing that we should only use the jackson-powered object mapper with the DTO layer and not try to make it work with the marshallers and asResponseBody() methods.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the "More DTO" point was just a comment on how we get from where we are (mixed serialization) to where we want to be (a single serialization approach using Jackson). Each API controller will need to be migrated or we just leave the existing controllers alone for the time being. Any new API uses the Jackson approach. In either case, the objective should be Everything that goes out over the wire needs a DTO which means we need to add more DTOs.

* At the end of that process, the BeanSerializerFactory iterates over each BeanSerializerModifier, calling
* modifySerializer to extend/wrap it with the specified behaviour.
*/
class OpenBoxesBeanSerializerModifier extends BeanSerializerModifier {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll be honest. This thing scares me. So question: Does this go away if/when we migrate away from the other two solutions?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what other solutions? I get that this config is intimidating (it was for me too at first) but essentially all this is doing is allowing us to hook in custom serialization logic that triggers before the default ObjectMapper logic.

And the custom logic is automatically calling any toJson(). asResponseBody() methods that a DTO has

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what other solutions?

I'm just talking about the existing serialization solutions we use:

  • ObjectMarshaller + toJson()
  • ResponseBodyFormattable
Image

I get that this config is intimidating (it was for me too at first) but essentially all this is doing is allowing us to hook in custom serialization logic that triggers before the default ObjectMapper logic.

It's not that it's intimitading. It's just seems like a workaround. And it seems like it's only necessary because we cannot implement your ideal approach across the board (DTO+Jackson). This is not a criticism, it's a reality we have to deal with because we have existing serialization solutions in use. So now we now need to add a wrapper in order to make sure it all works happily together. This is only problematic because implementing this workaround (instead of the ideal solution of DTO+Jackson) leads me to believe it has the potential for harm.

Another way to frame this: If we started from scratch with DTO+Jackson for the entire API, would this class be necessary? If the new APIs are using just the DTO+Jackson approach do we need our own serialization wrapper class. The answer might still be yes. But if this wrapper is necessary even with DTO+Jackson+sensible config solution then I would double-down and say maybe this isn't the right solution.

I think I understand the point of this class ... it's to ease our burden / bridge the gap between where we are (using a mix of serialization approaches) vs the ideal (Jackson-based serialization). But, if we're going down this route of rewiring serialization across the board, I don't actually want to ease the burden with workarounds and wrappers because that's potentially where all the bugs will be coming from. It's possible that this PR nails all of the issues we might encounter (re: circular dependency, explicit vs implicit properties and getters, Hibernate issues, how are exceptions handled - fail silently vs exception).

But if it doesn't we're potentially going to be chasing our tailing to debug/troubleshoot issues we didn't anticipate, applying more and more workarounds, and getting to a place where we're further from a more consistent, stable, and maintainable solution.

I don't mean to be a curmudgeon. I say all this having seen how complicated serialization becomes through the OpenMRS sync project (maybe we can ask @mseaton to see what he thinks).

@ewaterman ewaterman Jul 8, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we started from scratch with DTO+Jackson for the entire API, would this class be necessary?

It is not necessary to get Jackson working but it is not a work around. We don't need it to support existing structures. I'm trying to enhance the Jackson objectmapper with additional functionality.

There are two main enhancements that I want to support:

  1. auto-serializing when we have a ResponseMapper component for a DTO
  2. auto-serializing when we have a DTO that implements ResponseBodyFormattable

Both of these solutions are new, and this serialization wrapper class adds support for those enhancements. With this class, when serializing a DTO, this wrapper allows Jackson to invoke the ResponseMapper associated with that DTO, or call that DTO's asResponseBody method if it implements ResponseBodyFormattable.

In most cases we should not need 1 or 2. We can just use Jackson annotations and that is enough. The enhancements I added are meant to add additional functionality that allows us to easily define custom, more complicated serialization behaviour.

For example, if you need to call into some component when converting to json, the way we do it now is statically access that component inside the DTO's toJson method using Holders.grailsApplication.mainContext.getBean(X), which feels really hacky. If instead you define a ResponseMapper you can do it like this:

@Component
class XMapper implements ResponseMapper<X> {

    MessageLocalizer messageLocalizer
    SomeComponent someComponent

    @Override
    Map<String, Object> asResponseBody(X source) {
        return [
            field1: messageLocalizer.localize("x.y.z", [source.field1])
            field2: someComponent.doSomething(source.field2)
        ]
    }
}

which keeps the logic out of the DTO, letting it be a simple wrapper on fields.

So if we serialize X to json, thanks to the logic in OpenBoxesBeanSerializerModifier, Jackson will know to call XMapper.asResponseBody.

And similarly, if we want to customize the serialization process beyond what Jackson supports but we don't want to bother creating a whole ResponseMapper component we can use ResponseBodyFormattable:

@Component
class X implements ResponseBodyFormattable {

    String field1
    String field2

    @Override
    Map<String, Object> asResponseBody() {
        return [
            field1: field1,
            field2: field2,
            somethingElse: field1 + field2,
        ]
    }
}

And again thanks to the logic in OpenBoxesBeanSerializerModifier, if we serialize X to json, Jackson will know to call its asResponseBody() method.

(That's not the greatest example because you can still use Jackson annotations to solve it, but hopefully you get the idea.)

But if you didn't need to do either of the above and just want to serialize the fields of X as they are, then all you need to do is:

class X {
    String field1
    String field2
}

and Jackson will serialize that to:

{
    "field1": "a",
    "field2": "b",
}

So no this OpenBoxesBeanSerializerModifier is not required to support Jackson. I need it to support using ResponseMapper and ResponseBodyFormattable as custom serialization methods, which I created for the sake of convenience for devs when they need more custom serialization behaviour than what Jackson alone can support.

@ewaterman

Copy link
Copy Markdown
Member Author

@jmiranda I wanted to use Jackson for a few reasons:

  1. It's the standard serialization process for Spring (which is why Grails already has the ObjectMapper bean even though it doesn't use it) so is very well supported/documented
  2. The annotations are quite convenient for customizing how a DTO gets serialized
  3. Jackson is apparently orders of magitude faster than Grails' serialization (but I have yet to test this)
  4. No more needing to register the DTO with Grails' marshaller in BootStrap.groovy

However, as long as this doesn't break the existing render object as JSON convention then I can live with it as we learn and migrate towards Jackson as our serialization solution.

yes we can still do as JSON. This only impacts controllers that extend the newly added BaseController (which is just receiving currently).

I can appreciate wanting to have a discussion about the pros/cons first. I'm happy to discuss on a tech huddle. I was hoping that we'd be able to test it out in isolation with the receiving feature to iron out the kinks and assess performance before using it more broadly going forward.

@kchelstowski kchelstowski left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this generally looks good to me, but as usually, I have a hard time trying to figure out, how the flow works:

  1. Having a DTO defined, what are the steps that the mapper goes through?
  2. What's the priority of the mapper components? Which one is the "main" one, which are the helper ones, etc. I was trying to figure out where the ObjectMapperConfigurer is included, but couldn't figure that out. Is there any "sequence" of particular classes being used for a serialization?
  3. Ad2: in a nutshell - we serialize a ReceiptItemDto - what are the steps? where does it start the mapping, where does it end? what methods are called throughout the whole cycle?

Comment on lines +38 to +39
objectMapper.setVisibility(PropertyAccessor.GETTER, JsonAutoDetect.Visibility.NONE)
objectMapper.setVisibility(PropertyAccessor.IS_GETTER, JsonAutoDetect.Visibility.NONE)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does the mechanism of PropertyAccessor work to determine if something is a getter? Does it check if a method name (getter) starts with get*? If it doesn't, I'm worried that it can treat all the class methods as getters and try to include any methods without arguments in the serialization?


// Add support for serializing Hibernate domain entities (though we should typically
// prefer converting an entity to a DTO and serializing the DTO instead).
objectMapper.registerModule(new Hibernate5Module())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this would only be applicable for the controllers that would extend the BaseController, I would be restrictive and force us not to serialize domains if possible.

Comment thread src/main/groovy/org/pih/warehouse/core/http/ObjectMapperConfigurer.groovy Outdated
* How many layers deep into a JSON object we allow before short circuiting and returning null.
* TODO: Make this an application property once we're confident in this flow.
*/
private static final int MAX_JSON_DEPTH = 50

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it prevent the circular dependency? E.G. receipt has receiptItems returned, but receiptItem returns receipt in its DTO, so we end up with an infinite loop? I think Grails' converter did a great job in that, and it potentially cut the dependency to include only id for a child, that would potentially return the parent again and again.

* How many layers deep into a JSON object we allow before short circuiting and returning null.
* TODO: Make this an application property once we're confident in this flow.
*/
private static final int MAX_JSON_DEPTH = 50

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember there were some jackson's annotations that needed to be added both sides, to determine which one is the child and which one is the parent, to avoid that problem. Probably @JsonManagedReference and @JsonBackReference

Comment thread src/main/groovy/org/pih/warehouse/core/http/ObjectMapperConfigurer.groovy Outdated

ResponseMapper responseMapper = mapperComponentResolver.getResponseMapper(valueClass)
if (responseMapper) {
return responseMapper.asResponseBody(value)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember you saying we don't have to define the asResponseBody anymore for DTO. Where does this asResponseBody come from now? How does the flow work?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we still support implmenting a ResponseMapper or extending ResponseBodyFormattable if you need to customize the serialization for a DTO.

For example you can have:

class SomeDto implements ResponseBodyFormattable {
    String field
  
    @Override
    Map<String, Object> asResponseBody() {
        return [
                field: field,
        ]
    }
}

but because all you're doing in the above is serializing all the fields as they are, you don't actually need to do this and can just do:

class SomeDto {
    String field
}

and it'll do the equivalent of the first version.

But if you want to customize the serialization in a non-standard way or if you just want to be explicit about it, you can still implement ResponseBodyFormattable and define the asResponseBody() method.

The main goal of the PR is to avoid the need for the boilerplate in the case where you're not doing anything custom with it.

It'd make sense to still implement ResponseBodyFormattable if you were doing something like the following:

class SomeDto implements ResponseBodyFormattable {
    String field
    String field2
  
    @Override
    Map<String, Object> asResponseBody() {
        return [
                newField: field + field2,
        ]
    }
}

because you're customizing the response JSON in a non-standard way


@Component
class CycleCountCandidateMapper extends ResponseMapper<CycleCountCandidate> {
class CycleCountCandidateMapper implements ResponseMapper<CycleCountCandidate> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did it need its own "Mapper"? I haven't seen that before

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it because we serialize the domain, and we don't have a DTO for the CycleCountCandidate?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the ResponseMapper it's primarily to break the serialization logic out into its own component for the case where we need it to depend on other components.

In this case we don't depend on other components so we don't strictly need a dedicated ResponseMapper (we could have just had CycleCountCandidate implement ResponseBodyFormattable), but I broke it out because the logic here was getting long, and because I eventually want to use asExportRow as well once we support the new exporter flow

@ewaterman

Copy link
Copy Markdown
Member Author

I removed support from serializing Hibernate entities and it now returns a (hopefully helpful) error message if you try to do so. It ballooned the review by another 250 lines but it appears to be working

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: backend Changes or discussions relating to the backend server flag: config change Hilights a pull request that contains a change to the app config type: feature A new piece of functionality for the app

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants