Support custom Json deserializer/serializer

[volyx]

For example Gson instead Jackson.
Move json to another maven module and allow to add it as separate jar.

[voidmain]

Hi @volyx. Thanks for submitting this request. The use of Jackson in the library is mostly private, meaning that Jackson classes aren't returned from the methods. Is your need for GSON so that you don't need to include Jackson in your classpath?

[volyx]

Yes, I already exclude all jackson libraries from my project for using only Gson, but I can't replace Jackson with Gson in fusionauth-jwt. So it would be nice to have a opportunity to somehow register custom serializer/deserializer. For example

JsonSerializer ser = new GsonSerializer()
 JWT jwt = new JWT()
                .setSerializer(ser)
                .setIssuer("www.example.com")
                .setIssuedAt(iss)
                .setSubject(subject)
                .setExpiration(exp);

[robotdan]

Thanks for that additional info @volyx . In theory we could abstract the mapper interface and allow a different module / library to be bound for serializing and deserializing.

Most of that code is in Mapper.java - however - we are using some Jackson annotations in the domain which makes it a bit tough without some re-work to the library.

We are using the Jackson @AnySetter and @AnyGetter annotations from Jackson, a similar function would need to exist in Gson.

While not ideal, you could fork the repo and just rip out Jackson.

I'm open to this feature in FusionAuth JWT - we'd need to move the Jackson annotations out of the domain and into some other configuration, or perhaps we could use Jackson mixins to decorate the domain with the annotations we'd need. If that was all done, then it wouldn't be too hard to make a new interface for Guice to bind one of several implementations.

This isn't a high priority - but we can leave this open and see if we have some time to hack on it a bit. Feel free to offer some example code or PRs if you have an idea of how it would work.

Thanks!

[volyx]

outdated

As I see there are two options:

1. First is to create a multi-module maven project.
   So fusionauth-jwt-parent will have such structure:

fusionauth-jwt-parent
- jwt
- jacksson-serializer
- fusionauth-jwt(depends on jwt,  jacksjon-serializer)
- gson-serializer

So users can still use fusionauth-jwt library and don't notice any differences after versoin update
2) Do separation in runtime. We could check in runtime availability of JSON library:

if (ObjectMapper class exists) use jackson serserializer
if (Gson class exists) use gson serserializer

So user could exclude Jackson, and add Gson library.

First approach more modular, and allow future extension but needs more work and restructurization.
Second one needs less time, but use some kind of runtime class discovery that is not a very good pattern but still a workable solution.

[volyx]

PoC for removing Jackson

Problem: Jackson libraries are performant but heavy weight and clash with other Jackson libraries in a project

Solution: use lightweight JSON library or do not even use any JSON library - decode/encode manually

Library sizes:

340K jackson-core-2.10.1.jar
66K jackson-annotations-2.10.1.jar
1.3M jackson-databind-2.10.1.jar
136K fusionauth-jwt-3.1.6.jar

fusionauth-jwt library is just 136 Kilobytes.

For comparison
29K nanojson-1.4.jar
33K minimal-json-0.9.5.jar
8.1K JsonReader.java(manually written json reader)

I have added JSON serializers, deserializers which can replace Jackson. I have tested three options: minimal-json, nanojson(embeded), plain(manual)

All them can be tested for compatibility with Jackson ObjectMapper like this:
./mvnw test -Dminimal.json.decode=true use minimal-java for decoding, Jackson for decoding
./mvnw test -Dminimal.json.encode=true use minimal-java for encoding, Jackson for encoding
./mvnw test -Dminimal.json.encode=true -Dminimal.json.decode=true use minimal-java for encoding and decoding

The same is for others encoders/decoders:
./mvnw test -Dnano.json.encode=true -Dnano.json.decode=true
./mvnw test -Dplain.json.encode=true -Dplain.json.decode=true

I wrote benchmark for testing the performance of all these JSON mappers - JsonBenchmark.

Benchmark                                    (serializer)    Mode      Cnt       Score    Error   Units
JsonBenchmark.testDecode                          Jackson   thrpt        5       0,800 ±  0,329  ops/us
JsonBenchmark.testDecode                      MinimalJson   thrpt        5       0,247 ±  0,093  ops/us
JsonBenchmark.testDecode                         NanoJson   thrpt        5       0,063 ±  0,062  ops/us
JsonBenchmark.testDecode                            Plain   thrpt        5       0,538 ±  0,229  ops/us
JsonBenchmark.testDecode                             None   thrpt        5       6,863 ±  1,322  ops/us
JsonBenchmark.testEncode                          Jackson   thrpt        5       0,235 ±  0,040  ops/us
JsonBenchmark.testEncode                      MinimalJson   thrpt        5       0,097 ±  0,013  ops/us
JsonBenchmark.testEncode                         NanoJson   thrpt        5       0,088 ±  0,102  ops/us
JsonBenchmark.testEncode                            Plain   thrpt        5       0,157 ±  0,077  ops/us
JsonBenchmark.testEncode                             None   thrpt        5      24,158 ± 21,829  ops/us
JsonBenchmark.testDecode                          Jackson    avgt        5       1,189 ±  0,222   us/op
JsonBenchmark.testDecode                      MinimalJson    avgt        5       4,250 ±  2,976   us/op
JsonBenchmark.testDecode                         NanoJson    avgt        5      14,250 ±  6,014   us/op
JsonBenchmark.testDecode                            Plain    avgt        5       1,955 ±  2,875   us/op
JsonBenchmark.testDecode                             None    avgt        5       0,137 ±  0,033   us/op
JsonBenchmark.testEncode                          Jackson    avgt        5       5,854 ± 10,353   us/op
JsonBenchmark.testEncode                      MinimalJson    avgt        5      10,302 ± 10,342   us/op
JsonBenchmark.testEncode                         NanoJson    avgt        5       9,826 ±  7,726   us/op
JsonBenchmark.testEncode                            Plain    avgt        5       5,453 ±  0,905   us/op
JsonBenchmark.testEncode                             None    avgt        5       0,043 ±  0,013   us/op
JsonBenchmark.testDecode                          Jackson  sample  1201091       1,692 ±  0,097   us/op
JsonBenchmark.testDecode:testDecode·p0.00         Jackson  sample                0,957            us/op
JsonBenchmark.testDecode:testDecode·p0.50         Jackson  sample                1,024            us/op
JsonBenchmark.testDecode:testDecode·p0.90         Jackson  sample                1,880            us/op
JsonBenchmark.testDecode:testDecode·p0.95         Jackson  sample                2,184            us/op
JsonBenchmark.testDecode:testDecode·p0.99         Jackson  sample                4,200            us/op
JsonBenchmark.testDecode:testDecode·p0.999        Jackson  sample               78,336            us/op
JsonBenchmark.testDecode:testDecode·p0.9999       Jackson  sample              786,226            us/op
JsonBenchmark.testDecode:testDecode·p1.00         Jackson  sample            26869,760            us/op
JsonBenchmark.testDecode                      MinimalJson  sample  1248740       6,599 ±  0,489   us/op
JsonBenchmark.testDecode:testDecode·p0.00     MinimalJson  sample                2,860            us/op
JsonBenchmark.testDecode:testDecode·p0.50     MinimalJson  sample                3,360            us/op
JsonBenchmark.testDecode:testDecode·p0.90     MinimalJson  sample                5,472            us/op
JsonBenchmark.testDecode:testDecode·p0.95     MinimalJson  sample                5,872            us/op
JsonBenchmark.testDecode:testDecode·p0.99     MinimalJson  sample               17,280            us/op
JsonBenchmark.testDecode:testDecode·p0.999    MinimalJson  sample              410,245            us/op
JsonBenchmark.testDecode:testDecode·p0.9999   MinimalJson  sample             4738,070            us/op
JsonBenchmark.testDecode:testDecode·p1.00     MinimalJson  sample            90177,536            us/op
JsonBenchmark.testDecode                         NanoJson  sample   925259      14,588 ±  0,234   us/op
JsonBenchmark.testDecode:testDecode·p0.00        NanoJson  sample                7,344            us/op
JsonBenchmark.testDecode:testDecode·p0.50        NanoJson  sample               10,256            us/op
JsonBenchmark.testDecode:testDecode·p0.90        NanoJson  sample               15,440            us/op
JsonBenchmark.testDecode:testDecode·p0.95        NanoJson  sample               21,440            us/op
JsonBenchmark.testDecode:testDecode·p0.99        NanoJson  sample               59,802            us/op
JsonBenchmark.testDecode:testDecode·p0.999       NanoJson  sample              663,552            us/op
JsonBenchmark.testDecode:testDecode·p0.9999      NanoJson  sample             2514,731            us/op
JsonBenchmark.testDecode:testDecode·p1.00        NanoJson  sample            20054,016            us/op
JsonBenchmark.testDecode                            Plain  sample  1526500       2,457 ±  0,138   us/op
JsonBenchmark.testDecode:testDecode·p0.00           Plain  sample                1,480            us/op
JsonBenchmark.testDecode:testDecode·p0.50           Plain  sample                1,572            us/op
JsonBenchmark.testDecode:testDecode·p0.90           Plain  sample                2,788            us/op
JsonBenchmark.testDecode:testDecode·p0.95           Plain  sample                3,184            us/op
JsonBenchmark.testDecode:testDecode·p0.99           Plain  sample                6,416            us/op
JsonBenchmark.testDecode:testDecode·p0.999          Plain  sample               98,304            us/op
JsonBenchmark.testDecode:testDecode·p0.9999         Plain  sample              891,597            us/op
JsonBenchmark.testDecode:testDecode·p1.00           Plain  sample            54001,664            us/op
JsonBenchmark.testDecode                             None  sample  1371906       0,821 ±  0,246   us/op
JsonBenchmark.testDecode:testDecode·p0.00            None  sample                0,139            us/op
JsonBenchmark.testDecode:testDecode·p0.50            None  sample                0,163            us/op
JsonBenchmark.testDecode:testDecode·p0.90            None  sample                0,297            us/op
JsonBenchmark.testDecode:testDecode·p0.95            None  sample                0,311            us/op
JsonBenchmark.testDecode:testDecode·p0.99            None  sample                0,554            us/op
JsonBenchmark.testDecode:testDecode·p0.999           None  sample               19,744            us/op
JsonBenchmark.testDecode:testDecode·p0.9999          None  sample              801,597            us/op
JsonBenchmark.testDecode:testDecode·p1.00            None  sample            38010,880            us/op
JsonBenchmark.testEncode                          Jackson  sample  1411865       4,719 ±  0,049   us/op
JsonBenchmark.testEncode:testEncode·p0.00         Jackson  sample                3,468            us/op
JsonBenchmark.testEncode:testEncode·p0.50         Jackson  sample                3,584            us/op
JsonBenchmark.testEncode:testEncode·p0.90         Jackson  sample                5,936            us/op
JsonBenchmark.testEncode:testEncode·p0.95         Jackson  sample                7,056            us/op
JsonBenchmark.testEncode:testEncode·p0.99         Jackson  sample               15,568            us/op
JsonBenchmark.testEncode:testEncode·p0.999        Jackson  sample              128,913            us/op
JsonBenchmark.testEncode:testEncode·p0.9999       Jackson  sample              849,538            us/op
JsonBenchmark.testEncode:testEncode·p1.00         Jackson  sample             4415,488            us/op
JsonBenchmark.testEncode                      MinimalJson  sample  1263047      10,270 ±  0,091   us/op
JsonBenchmark.testEncode:testEncode·p0.00     MinimalJson  sample                7,064            us/op
JsonBenchmark.testEncode:testEncode·p0.50     MinimalJson  sample                7,632            us/op
JsonBenchmark.testEncode:testEncode·p0.90     MinimalJson  sample               13,536            us/op
JsonBenchmark.testEncode:testEncode·p0.95     MinimalJson  sample               15,920            us/op
JsonBenchmark.testEncode:testEncode·p0.99     MinimalJson  sample               40,448            us/op
JsonBenchmark.testEncode:testEncode·p0.999    MinimalJson  sample              265,728            us/op
JsonBenchmark.testEncode:testEncode·p0.9999   MinimalJson  sample             1106,095            us/op
JsonBenchmark.testEncode:testEncode·p1.00     MinimalJson  sample            10371,072            us/op
JsonBenchmark.testEncode                         NanoJson  sample  1378816       9,582 ±  0,330   us/op
JsonBenchmark.testEncode:testEncode·p0.00        NanoJson  sample                7,032            us/op
JsonBenchmark.testEncode:testEncode·p0.50        NanoJson  sample                7,856            us/op
JsonBenchmark.testEncode:testEncode·p0.90        NanoJson  sample                9,344            us/op
JsonBenchmark.testEncode:testEncode·p0.95        NanoJson  sample               12,928            us/op
JsonBenchmark.testEncode:testEncode·p0.99        NanoJson  sample               23,232            us/op
JsonBenchmark.testEncode:testEncode·p0.999       NanoJson  sample              164,911            us/op
JsonBenchmark.testEncode:testEncode·p0.9999      NanoJson  sample             1223,867            us/op
JsonBenchmark.testEncode:testEncode·p1.00        NanoJson  sample           116391,936            us/op
JsonBenchmark.testEncode                            Plain  sample  1085856       6,361 ±  0,711   us/op
JsonBenchmark.testEncode:testEncode·p0.00           Plain  sample                4,512            us/op
JsonBenchmark.testEncode:testEncode·p0.50           Plain  sample                4,848            us/op
JsonBenchmark.testEncode:testEncode·p0.90           Plain  sample                5,688            us/op
JsonBenchmark.testEncode:testEncode·p0.95           Plain  sample                8,192            us/op
JsonBenchmark.testEncode:testEncode·p0.99           Plain  sample               15,303            us/op
JsonBenchmark.testEncode:testEncode·p0.999          Plain  sample              122,752            us/op
JsonBenchmark.testEncode:testEncode·p0.9999         Plain  sample             1221,105            us/op
JsonBenchmark.testEncode:testEncode·p1.00           Plain  sample           228589,568            us/op
JsonBenchmark.testEncode                             None  sample  1201647       0,136 ±  0,016   us/op
JsonBenchmark.testEncode:testEncode·p0.00            None  sample                0,027            us/op
JsonBenchmark.testEncode:testEncode·p0.50            None  sample                0,073            us/op
JsonBenchmark.testEncode:testEncode·p0.90            None  sample                0,105            us/op
JsonBenchmark.testEncode:testEncode·p0.95            None  sample                0,133            us/op
JsonBenchmark.testEncode:testEncode·p0.99            None  sample                0,288            us/op
JsonBenchmark.testEncode:testEncode·p0.999           None  sample                9,904            us/op
JsonBenchmark.testEncode:testEncode·p0.9999          None  sample               61,941            us/op
JsonBenchmark.testEncode:testEncode·p1.00            None  sample             2854,912            us/op
JsonBenchmark.testDecode                          Jackson      ss           118276,270            us/op
JsonBenchmark.testDecode                      MinimalJson      ss             8362,015            us/op
JsonBenchmark.testDecode                         NanoJson      ss            21499,819            us/op
JsonBenchmark.testDecode                            Plain      ss             3517,849            us/op
JsonBenchmark.testDecode                             None      ss              100,837            us/op
JsonBenchmark.testEncode                          Jackson      ss           106437,109            us/op
JsonBenchmark.testEncode                      MinimalJson      ss            14163,948            us/op
JsonBenchmark.testEncode                         NanoJson      ss            20346,964            us/op
JsonBenchmark.testEncode                            Plain      ss             1746,693            us/op
JsonBenchmark.testEncode                             None      ss               40,539            us/op

Jackson and Plain mappers are comparable by average speed and throughput. So, can we maybe completely replace Jackson with own custom JsonReader/PlainObjectMapper implementation? WDYT? @robotdan

Code is here - https://github.com/volyx/fusionauth-jwt/tree/minimal-json

PlainObjectMapper - https://github.com/volyx/fusionauth-jwt/blob/minimal-json/src/main/java/io/fusionauth/jwt/json/PlainObjectMapper.java

NanoJsonObjectMapper - https://github.com/volyx/fusionauth-jwt/blob/minimal-json/src/main/java/io/fusionauth/jwt/json/NanoJsonObjectMapper.java

MinimalJsonObjectMapper - https://github.com/volyx/fusionauth-jwt/blob/minimal-json/src/main/java/io/fusionauth/jwt/json/MinimalJsonObjectMapper.java

[robotdan]

Wow, thanks for the excellent and thorough suggestion @volyx

I'll take a look at you're proposal and follow up with comments.

[virtual-machinist]

Can this be pluggable (i.e. done via an interface)? While I do understand that not all projects use Jackson, it's de facto standard for anyone using Spring Boot (and possibly other popular web frameworks). Meaning that it wouldn't really add any extra dependencies. Pluggability may also help with writing simpler tests (possibly allow to test the validation and JSON parsing logic separately) and allow further Jackson mapper customization via the usual module/mixin/options route.

This will not be as easy as throwing away Jackson completely (you may have to do Mixins for serialization/deserialization to keep using Jackson annotations and do default ObjectMapper configuration), but avoids depending on yet-another-custom-json-parser that's only used here.

[volyx]

@virtual-machinist

it's de facto standard for anyone using Spring Boot (and possibly other popular web frameworks).

• For spring projects common way of integration is separate module for example spring-fusionauth-jwt. So if we suppose that it is spring project we can manually configure JSON library.
• For other frameworks it is also possible to write custom integration library - ${other_popular_web_framework}-fusionauth-jwt.

This will not be as easy as throwing away Jackson completely

Example is here - PlainObjectMapper. It is as fast as Jackson and uses just standard Java APIs.

I don't see any reason to support custom JSON serialization, if I can do encode/decode internally without any dependencies. PlainObjectMapper is fully compatible with ObjectMapper for all fusion-auth classes JWT, Header, Map, JSONWebKey. You can check it with:

// will encode with PlainObjectMapper, decode with ObjectMapper
./mvnw test -Dplain.json.encode=true

// will decode with PlainObjectMapper, encode with ObjectMapper
./mvnw test -Dplain.json.decode=true 

[volyx]

The same approach of embedding JSON serialization/deserialization code inside library is used in jose4j - https://bitbucket.org/b_c/jose4j/src/master/src/main/java/org/jose4j/json/

[virtual-machinist]

First of all I don't think speed is the main reason anyone uses Jackson (or Spring for that matter). Jackson and Gson provide a wide range of serialization/deserialization customization options, are well established, with decent documentation and have plenty integrations with other libraries.

As a developer I don't want to write glue logic just to interface with the embedded serializer of a yet-another-jwt-library with my Jackson object model when I have the option of not doing so if I use a library that natively supports Jackson, even if it's a little slower/bulkier/...
I think I may even lose any gained performance doing DTO-to-claim mapping vs configuring the serializer to write claims the way I need.

I'd like to emphasize that I'm not against having a custom-very-fast-json-library even as the default option, but please don't make it hard not using it.