Datasonnet is a very powerful tool to have in a Spring Boot application because it can speed up development time considerably and is only a very small efficiency difference compared to Java POJO's. But utilizing datasonnet definitely comes with some pitfalls, the biggest one is that you have to rethink your api set up compared to vanilla Spring. The first time I used datasonnet in Spring, the set up was simple. In my controller class the mapped methods would only take in three possibilities - uri parameters, query parameters, and a string request body. The basic idea was to add all the uri paramters and query paramaters to a separate Java map and use the main the datasonnet map with the additional variables. The actual datasonnet script would be stored in a resource file and would just be loaded in. This works fine for some basic uses but it starts to get really sloppy when you have larger applications. So what are all the problems with DataSonnet that need to be solved?
-
Input Validation
With a normal Spring Boot application, the input validation is automatically handled using Objects & annotations such as @Required,@NotNull,@Size,@Length. But when working on a DataSonnet focused application you revolve arround strings which are not easily evaluated.
-
Swagger Console
Its not uncommon for a REST API to have a console of some sort but many spring consoles are built using POJOs where as I needed one built from an OASv3 specification. Theres also some caveats with my particular solution to discuss when deploying behind a proxy.
-
Database Queries
Spring JPA Data is also largely based on POJOs but you wouldnt want to map the response to an object and then convert it to a string, that would create too much overhead.
-
HTTP Requestor
HTTP Requests are pretty similar to normal but they need to be set to use the String class, but stick arround and youll get a bonus HTTP Request configuration
Input Validation
Since DS uses Strings, some the normal entity annotations in spring wont really help us in our application. Some people may be thinking to map the request to an object for the validation then just convert it to a string for datasonnet mapping. This idea would work but its not an optimal solution in both the programming and cpu cycle aspect. When researching open api schema validators I ran across openapi.tools which displays all sorts of tools meant to be used for OAS. Each tool has a listed language support an OAS version supported and a small description of what the tool accomplishes. Under data validators I found the tool openapi4j. Openapi4j is a Java tool that will parse our OASv3 file and validate that a given request follows the specification. If it fails we can set up custom error handling depending on how it failed.
Setup openapi4j
First, lets add our dependency to maven.
<!-- Dependency for validation using HttpServlet -->
<dependency>
<groupId>org.openapi4j</groupId>
<artifactId>openapi-operation-servlet</artifactId>
<version>1.0.4</version>
</dependency>
<!-- Main input validation -->
<dependency>
<groupId>org.openapi4j</groupId>
<artifactId>openapi-operation-validator</artifactId>
<version>1.0.4</version>
</dependency>
Now to begin, we have to load in our api spec at the start of our spring application.
@SpringBootApplication
public class DsInSbApplication {
static OpenApi3 apiSpec;
public static RequestValidator validator;
public static void main(String[] args) throws ResolutionException, ValidationException {
//Get URL path of file in resources folder
URL apiFile = Thread.currentThread().getContextClassLoader().getResource("oas.yaml");
//parse the file to generate OpeanApi3 object
apiSpec = new OpenApi3Parser().parse(apiFile, true);
//validator object that handles the validation
validator = new RequestValidator(DsInSbApplication.apiSpec);
SpringApplication.run(DsInSbApplication.class, args);
}
public static OpenApi3 getApiSpec(){return apiSpec;}
}
Finally, my original solution to validate every request was to put the check in a function in the controller, this check would either return a ResponseEntity<?> or a null object. The controller endpoint would then call this function and wrap it in an Objects.requireNonNullElse() so that if it was null it would call the service, if it wasnt it would return the generated error. This is great but theres a simpler solution using Interceptors. If you dont know, an Interceptor class wraps the controller classes in spring so you can add custom logic before and after the controller class. In our case, we want to validate the input before the controller. We will need an Interceptor calss and a InterceptorConfig class.
@Component
public class ControllerInterceptor implements HandlerInterceptor {
private final Logger log = LoggerFactory.getLogger(this.getClass());
//This array is for diasabling validation for certain endpoints
private final String[] skippable = new String[]{
"/console",
"/api-docs",
"/favicon.ico",
"/swagger-ui",
"/error"
};
//executes before the controller class, returns false on error
@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
//skip console and Spring related paths
if(Arrays.stream(skippable).anyMatch(request.getServletPath()::startsWith)) {
log.info("[Data Validation] Skip");
return true;
}
Request apiSpecRequest = ServletRequest.of(request);
try {
//validate the request follows spec
DsInSbApplication.validator.validate(apiSpecRequest);
log.info("[Data Validation] Pass");
return true;
} catch (ValidationException e) {
log.info("[Data Validation] Failure");
Map<String, ArrayList<String>> errorMap = new HashMap<>();
e.results().items().forEach( item -> {
//map the validation responses to their crumb
String reducedMsg, replacedCrumb;
if(item.dataCrumbs().isEmpty()){
replacedCrumb = "other";
reducedMsg = item.toString();
}
else {
reducedMsg = item.toString().substring(
item.toString().indexOf(":") + 2,
item.toString().indexOf("("));
replacedCrumb = item.dataCrumbs().replace("body","payload");
}
//create empty value if absent
errorMap.putIfAbsent(replacedCrumb, new ArrayList<>());
//append arraylist
errorMap.get(replacedCrumb).add(reducedMsg);
});
//log the errors
errorMap.forEach( (name,errors) -> log.info(name +": " + String.join(",", errors)));
//map the overall error message
Map<String, Object> payload = new HashMap<>();
payload.put("message", "Failed input validation.");
payload.put("errors", errorMap);
response.setStatus(400);
response.setContentType(MediaType.APPLICATION_JSON_VALUE);
response.getWriter().write((new ObjectMapper()).writeValueAsString(payload));
return false;
}
}
}
@Configuration
public class ControllerInterceptorConfig extends WebMvcConfigurerAdapter {
@Autowired
ControllerInterceptor controllerInterceptor;
@Override
public void addInterceptors(InterceptorRegistry registry) {
registry.addInterceptor(controllerInterceptor);
}
}
Because this interceptor uses a HttpServlet and openapi4j pulls the body from the servlet's input stream, the input stream will no longer be readable from the controller function. This is due to how input streams operate. To resolve this you need to cache the request body so that it can be read more than once. This cache configuration comes from baeldung.com
CachedBodyHttpServletRequest.java
public class CachedBodyHttpServletRequest extends HttpServletRequestWrapper {
private byte[] cachedBody;
public CachedBodyHttpServletRequest(HttpServletRequest request) throws IOException {
super(request);
InputStream requestInputStream = request.getInputStream();
this.cachedBody = StreamUtils.copyToByteArray(requestInputStream);
}
@Override
public ServletInputStream getInputStream() throws IOException {
return new CachedBodyServletInputStream(this.cachedBody);
}
@Override
public BufferedReader getReader() throws IOException {
// Create a reader from cachedContent and return it
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(this.cachedBody);
return new BufferedReader(new InputStreamReader(byteArrayInputStream));
}
}
CachedBodyServletInputStream.java
public class CachedBodyServletInputStream extends ServletInputStream {
private InputStream cachedBodyInputStream;
public CachedBodyServletInputStream(byte[] cachedBody) {
this.cachedBodyInputStream = new ByteArrayInputStream(cachedBody);
}
@Override
public boolean isFinished() {
try {
return cachedBodyInputStream.available() == 0;
} catch (IOException e) {
e.printStackTrace();
}
return false;
}
@Override
public boolean isReady() {
return true;
}
@Override
public void setReadListener(ReadListener readListener) {
throw new UnsupportedOperationException();
}
@Override
public int read() throws IOException {
return cachedBodyInputStream.read();
}
}
ContentCachingFilter.java
@Order(value = Ordered.HIGHEST_PRECEDENCE)
@Component
@WebFilter(filterName = "ContentCachingFilter", urlPatterns = "/*")
public class ContentCachingFilter extends OncePerRequestFilter {
@Override
protected void doFilterInternal(
HttpServletRequest httpServletRequest,
HttpServletResponse httpServletResponse,
FilterChain filterChain)
throws ServletException, IOException {
CachedBodyHttpServletRequest cachedBodyHttpServletRequest = new CachedBodyHttpServletRequest(httpServletRequest);
filterChain.doFilter(cachedBodyHttpServletRequest, httpServletResponse);
}
}
Now your api validation should be setup and you should not run into any problems with trying to read from an input stream that has already closed.
Swagger Console
Springdoc and Springfox are two pretty known console dependencies for spring application. Our setup will still use springdoc, but sice it does not use entiteis we will have a few a-typical configurations to add in. First of course, is adding the dependency to our applications pom file.
<dependency>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-ui</artifactId>
<version>1.4.4</version>
</dependency>
Because we dont use entities, spring doc will not be able to generate out api-docs automatically, instead we will have to disable the spring-docs and provide our own endpoint for serving the specification file.
#In my use case, we use a context path of /api which springdoc will append
server.servlet.context-path=/api
#disable the autogenerated api-docs
springdoc.api-docs.enabled=false
#set our api-docs endpoint and ui endpoint
springdoc.api-docs.path=/api-docs
springdoc.swagger-ui.path=/swagger-ui
Now that we disabled the autogenerated api-docs we need to provide the /api-docs endpoint that will return the specification file.
Console.java
@RestController
public class Console {
private final Logger log = LoggerFactory.getLogger(this.getClass());
//this endpoint is for springdoc to parse our specification for the console
@GetMapping(value = "/api-docs")
public ResponseEntity getSwaggerDoc() throws IOException {
String value = Utilities.getFileFromResources("specification.yaml");
return ResponseEntity.status(200).body(value);
}
}
Utilities.java
public class Utilities {
public static String getFileFromResources(String fileName) throws IOException {
BufferedReader reader = new BufferedReader(
new InputStreamReader(
new ClassPathResource(fileName).getInputStream()));
return reader.lines().collect(Collectors.joining("\n"));
}
}
Finally, we do need to add a couple of configuation beans to our main application, so we are going to append our DsInSbApplication.java file.
DsInSbApplication.java
@SpringBootApplication
public class DsInSbApplication {
......//other stuff
@Bean
SpringDocConfiguration springDocConfiguration(){
return new SpringDocConfiguration();
}
@Bean
public SpringDocConfigProperties springDocConfigProperties() {
return new SpringDocConfigProperties();
}
}
That should be it. you should now be able to access the console at host:port/api/swagger-ui (endpoint is context-path + springdoc.swagger-ui.path defined in properties file)
Caveats
When you make a request to /swagger-ui to access the console springdoc does some redirection magic to point you to the correct endpoint containing the specification data. In a local environment, this works great. But when deployed behind a reverse proxy you need some special headers which you can read more about here. Kong currently does not allow setting a few of these headers so I had to come up with a work around, my solution was to simply handle the redirection inside the application itself and return the end result to the user. I accomplished this by appending a /console endpoint to the Console.java file.
Console.java
//loop back function to negate the redirect
@GetMapping(value = "/console")
public ResponseEntity<?> getSwaggerConsole() throws Exception {
WebClient webClient = WebClient.create("http://localhost:8080");
String body;
try {
body = webClient.get()
//my spring doc endpoint
.uri("/api/swagger-ui/index.html?configUrl=/api/api-docs/swagger-config")
.retrieve()
.bodyToMono(String.class)
.block();
//this fixes some location issues in the JavaScript
body = body.replaceAll("src=\".", "src=\"./swagger-ui")
.replaceAll("href=\".", "href=\"./swagger-ui")
.replace("url: \"https://petstore.swagger.io/v2/swagger.json\",", "url: \"/api/api-docs\",");
return ResponseEntity.ok(Objects.requireNonNull(body));
} catch (WebClientResponseException ex){
log.error(ex.getLocalizedMessage());
throw new Exception();
}
}
Database Queries
You have a couple of options for database queries, You can either create a custom repository with an entity manager, or you can use a normal JpaRepository with a dummy entity. Personally, a custom repository interface is a bit of work so I went the JPA route. Either option you choose, you will probably have to write more native SQL than a normal spring application. I would recommend stored procedures and functions where possible
DummyEntity.java
@Entity
public class DummyEntity {
@Id
private int id;
}
Then you just create your repository class like normal
BillionairesRepository.java
@Repository
public interface BillionairesRepository extends JpaRepository<DummyEntity, Integer> {
@Query(value = "...", nativeQuery = true)
String getAllBillionaires();
}
For the actual SQL queries, you have a few options but I can only really recommend one which is built in JSON functions. Many databases now adays have similar built in JSON_OBJECT functions though the syntax may diffentiate. Even the embeded h2 database I use in my examples has JSON_OBJECT and JSON_ARRAY functions. So the idea is simple, use the database to generate a JSON value and return it as a string. Then you can use that string in DataSonnet however you need it or you can directly return the string to the user and just modify the content type to json. Heres an example query I used for getAllBillionaires
SELECT CAST(JSON_ARRAY(
(SELECT JSON_OBJECT(
'id'\:b.id,//you need to escape the : due to JPA variables
KEY 'first_name' VALUE b.first_name, //Just another format
'last_name'\:b.last_name,
'career'\:b.career)
FROM billionaires b)) as varchar)
This will return a string containing a json array of json objects. When it comes to inserting data, most databases have methods to parse json data which combined with a Stored Procedure means you can just priave a json object with all the values. In the cases that dont support it, you can use datasonet to pull the values from the payload and a Repository function to accept the parameters and execute the insert.
HTTP Requestor
The http requests using a web client will be pretty similar to normal but instead of casting the input and response to an object we need to cast it to the String class, heres an example post:
String formattedString = String.format(path.replaceAll("{(.*?)}", "%s"), (Object[]) uriParams);
log.info("[API Call] [POST] " + getApiUrl() + formattedString);
try {
return webClient.method(HttpMethod.POST)
.uri(getApiUrl() + path, (Object[]) uriParams)
.headers(httpHeaders -> {
httpHeaders.setContentType(MediaType.APPLICATION_JSON);
httpHeaders.set(HttpHeaders.AUTHORIZATION, auth);
})
.body(Mono.just(payload), String.class).retrieve()
.toEntity(String.class).block();
}catch (WebClientResponseException ex){
//Log the error here, then re-throw the error so the caller can do something with it
log.error("Error posting path {}. Status code is {} and the message is {}\nPayload attempted:\n{}\n Authorization attempted:{}",
formattedString, ex.getRawStatusCode(), ex.getResponseBodyAsString(), payload,auth);
throw ex;
}
For the bonus, I created an easily configurable HTTPRequestProperties class that uses a builder class to easily generate apiCall functions. then you just need to provide the data. First you need a configuration class.
CustomHttpPropertyConfigurer
@Configuration
public class CustomHttpPropertyConfigurer {
@Bean
@ConfigurationProperties("downstream.api.one")
public HttpRequestConfiguration httpRequestApiOne(){
return new HttpRequestConfiguration();
}
@Bean
@ConfigurationProperties("downstream.api.two")
public HttpRequestConfiguration httpRequestApiTwo(){
return new HttpRequestConfiguration();
}
}
The configuration properties need to relate to some set properties in your application.properties file for instance,
#-----------------------------------------------
# Application
#-----------------------------------------------
#---------- Api one -------
downstream.api.one.hostname=https://host1.com
downstream.api.one.port=443
downstream.api.one.baseurl=/api
#---------- Api two -------
downstream.api.two.hostname=http://host2.com
downstream.api.two.port=80
downstream.api.two.baseurl=/api
Then you just need the properties class
HttpRequestProperties.java
@Configuration
@ConfigurationProperties("downstream.properties") //default properties variable
public class HttpRequestConfiguration {
private String hostname;
private String port;
private String baseurl;
private String apiUrl;
public HttpRequestConfiguration(){}
public HttpRequestConfiguration(String hostname, String port, String baseurl) {
this.hostname = hostname;
this.port = port;
this.baseurl = baseurl;
this.apiUrl = hostname+":" +port+""+baseurl;
}
public String getHostname() {
return hostname;
}
public void setHostname(String hostname) {
this.hostname = hostname;
}
public String getPort() {
return port;
}
public void setPort(String port) {
this.port = port;
}
public String getBaseurl() {
return baseurl;
}
public void setBaseurl(String baseurl) {
this.baseurl = baseurl;
}
public String getApiUrl(){
if(apiUrl == null){
this.apiUrl = hostname+":" +port+""+baseurl;
}
return apiUrl;
}
public RequestBuilder getRequestBuilder(WebClient webClient, String urlPath){
return new RequestBuilder(webClient, getApiUrl(), urlPath);
}
public static class RequestBuilder {
private HttpMethod method;
private WebClient webClient;
private Class<?> clazz = Object.class;
private String urlPath;
private Object payload = "";
private List<String> urlParams = List.of();
private Map<String, String> headers = Map.of();
private String baseUrl;
private RequestBuilder() { }
public RequestBuilder(WebClient webClient, String baseUrl, String urlPath) {
this.webClient = webClient;
this.baseUrl = baseUrl;
this.urlPath = urlPath;
}
public RequestBuilder setMethod (HttpMethod method){
this.method = method;
return this;
}
public RequestBuilder setMethod (String method){
this.method = HttpMethod.resolve(method);
return this;
}
public RequestBuilder setClass (Class < ? > clazz){
this.clazz = clazz;
return this;
}
public RequestBuilder setUrlPath (String urlPath){
this.urlPath = urlPath;
return this;
}
public RequestBuilder setBody (Object payload){
this.payload = payload;
return this;
}
public RequestBuilder setHeaders (Map < String, String > headers){
this.headers = headers;
return this;
}
public RequestBuilder setUrlParameters (List < String > urlParams) {
this.urlParams = urlParams;
return this;
}
public ResponseEntity<?> get() throws WebClientResponseException {
this.method = HttpMethod.GET;
return execute();
}
public ResponseEntity<?> post() throws WebClientResponseException {
this.method = HttpMethod.POST;
return execute();
}
public ResponseEntity<?> patch() throws WebClientResponseException {
this.method = HttpMethod.PATCH;
return execute();
}
public ResponseEntity<?> put() throws WebClientResponseException {
this.method = HttpMethod.PUT;
return execute();
}
public ResponseEntity<?> delete() throws WebClientResponseException {
this.method = HttpMethod.DELETE;
return execute();
}
private ResponseEntity<?> execute () throws WebClientResponseException {
return webClient.method(method)
.uri(baseUrl + urlPath, urlParams.toArray())
.headers(httpHeaders -> httpHeaders.setAll(Objects.requireNonNullElse(headers, Map.of())))
.body(Mono.just(payload), String.class).retrieve()
.toEntity(clazz).block();
}
}
}
To use the property, you would autowire an HttpRequestProperties with that same name as the function you defined in the CustomHttpPropertyConfigurer and thats it. In order to make a request you would call the getRequestBuilder inside the configuration, use the setters to set your request data then finally finish it off with a method call like get or post.
Conclusion
You can view an entire example application with everything discussed above implemented in my blogs repository on github which can be found here.
