Pathom Updates 10, Dynamic Integration begins!

Hello everyone, it’s time for the 10th edition of Pathom updates!

Error handling

I’ve made some changes to how we process attributes in Pathom 3. This is what Pathom 3 does now to check if a given attribute has some error:

Check the new Error Handling documentation page covering all the details!

Pathom Viz

The first news, Pathom Viz application is now signed on Mac 🎉! Now you will be able to open it without having to go around the gatekeeper (by doing the weird process of right-click, open, …).

The signed app also enables auto-updates! Once you get Pathom Viz 2021.5.3, the follow-up versions will be automatically downloaded and installed.

Now a list of other fixes and improvements:

• On the graph view, double click will center the graph contents (great when you accidentally scroll away and lose track of the graph)
• Fix bug that some maps with mixed keys could crash Pathom Viz
• Add some new error boundaries to prevent local issues from crashing the whole app
• CSS based on Tailwind JIT, makes loading faster (compare to the previous usage of tailwind-garden)
• Fixed bug where blank trace data could crash the app
• Fix trace bug when all children disappear when the root children have over 20 items
• Ensure consistent background on CM6
• Show node details on snapshots
• Improve performance of snapshots rendering by lazy processing the elements
• Fix bug on the trace that made tooltip stay on screen
• Remove node zoom on click on the graph view
• In the request tab, recent requests now show on top of the list
• Request tab items now have a border at the right to make it easier to scroll the list
• Request tab max size increased to cover five lines of query

Planner Optimizations

In the last update, I talked about planner changes that removed the optimizations on the plan. Now I started to get them back.

To show the current state, let’s first remember what the optimal graph looked like for this example request (8 nodes):

Now let’s remember its state after deoptimization (57 nodes):

Finally, the current state with the new initial optimizations (29 nodes):

The new optimizations now happen after the initial (unoptimized) graph is done. Below here you can see the steps to build this plan.

Note that in the === Optimize === step, the graph is the same as the unoptimized one. Then you can see the steps the algorithm takes to reduce the number of nodes.

The current optimizations are conservative. Right now its more important to have a “correct graph” than it is to have a fully “optimized graph”. The optimizations it looks for and does currently are:

• On AND nodes, look for repeated resolvers and merge them
• If an AND node has only a single branch and no after node, simplify removing the AND node

That’s it. Currently, Pathom doesn’t attempt to optimize OR nodes at all. As I played with those, they proved to be tricky to optimize generally. I want to see more different use cases of what happens on graphs to give another step here.

For static resolvers, those optimizations don’t matter that much. It’s when we get to dynamic resolvers that they turn important.

Distributed Pathom

The major goal of dynamic resolvers is to enable the integration of distributed Pathom systems.

You may already use Pathom to do this, connecting different HTTP services, for example.

This means different Pathom instances, that may be running in different machines.

To illustrate the idea, imagine we have servers A and B.

Server A has the following setup, capable of responding for the attribute :x:

(ns com.acme.services.server-a
  (:require
    [com.wsscode.pathom3.connect.built-in.resolvers :as pbir]
    [com.wsscode.pathom3.connect.indexes :as pci]
    [com.wsscode.pathom3.interface.eql :as p.eql]))

(def env
  (pci/register (pbir/constantly-resolver :x 10)))

(p.eql/process env [:x])
; => {:x 10}

Server B, capable of resolving attribute :y:

(ns com.acme.services.server-b
  (:require
    [com.wsscode.pathom3.connect.built-in.resolvers :as pbir]
    [com.wsscode.pathom3.connect.indexes :as pci]
    [com.wsscode.pathom3.interface.eql :as p.eql]))

(def env
  (pci/register (pbir/constantly-resolver :y 20)))

(p.eql/process env [:y])
; => {:y 20}

Now imagine if we have a client that wants to request both :x and :y, like:

(integrator [:x :y])
; => {:x 10, :y 20}

How can we implement an integrator?

If we had access to the sources of the services, we could merge the envs:

(defn integrator [tx]
  (p.eql/process (pci/register env-a env-b) tx))

But that’s not the case I’m looking for. Those are services and the goal is to use them as services. Like at the image below:

We can implement the resolvers individually in the third service:

(pco/defresolver remote-x []
  {:x (get (request-service-a [:x]) :x)})

(pco/defresolver remote-y []
  {:x (get (request-service-b [:y]) :y)})

(def env-c
  (pci/register
    [remote-x remote-y]))

(defn integrate [tx]
  (p.eql/process env-c tx))

This approach works but requires adding extension code for every single addition in the servers, with the potential for new bugs at each new thing.

Is it possible to automate this integration? I think so.

The indexes are the key point for this to work. If the client can access the indexes of servers (the data part, excluding the functions of resolvers and mutations) that has all the data we need to know what to send to whom.

Boundary interface

Thinking about what we need the server to provide, from the things we talked about:

• We must be able to request data using EQL from the server
• We must have a way to load the indexes from the server

On top of that, some requirements I want to add:

• We must be able to send initial data to the root entity <1>
• We must be able to send a request as AST form <2>
• The message format must be extensible

Another way to say this is that we need some contract for the boundary of the server communication.

Previously we mocked the calls to the server as (request-service-a [:x]). In this case the whole message is the EQL request. Notes on this format:

• Allow us to send EQL requests
• We can request the indexes via some attribute, like (request-service-a [::pci/indexes])
• It’s an interface already used by previous Pathom graph APIs, providing data for Fulcro applications, for example

To solve the other requirements, we can have a more explicit message version, using a map to wrap the EQL request, allowing for extra data to flow aside the EQL, like:

(request-service
  {:pathom/tx [:x]})

; example extending to send initial data, and AST instead of EQL
(request-service
  {:pathom/ast    {:type     :root
                   :children [{:type         :prop
                               :key          :x
                               :dispatch-key :x}]}
   :pathom/entity {:initial "data"}})

It’s also really important that this boundary interface is consistent across services.

To help make this consistent, Pathom 3 now provides the p.eql/boundary-interface helper, that wraps all the requirements we list before.

(def request-a (p.eql/boundary-interface env-a))

; standard mode
(request-a [:x])
; => {:x 10}

; sending a map
(request-a {:pathom/tx [:x]})
; => {:x 10}

; request via ast
(request-a {:pathom/ast {:type     :root
                         :children [{:type         :prop
                                     :key          :x
                                     :dispatch-key :x}]}})
; => {:x 10}

; request indexes
(request-a [::pci/indexes])
; => {::pci/indexes ...}

Then we can use this interface at the boundary at the system, as some HTTP handle for example:

(defn graph-handler [request]
  (let [req      (decode-request request)
        response (request-a req)]
    {:status 200
     :body   (encode-output response)}))

From now on, this is the preferred way to expose a Pathom environment for remote access. I recommend using it HTTP boundaries, WebSocket boundaries, RPC boundaries, etc…

Dynamic Integration

In the previous section, we upgraded the services to provide the boundary interface we need to make the integration possible. Now the client needs to figure how to handle this information.

The client needs to import the indexes from each service on its index and have a way to know this is from a foreign system, so it should delegate the sub-queries there.

Without getting in the weeds of how this happens, this how you do it:

; add :require [com.wsscode.pathom3.connect.foreign :as pcf]

(def env-c
  (pci/register
    [(pcf/foreign-register request-a)
     (pcf/foreign-register request-b)]))

In this example, request-a and request-b are functions that forward a request to some external service.

That’s it! This way, we don’t need to know anything specific about the servers other than they support the boundary API! Let’s test it out:

(p.eql/process env-c [:x :y])
; => {:x 10, :y 20}

If you like to play with this, you can simulate the whole thing in a single file as:

(ns com.wsscode.blog.blog-sources.pathom-updates-10
  (:require
    [com.wsscode.pathom3.connect.built-in.resolvers :as pbir]
    [com.wsscode.pathom3.connect.foreign :as pcf]
    [com.wsscode.pathom3.connect.indexes :as pci]
    [com.wsscode.pathom3.interface.eql :as p.eql]))

(def env-a
  (pci/register (pbir/constantly-resolver :x 10)))

(def env-b
  (pci/register (pbir/constantly-resolver :y 20)))

; boundary interfaces
(def request-a (p.eql/boundary-interface env-a))
(def request-b (p.eql/boundary-interface env-b))

(def env-c
  (pci/register
    [(pcf/foreign-register request-a)
     (pcf/foreign-register request-b)]))

(p.eql/process env-c [:x :y])
; => {:x 10, :y 20}

Note that env-c can also add its own resolvers, depending on the foreign attributes:

(def env-c
  (pci/register
    [(pcf/foreign-register request-a)
     (pcf/foreign-register request-b)
     (pco/resolver 'combine
       {::pco/input  [:x :y]
        ::pco/output [:z]}
       (fn [_ {:keys [x y]}]
         {:z (+ x y)}))]))

(p.eql/process env-c [:z])
; => {:z 30}

This is the beginning of the Pathom Maximal Graph integration idea. Internally, a lot is going on to make this possible. The idea is to support full integration so that you may have transient dependencies across services, and Pathom should figure which services to call, in which order.

The graph optimizations I talked about before are an important part of making this efficient. Having lesser nodes on dynamic resolvers mean reduced round trips.

Supporting this kind of integration is powerful, even if you work on a monolith. For example, a client-server web application can benefit from it. Having parts of the data requirements filled in the client (using Pathom on CLJS) and parts on the server.

I’m very excited about this, after all getting this integration working was the main reason for me to start Pathom 3.

This is the beginning, we need to play and experiment more, see where it works, where it fails and evolve.

Other updates

A few other updates:

• Fixed nested inputs + batch process
• Pathom 3 Smart Maps are fully working on Babashka
• New pco/wrap-resolve and pco/wrap-mutate helpers to extend operations
• Batch resolvers make inputs distinct (don’t process the same input multiple times)

That’s what I have for today, see you in the next one!

Follow closer

If you like to know in more details about my projects check my open Roam database where you can see development details almost daily.

Support my work

I'm currently an independent developer and I spent quite a lot of my personal time doing open-source work. If my work is valuable for you or your company, please consider supporting my work though Patreon, this way you can help me have more available time to keep doing this work. Thanks!

Current supporters

And here I like to give a thanks to my current supporters: