Exploration with Pathom Viz

A happy and growing index can get hard to tame, and that’s why the Index Explorer is here to help you.

The index explorer requires Pathom version 2.2.13+

The index explorer is a tool to help you navigate and understand the relationships between the attributes in your system.

Better show than tell, here is a demonstration of the index explorer:

Don’t worry if you got confused with all the information, in the next section we are going to drill down and explain each section of the explorer.

Explorer Menu

The menu is always visible in the left bar, you can use this to find attributes, resolvers and mutations. By default, it shows a complete index of things, you can click on the grey headers to collapse a group. Try it out in the demo above.

There is a search input on top of the menu, it will do a fuzzy search on everything.

Stats

The first screen you see in the index contains the main stats about the index.

In the counters section here is explanation for some non-obvious counters:

  • Globals count: the number of attributes accessible that doesn’t depend on any data

  • Idents count: number of attributes that by themselves can provide more data

  • Edges count: the number of edges connecting the attributes in the system

The most connected attributes section will give you a top list of attributes with most connections, a few attributes in a system tend to raise up on this list and can point to effective "hubs" in the center of your data.

Attribute View

When you navigate to an attribute you will be at the attribute view. This view can tell you details about a single attribute.

Graph View

Right after the title there is a Graph View button, this gives you a visual representation of the attribute and its connections.

This graph is dense on information points, to explain that let’s start with a simple graph with a single resolver that can read a user name from a user id:

; registry
[{::pc/sym    'user-by-id
  ::pc/input  [:user/id]
  ::pc/output [:user/name]}]

The following graph represents the attribute :user/id from this system:

Base elements

Let’s start with the graph itself, these are the basic elements:

Circles represent attributes, a yellow color points to the current attribute, which is :user/id in this case.

Lines represent resolvers (the effective edges), it means how the resolver inputs connect to the resolver outputs; note the arrow points from :user/id to :user/name, this means :user/id provides :user/name.

Available controls:

  • Click and drag on canvas - pan canvas

  • Mouse scroll - zoom

  • Click and drag circles - rearrange nodes

  • Mouse over circles - highlight attribute

  • Mouse over lines - highlight resolver

When you highlight some element, you can see a label for it in the top left corner. The edges get a highlight color as well, when highlighting an attribute, a green edge means it goes from current to the target, red edges are the reverse.

When you highlight an edge it will turn blue and every other occurrence of that same resolver will get highlighted as well.

Let’s add more attributes for a bigger view:

; registry
[{::pc/sym    'user-by-id
  ::pc/input  [:user/id]
  ::pc/output [:user/name
               :user/email
               :user/dob
               :twitter/url]}]

In this example, notice there is one circle with a different stroke color. The stroke color represents the namespace, this way you can see related namespaces by color.

The color pallet for namespaces contains 10 colors, so if you end up with a graph containing more than 10 namespaces they will start repeating colors.

Time to make it more fun, let’s add a second resolver to fetch user data from email:

; registry

[{::pc/sym    'user-by-id
  ::pc/input  #{:user/id}
  ::pc/output [:user/name
               :user/email
               :user/dob
               :twitter/url]}

 {::pc/sym    'user-by-email
  ::pc/input  #{:user/email}
  ::pc/output [:user/name
               :user/id
               :user/dob
               :twitter/url]}]

Nested connections

So far we have only seen direct connections, this means the values are the same "context space", the other option is nested connections, here is an example:

; new resolver
{::pc/sym    'user-groups
 ::pc/input  #{:user/id}
 ::pc/output [{:user/groups
               [:group/id :group/name]}]}

Note the attributes :group/id and :group/name are not visible in this graph, that’s because they are an indirect connection, use the Nested Outputs control to toggle nested outputs and they should show up. Note we represent nested connections using dashed lines.

When we have a chain of many connected direct connections, Pathom can walk any number of paths automatically but due to ambiguity that’s not true for nested connections.

Let’s see the same graph again, but this time the center will be :group/id:

Not much, right? Well, there is no direct connections to this attribute, please turn on the Nested Inputs, this will make visible the connection between :group/id and :user/id.

Now try increasing the Depth, this number indicates how many steps to walk from the center attribute, increasing the reach.

To finish up you can also enable Nested Outputs, this should end up similar to the one we had before with the center in :user/id (considering Nested Outputs is on).

Attribute Sizes

You may have have noticed that the circles don’t have the same size, that’s because its another point of information. Let’s get a clear example of that:

; registry
{::index
 [{::pc/sym    'user-by-id
   ::pc/input  #{:user/id}
   ::pc/output [:user/name
                :user/email
                :user/dob
                :twitter/url
                :youtube/url
                :linked-in/url
                :user/attr1
                :user/attr2
                :user/attr3
                :user/attr4
                :user/attr5]}

  {::pc/sym    'email-by-twitter
   ::pc/input  #{:twitter/url}
   ::pc/output [:user/email]}

  {::pc/sym    'email-by-youtube
   ::pc/input  #{:youtube/url}
   ::pc/output [:user/email]}

  {::pc/sym    'email-by-linkedin
   ::pc/input  #{:linked-in/url}
   ::pc/output [:user/email]}]}

The size of the attribute inner circle represents the number of attributes it provides, while the stroke size depends on how many attributes can be used to reach it. Notice the center attribute :user/id has the inner circle bigger than any other while :user/email has the biggest stroke size.

The sizes grows in a quadratic scale, so the difference can be hard to notice on small demos like this, but in real system it grows at a relevant rate.

Attribute Groups

So far, every attribute we saw was a one to one attribute connections but in Pathom we also have connections that depend on multiple inputs. In the graph we represent multiple attributes as grey circles, always with black borders. Here is an example:

; registry
[{::pc/sym    'user-by-id
  ::pc/input  #{:github.repository/name :github.repository/owner}
  ::pc/output [:github.repository/id
               :github.repository/url
               :github.repository/name-with-owner]}]

Notice when you mouse over the group, you can set the set described in the label section.

There is also a special group, the globals (or you can also call empty set: #{}). This attribute is always available and it connects to attributes with no dependency. Example:

; registry
[{::pc/sym    'time
  ::pc/output [:time/now]}
 {::pc/sym    'pi
  ::pc/output [:math/pi]}]

Reach Via

The Reach Via panel lists the direct and nested paths to reach current attribute in a single step.

You should look at this view as a tree. The first depth of the tree will always contains sets that represent the input you need to reach this attribute. If the set is bold, it means that input can directly reach the current attribute, otherwise it will have some nested list that will provide that necessary path.

You can click in any attribute to navigate into it.

Provides

The Provides panel lists all the direct and nested attributes that you can reach from the current in a single step.

This is a tree, imagine if you merged every resolver output that has the current attribute in the input.

As you mouse over the resolver that makes the link will show up below the attribute.

Output In

List of resolvers where this attribute appears as output.

Input In

List of resolvers where this attribute appears as input.

Input Combinations

In case this attribute appears as a input group with other attributes, all these groups will be listed here.

Mutation Param In

List the mutations that mention this attribute as params.

Mutation Output In

List the mutations that mention this attribute as output.

Spec

In case the attribute has a defined spec, you can see the spec form in this panel.

Examples

When the spec is available you can see some generated examples in this panel. You can generate new examples using the button in this panel header.

Resolver View

In the resolver view, the left column will give you details about the resolver input and output. Mouse over items to highlight it in the graph.

The right side will have the graph will all attributes that participate in this resolver, the center of the graph will be the resolver input.

Mutation View

The mutation view lists the mutation parameters and the mutation output.

Full Graph

If you click in the Full Graph button it will display a complete graph of the attributes connection in the system. Use this view to get a general feeling of the system, you can understand the main clusters and how they organize.

Setting up the index explorer resolver

To expose the index for the index explorer you need to write a resolver that gets your index out.

(pc/defresolver index-explorer [env _]
  {::pc/input  #{:com.wsscode.pathom.viz.index-explorer/id}
   ::pc/output [:com.wsscode.pathom.viz.index-explorer/index]}
  {:com.wsscode.pathom.viz.index-explorer/index
   (get env ::pc/indexes)})

Using this you can control what gets out to the explorer.

Visualizing your index

Here you will find some ways to visualize your index.

Fulcro Inspect

The simplest way is to use the explorer though Fulcro Inspect, this is of course limited to Fulcro Apps. All you need to do is open the Index Explorer tab and click to load the index, happy exploring!

Workspaces

Pathom Viz package includes some helpers to setup a card with an index explorer, you can use the following code as a starting point:

(ns pathom-index-explorer-workspaces-demo
  (:require [com.wsscode.pathom.core :as p]
            [com.wsscode.pathom.viz.workspaces :as p.viz.ws]
            [nubank.workspaces.core :as ws]))

(def parser ...) ; implement your parser, can be sync or async

(ws/defcard index-explorer
  (p.viz.ws/index-explorer-card
    {::p/parser parser}))

Stand alone app

Use the following example as a base to mount the index explorer app in any DOM node:

(ns pathom-index-explorer-stand-alone-mount
  (:require [com.wsscode.pathom.viz.index-explorer :as iex]
            [fulcro.client :as fulcro]
            [fulcro.client.data-fetch :as df]
            [fulcro.client.primitives :as fp]))

(fp/defsc Root
  [this {:keys [ui/root]}]
  {:query [{:ui/root (fp/get-query iex/IndexExplorer)}]}
  (iex/index-explorer root))

(def root (fp/factory Root))

(defn init []
  (let [app (fulcro/make-fulcro-client
              {:client-did-mount
               (fn [app]
                 (df/load app [::iex/id "singleton"] iex/IndexExplorer
                   {:target [:ui/root]}))})]
    (fulcro/mount app Root (js/document.getElementById "appContainerNode"))))

Fixing transit encoding issues

One common issue with the index explorer is the fact that resolvers include fns and may include other things that are not possible to encode with transit by default. We suggest you setup a default write handler on Transit so it doesn’t break when it encounter a value that it doesn’t know how to encode.

If you are running Pathom in Clojure, then you also need to know there is a bug in the current Clojure writer, it doesn’t support default handlers (although the docs say it does).

To fix this, here is a code snippet example on how to get around the bug:

(ns your-ns
  (:require [cognitect.transit :as transit])
  (:import [com.cognitect.transit WriteHandler TransitFactory]
           [java.io ByteArrayOutputStream OutputStream]
           [java.util.function Function]))

(deftype DefaultHandler []
  WriteHandler
  (tag [this v] "unknown")
  (rep [this v] (pr-str v)))

(defn writer
  "Creates a writer over the provided destination `out` using
   the specified format, one of: :msgpack, :json or :json-verbose.
   An optional opts map may be passed. Supported options are:
   :handlers - a map of types to WriteHandler instances, they are merged
   with the default-handlers and then with the default handlers
   provided by transit-java.
   :transform - a function of one argument that will transform values before
   they are written."
  ([out type] (writer out type {}))
  ([^OutputStream out type {:keys [handlers transform default-handler]}]
   (if (#{:json :json-verbose :msgpack} type)
     (let [handler-map (merge transit/default-write-handlers handlers)]
       (transit/->Writer
         (TransitFactory/writer (#'transit/transit-format type) out handler-map default-handler
           (when transform
             (reify Function
               (apply [_ x]
                 (transform x)))))))
     (throw (ex-info "Type must be :json, :json-verbose or :msgpack" {:type type})))))

(defn write-transit [x]
  (let [baos (ByteArrayOutputStream.)
        w    (writer baos :json {:handlers transit-write-handlers ; use your handlers here
                                 :default-handler (DefaultHandler.)})
        _    (transit/write w x)
        ret  (.toString baos)]
    (.reset baos)
    ret))

And this is how to do in Clojurescript:

(deftype DefaultHandler []
  Object
  (tag [this v] "unknown")
  (rep [this v] (pr-str v)))

(def write-handlers
  {"default" (DefaultHandler.)})

(defn write-transit [x]
  (let [writer (transit/writer {:handlers write-handlers})]
    (transit/write writer x)))

Extending the explorer #TODO