An alternative “shortcode” (mdxType alternative) proposal

[wooorm]

Currently, MDX is walking the whole JS program, and adding an mdxType prop to all JSX elements. It does this, so components can be overwritten at runtime.

Take this MDX:

* a
* b

Which turns into (simplified to get the point across, the actual is more complex):

var ol = () => throw "error"
var li = () => throw "error"

<ol mdxType="ol">
  <li mdxType="li" parentName="ol">a</li>
  <li mdxType="li" parentName="ol">b</li>
</ol>

After compiling the JSX away, you get:

var ol = () => throw "error"
var li = () => throw "error"

func(ol {mdxType: "ol"}, [
  func(li, {mdxType: "li", parentName: "ol"}, ["a"]),
  func(li, {mdxType: "li", parentName: "ol"}, ["b"]),
])

Then, the runtime, supports replacing things through components.

It has a func that ignores its first argument, instead taking mdxType (and parentName) to look whether it has a different component defined in components, and uses that instead of the either imported or defined throwing components.

The above is the simplified example for “normal” tags. The same happens for actual components (Capitalized):

var MyComponent = () => throw "error"

func(MyComponent, {mdxType: "MyComponent"}, [])

Which is then overwritten at runtime...

Even weirder is:

import MyComponent from "place"

func(MyComponent, {mdxType: "MyComponent"}, [])

// Passed in runtime:
{components: {"MyComponent": () => /* this is used instead of the imported component! */}}

This ✨ magic ✨ is complex and undocumented. I want to remove that complexity. Here’s one alternative.

We instead compile all standard markdown (html?) things to a namespace object: p becomes mdx.p, strong becomes mdx.strong, etc:

<mdx.ol>
  <mdx.li>a</mdx.li>
  <mdx.li>a</mdx.li>
</mdx.ol>

Then, in mdxContent (the wrapper function around all JSX components), we’re being passed runtime components. We take those and use them to fill mdx.p = components.p || 'p'.

This might solve #821, because the JSX <p>x</p> is not overwritten whereas the markdown paragraph x is, depending on where we implement the changing of p to mdx.p.

This then has no knowledge of JSX: there is no tree traversed for whether <MyComponent /> is imported or not. You have to import it, you can’t just pass it Youtube to components, you have to import it.

/cc @johno, @ChristianMurphy

[ChristianMurphy]

Overall I prefer proposed approach, the idea of mdx.* components to allow for shortcodes.
It fits cleanly in a component based model of viewing the document, and the JSX translates directly to the hyperscript calls.
Intercepting the hyperscript calls is a powerful approach, but feels more unexpected, I usually wouldn't think to look for hyperscript calls being intercepted, and the wrapping could potentially be hidden by source mapping.

Some considerations:

1. this would add another global identifier to the page mdx, some consideration of how that would interact with import mdx from 'some-package' would need to be done/documented.
2. it would potentially remove the current ability to override components based off their parent with parent.child syntax. Personally I have seen that syntax as also being unexpected, something that may be a better fit for a plugin, and would be fine with removing it. But if removed, it would be a breaking change.

[johno]

Personally I think that presentational context is a pretty powerful feature that allows MDX to be more flexible amongst rendering environments. MDX, as a format, isn't necessarily guaranteed to be coupled to code (consider a CMS). It also allows for more ergonomic composition when you want to override rendering at the sub-document level.

Forcing imports for additional components like Youtube sort of breaks the best part of shortcodes IMO because a parent or template can no longer expose commonly used globals. Part of the motivation behind the API is removing other forms of bespoke shortcode syntax and overrides/overloading of existing syntax.

This custom pragma functionality undoubtedly adds complexity and bit of magic/surprise (and some runtime weight), but I think it's worth it. Especially if we can eventually compile away the runtime when it isn't needed.

Would love for more folks to weigh in here, though!

[ChristopherBiscardi]

You have to import it, you can’t just pass it Youtube to components, you have to import it.

That alone makes this a non-starter for me. full stop.

• If we force imports then we tie MDX content written in a CMS to the bundling and filesystem details of all of the sites that use that content.
• This is fine for someone who is writing the code for their own personal site, but not even close to ok for content authors who are used to shortcodes in other ecosystems (wordpress, hugo, etc) or large systems of MDX content.
• Even for the person writing for their own site, the import paths now force them to use similar non-mdx technical implementations in each of site that accesses the content.

This change will result in the following user behavior:

User will tire of remembering how their gatsby/next/toast/etc site handles file paths and seek to use a webpack alias or similar to paper over this. Effectively changing import YouTube from '../../components/youtube.js'; to import YouTube from 'shortcodes/youtube' or import { YouTube } from 'shortcodes'.

on a technical level, the last example is problematic because we're forcing people into tree-shaking which is widely misunderstood and poorly implemented by users.

the ../../components/youtube.js example is technically problematic because it relies on implementation details of the library used to build mdx (such as gatsby-plugin-mdx, or next-mdx-remote) and how/where imports are sourced from (which changes per-library and across library versions).

would potentially remove the current ability to override components based off their parent with parent.child syntax.

How would you replace the paragraphs rendered inside a blockquote after this change? (without changing global paragraphs or unintentionally affecting other paragraph elements).

---

In general I disagree with the approach of introducing end-user complexity for the benefit of a simpler implementation. I'm open to other opinions but this seems like a change for the benefit of maintainers at the cost of users.

[wooorm]

Thank you all for your feedback!

You have to import it, you can’t just pass it Youtube to components, you have to import it.

That alone makes this a non-starter for me. full stop.

Scratch the you. Replacing it with “someone [has to]” (at build time). And then I don’t think we’re far apart.

There are two completely different things going on here:

1. Overwriting all “markdown” "h1"s to render component.h1 instead, which can be done by a programmer or some CMS system (which passes certain components without a user knowing)
2. Editors being able to use JSX/markdown with some certain components magically being injected, like YouTube, which is done through components by a CMS, without the user knowing

For the first: that can be done nicely with (continued thought on) this proposal. In fact, this proposal would do away with the (p)react runtimes. So this use case would become smaller, less magical, and would include less bugs.

For the second, If some CMS wants to inject YouTube components, it can: we have a JavaScript AST (#1382). And wrapExport. And docs on how to do that in MDX@1.

And we have members: <Wordpress.Youtube />, <Wordpress.Vimeo /> can all be used with a single (user or injected) import.

the ../../components/youtube.js example is technically problematic because it relies on implementation details of the library used to build mdx (such as gatsby-plugin-mdx, or next-mdx-remote) and how/where imports are sourced from (which changes per-library and across library versions).

Expecting a Youtube component to be defined magically is just as problematic: it relies on the CMS/library/tool. Hence why I’d rather explicitly have folks import their dependencies: import Wordpress from "wordpress", where the CMS or whatnot could change that to whatever they want. Or have tools inject whatever they support.

How would you replace the paragraphs rendered inside a blockquote after this change? (without changing global paragraphs or unintentionally affecting other paragraph elements).

I propose compiling standard “markdown” things to something else:

Say you have this MDX file:

import MyComponet from "place"

<MyComponent id="123" />

Some *emphasis* and **strong**

This currently compiles to (roughly):

import MyComponet from "place"
function mdxContent(components) {
  return (
    <>
      <MyComponent id="123" />
      <p mdxType="p">Some <em mdxType="em" parentType="p">emphasis</em> and <strong mdxType="strong" parentType="p">strong</strong></p>
    </>
  )
}

And then there is a 5kb runtime specific to Vue, React, or Preact.

I think it’d be nice to instead compile to:

import MyComponet from "place"
function mdxContent(components) {
  var mdx = Object.assign({"p": "p", "em": "em", "strong": "strong"}, components)
  return (
    <>
      <MyComponent id="123" />
      <mdx.p>Some <mdx.em>emphasis</mdx.em> and <mdx.strong>strong</mdx.strong></mdx.p>
    </>
  )
}

This would remove all createElement stuff from the runtimes.

You could compile certain things to <mdx.blockquoteP>, being set as blockquoteP = components.blockquoteP || components.p || "p" s, and support a couple other parent child combos.

---

[ChristianMurphy]

I think it’d be nice to instead compile to:

import MyComponet from "place"
function mdxContent(components) {
  var mdx = Object.assign({"p": "p", "em": "em", "strong": "strong"}, components)
  return (
    <>
      <MyComponent id="123" />
      <mdx.p>Some <mdx.em>emphasis</mdx.em> and <mdx.strong>strong</mdx.strong></mdx.p>
    </>
  )
}

with the notes that other props should still be able to be passed and context would/should also be a part of that assignment.
This pattern seems more inline with react conventions than wrapping createElement/hyperscript.

[wooorm]

Yes, totally, abbreviated that just to be shorter, but it’s currently function MDXContent({components, ...props}) and I don’t intend changing that!

[ChristopherBiscardi]

There are two completely different things going on here:

1. Overwriting all “markdown” "h1"s to render component.h1 instead, which can be done by a programmer or some CMS system (which passes certain components without a user knowing)
2. Editors being able to use JSX/markdown with some certain components magically being injected, like YouTube, which is done through components by a CMS, without the user knowing

<h1/> is not a different user experience than <YouTube id=""/>. They were intentionally combined in implementation and user experience. This creates a single set of components (which is easier to understand and explain than namespacing or specially handling one group).

For the first: that can be done nicely with (continued thought on) this proposal. In fact, this proposal would do away with the (p)react runtimes. So this use case would become smaller, less magical, and would include less bugs.

You're using "magical" as a synonym for your personal architectural preference here and if it has fewer bugs it's because we're removing the code supporting functionality users have in production today.

We need to seriously consider how breaking this functionality would be (which doesn't look like it's been considered yet) and also consider how the hard-coding of elements at compile time would bring us closer to the pre-mdx implementations that existed and were generally insufficient for the purpose.

For the second, If some CMS wants to inject YouTube components, it can: we have a JavaScript AST (#1382). And wrapExport. And docs on how to do that in MDX@1.

This would lead to size bloat for MDX content that use common components. In the above case "a CMS system that passes components without a user knowing", that would require either deep CMS integration with the target system or including every usable component in the mdx ast.

Using the AST is purposefully reserved for tooling authors, not end-users. AST maniupulation is error-prone and a fundamentally worse UX compared to using components to achieve the same things. This kind of static analysis forces either us or the ecosystem to maintain code that we get for free from the respective frameworks we integrate with: which we can and should trust.

the ../../components/youtube.js example is technically problematic because it relies on implementation details of the library used to build mdx (such as gatsby-plugin-mdx, or next-mdx-remote) and how/where imports are sourced from (which changes per-library and across library versions).

Expecting a Youtube component to be defined magically is just as problematic: it relies on the CMS/library/tool. Hence why I’d rather explicitly have folks import their dependencies: import Wordpress from "wordpress", where the CMS or whatnot could change that to whatever they want. Or have tools inject whatever they support.

Again ignore your hand-waving "magic" comment, relying on tooling to supply a value is not the same as relying on the implementation details of those tools to provide a value. With the pragma/context the user has control over where and when a value is provided and MDX doesn't have an opinion on that. With the latter, the tooling developed by different teams of people that don't communicate (next, gatsby, etc) must work the same way across platforms which isn't going to happen and that differential burden falls on users.

import MyComponet from "place"
function mdxContent(components) {
  var mdx = Object.assign({"p": "p", "em": "em", "strong": "strong"}, components)
  return (
    <>
      <MyComponent id="123" />
      <mdx.p>Some <mdx.em>emphasis</mdx.em> and <mdx.strong>strong</mdx.strong></mdx.p>
    </>
  )
}

FYI this import gets stripped in the gatsby-plugin-mdx processing and included at the root of the application via context.

Passing differently rendered video players at different rendering positions in the application where MDX is rendered is also a valid usecase that wouldn't work with this proposal.

This approach locks a user into a single component everywhere that MDX document is used, reducing portability. What happens, for example, if a pre-compiled MDX component with this approach and AST modifications is transcluded with another MDX document? Different renderings should be effected via the declarative component-based approach, not an AOT AST.

This would remove all createElement stuff from the runtimes.

I don't see this as a useful goal unless it's enabling a better user experience and given the feature removal in this proposal, I don't think it is.

The createElement is currently opaque to the end-user (they don't have to worry about it at all) and most don't even know it exists. End-users leverage the flexibility the pragma gives them in their framework of choice and the code that implements it isn't complex or magic: it boils down to a single if statement. I'll also note that the code for the react pragma hasn't had to receive any major functional changes for over a year, it has been exceptionally stable.

You could compile certain things to <mdx.blockquoteP>, being set as blockquoteP = components.blockquoteP || components.p || "p" s, and support a couple other parent child combos.

Wouldn't this require an AST plugin?

I'll repeat that requiring end-users to interact with the remark/rehype/estree ASTs to achieve the functionality they need is both a step backwards and overly complicated when compared to the equivalent component based implementation. ASTs are for tooling authors, not end-users.

[wooorm]

@ChristopherBiscardi Thanks for the detailed write up! ✨ I’ve been thinking and working on this the last week which I was about to push. I’ll push that now so you can take a look at it if you want, but know that it was written right before your post and that I’ll take your feedback int account later!

[wooorm]

I think all the above feedback is handled in #1425. No ASTs needed, MDXProvider is used, and there are benefits.

[ChristianMurphy]

Personally I think that presentational context is a pretty powerful feature that allows MDX to be more flexible amongst rendering environments. MDX, as a format, isn't necessarily guaranteed to be coupled to code (consider a CMS). It also allows for more ergonomic composition when you want to override rendering at the sub-document level.

Agreed, my understanding is that setting components both via the props and react.Context presentation contexts would be preserved.

Expecting a Youtube component to be defined magically is just as problematic: it relies on the CMS/library/tool. Hence why I’d rather explicitly have folks import their dependencies: import Wordpress from "wordpress"

@wooorm my understanding is that this still allows YouTube to be passed via context and via props, but that the error message if it is missing would be different? Is that correct?

You're using "magical" as a synonym for your personal architectural preference here and if it has fewer bugs it's because we're removing the code supporting functionality users have in production today.

We need to seriously consider how breaking this functionality would be (which doesn't look like it's been considered yet) and also consider how the hard-coding of elements at compile time would bring us closer to the pre-mdx implementations that existed and were generally insufficient for the purpose.

@ChristopherBiscardi your insights are valued, could you give a more concrete example of what you see as breaking?
In it's current form your comment comes across more as FUD than as a technical objection.
My current understanding is that short codes, a feature you kindly brought to the ecosystem, would be preserved, and that this is a change to the internals to improve performance.

 import MyComponet from "place"
 function mdxContent(components) {
   var mdx = Object.assign({"p": "p", "em": "em", "strong": "strong"}, components)
   return (
     <>
       <MyComponent id="123" />
       <mdx.p>Some <mdx.em>emphasis</mdx.em> and <mdx.strong>strong</mdx.strong></mdx.p>
     </>
   )
 }

FYI this import gets stripped in the gatsby-plugin-mdx processing and included at the root of the application via context.

Passing differently rendered video players at different rendering positions in the application where MDX is rendered is also a valid usecase that wouldn't work with this proposal.

Am I interpreting correct, that if MyComponent was transformed to mdx.MyComponent, so that it could be instantiated by props and Context that this comment would be addressed?

[wooorm]

@wooorm my understanding is that this still allows YouTube to be passed via context and via props, but that the error message if it is missing would be different? Is that correct?

All of that remains the way it is now in #1425, including the runtime warning

Am I interpreting correct, that if MyComponent was transformed to mdx.MyComponent, so that it could be instantiated by props and Context that this comment would be addressed?

Almost. Yes, MyComponent still works in #1425, but I instead add it to the scope:

const {YouTube} = Object.assign({}, componentsFromProvider, componentsGivenToMdxContent)

<Youtube />

The reason is that well, it’s a used identifier, so it’s more apt in my opinion to define it in the scope. Whereas for non-identifiers (such as p), it doesn’t make sense to define them in the scope, so there I make them members:

const __components = Object.assign({p: 'p'}, componentsFromProvider, componentsGivenToMdxContent)

<__components.p />

[wooorm]

landed