How Medal.tv increased SEO performance and collects key insights with WebRender

5 min readJan 24, 2021

Today we’re doing an engineering post about our partners at WebRender.io, showcasing how we’ve used their platform to make Medal’s React.js web app more successful for SEO purposes. We’ve been a customer for close to a year.

We implement WebRender.io by calling our deployment URL from our Cloudflare worker, effectively letting WebRender handle all our bot and crawler traffic.

WebRender is also deployed on CloudFlare. So requesting the cached version from the deployment URL is a near-instant call.

Now into some of the major wins we’ve had, and some of the drawbacks we faced.

Win #1: Capped page limit due to JS rendering using up the crawl budget & Performance

At some point last year, Google stopped indexing our new pages. As a company that relies on user-generated content, that’s a big problem.

We quickly realized that we were wasting a lot of Google’s crawl budget by waiting for the JS to render React components. After implementing WebRender, which is able to cache the pages on the edge, we quickly saw Google exhaust it’s crawl budget over a larger number of pages, and that number has continued to grow.

Fast forward to today, we have over 200,000 valid indexed pages, because WebRender is able to serve them from cache immediately. Therefore, Google uses less crawl budget per page.

Even if you use a solution like Next.js, you likely won’t get the same result because the server still has to render the pages. Whereas with WebRender, they come pre-cached on the edge.

Therefore — The biggest gain by far was in the performance of the render. Search engines no longer have to wait for JS to serve up the result. This is especially noticeable on mobile.

Before Webrender:

After Webrender:

Win #2: Engineering Time

Not having to worry if SEO breaks or changes significantly when you deploy updates is delightful, since we control the cache behavior in WebRender entirely. We also saved a bunch of time being able to skip setting up more complicated web infrastructure in favor of a simple react application that runs in the browser, paired with WebRender to serve crawlers and bots. All the integrations and settings are configurable via a simple json.

This allows us to focus on our users, and let WebRender handle bot and crawler-facing traffic.

Win #3: Integrations

Using the web hooks feature, we are able to send data directly into HoneyComb, which we use for data processing. This allows us to quickly export audit logs, or become smarter about what Google is hitting if their own analytics aren’t helping us answer our questions. This is our own internal HoneyComb database, which WebRender routes traffic logs into.

Win #4: Insights

Insights allow us to quickly see how different search engines are behaving, and logs allow us to see what is happening in real-time. This is insight that we didn’t have before, or was very expensive to get at our scale. This was especially useful when we were first implementing. Using the HoneyComb integration, we also log traffic into our own database for analysis.

Win #5: Real-time logs

The logs are extremely useful when implementing and debugging, and there’s nothing quite like seeing how crawlers are hitting the site in real-time

Drawback #1: Page volume

Since Medal has millions of pages, it can take a while to index all the pages. We requested additional render machines to speed up the process, which support was happy to help out with. We recommend reaching out and getting all your pages processed since it will start paying dividends in crawl budget the soonest.

Drawback #2: Uncached pages

If a page is not yet cached by WebRender, it can sometimes be slower to pre-render it than to let a js-friendly crawler render the react. This problem gets better over time as WebRender learns about the paths you serve.

Pro tip: If you set the unfurlDelay on a path to -1, but preRenderDelay at your pre render value, WebRender will not cause a delay as it will serve up the JS version of the page if a miss occurs.

Conclusion

Overall our experience has been a huge win, enabling us to perform at a top tier level for SEO. Some examples of our SEO achievements after implementing WebRender: