Deriving feed attributes a catalog never stored

Original reference implementation. Not client source code.

Why a feed template

A hardware catalog is full of facts that are not fields. Whether a fixing is stainless or zinc-plated, whether a driver is a power tool or a hand tool, which of the advertising channel's several thousand taxonomy paths it belongs under. All of it exists only inside titles and tags that somebody typed while adding a product.

Four places can turn that text into a channel's schema.

The platform's own sales channel is what you get by writing nothing, and most stores should stop there. It syncs products automatically and maps the platform's product taxonomy onto the channel's, per product, re-syncing on change. It fails a catalog like this one in a narrow way: the mapping is applied product by product through the admin, and there is no rule a merchant can write once and have hold over ten thousand fixings that are all titled alike. Getting the categories right means touching every product by hand, including the ones added next week.

A private app serving a proxy route is the textbook answer and is genuinely better in one respect: it can derive attributes once on a webhook and serve a precomputed document. It loses on ownership. It is a second deployable with its own host, token, uptime and webhook backlog, and when it stalls the feed goes stale while the storefront it mirrors stays perfectly healthy.

An offline job writing a file to object storage removes every render-budget problem below, and replaces it with a freshness problem: a price change is invisible until the next run, and the derivation rules end up somewhere the merchant cannot see.

The template wins because the feed is a rendering of data the storefront already has. It reads the same product objects the category page reads, at the moment of the fetch, so there is no second copy to drift. The rules live in a theme setting the merchant edits. And there is nothing else to deploy.

The costs of that choice are on the plate above, and the first of them is not a style issue. It decides the shape of the whole template.

{%- layout none -%}
{%- comment -%}
  A bare `for` over a product array fetches 50 items and stops. No error, no marker:
  a truncated catalog is indistinguishable from a small one. `paginate` is the only
  way past it, so the feed is paginated and the merchant registers one fetch URL
  per page. PAGE_SIZE is deliberately below the platform maximum of 250.
{%- endcomment -%}
{%- assign rules = settings.feed_rules | newline_to_br | split: '<br />' -%}
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:g="http://base.google.com/ns/1.0">
  <channel>
    <title>{{ shop.name | escape }}</title>
    <link>{{ shop.url }}</link>
    {%- paginate collections.all.products by 200 -%}
      {%- if paginate.current_page > paginate.pages -%}
        {%- comment -%} Valid, empty, and parseable. Never a 404. {%- endcomment -%}
      {%- else -%}
        {%- for product in collections.all.products -%}
          {%- render 'feed-item', product: product, rules: rules -%}
        {%- endfor -%}
      {%- endif -%}
    {%- endpaginate -%}
  </channel>
</rss>

That fifty-item default is documented on the for tag, and paginate accepts a page size between 1 and 250 and cannot reach past the 25,000th item at any page size. Both numbers are ceilings rather than preferences, and the second is the upper bound on how large a catalog this pattern can serve at all.

The page number is the only request-time input the feed has, and the template never reads it. Liquid's request object carries the host, origin, path, page type and locale, and no query string. paginate reads page from the URL on the template's behalf and hands back a pagination object. So a theme template is not a general endpoint that happens to emit XML. It can be parameterised by its path, by an alternate template suffix, and by pagination, and by nothing else. Every other input has to arrive as a setting a human edits.

The contract the merchant edits

Which is what the rule set is. It is a theme setting, ordered, and the order is the whole contract.

# version: 1
# channel category | merchant product type | material | match phrases
Hardware > Fasteners > Screws         | Screws > Wood    | Steel     | wood screw, woodscrew
Hardware > Fasteners > Screws         | Screws > Machine | Steel     | machine screw
Hardware > Fasteners > Washers        | Washers > Split  | Stainless | split washer, spring washer
Hardware > Fasteners > Brass Fittings | Fittings > Brass | Brass     | brass
Hardware > Tools > Power Tools        | Tools > Driver   |           | impact driver, drill driver

The version marker is there so the format can change later; an unknown version is treated as no rule set at all rather than parsed into nonsense. Invalid lines are skipped rather than fatal.

The first field must come from the channel's published taxonomy, and the template does not check that it does. It cannot: the taxonomy is several thousand paths, a template has nowhere to hold it and no budget to search it per product. What the template does enforce is that whatever the merchant wrote is emitted whole, because the channel accepts a numeric category ID or the complete path and rejects a partial one. The merchant-defined product type beside it has no such constraint, which is why the two are separate fields rather than one derived from the other.

Derivation itself is one pass over a string the template already has.

{%- liquid
  assign haystack = product.title | append: ' ' | append: product.type
  assign haystack = haystack | append: ' ' | append: product.vendor
  assign haystack = haystack | append: ' ' | append: product.tags | join: ' '
  assign haystack = haystack | downcase
 
  assign category = settings.feed_default_category
  assign product_type = settings.feed_default_type
  assign material = blank
 
  for rule in rules
    assign fields = rule | split: '|'
    if fields.size < 4
      continue
    endif
    assign matched = false
    assign phrases = fields[3] | split: ','
    for phrase in phrases
      assign needle = phrase | strip | downcase
      if needle != blank and haystack contains needle
        assign matched = true
        break
      endif
    endfor
    if matched
      assign category = fields[0] | strip
      assign product_type = fields[1] | strip
      assign material = fields[2] | strip
      break
    endif
  endfor
-%}

First match wins, and the rule set is ordered top to bottom. That buys a derivation with no per-product lookup and no second data source, and it gives the merchant a lever: any single wrong answer is fixed by moving a line.

Where it gives the wrong answer

Run that rule set against a real product.

title:   "Brass-Plated Steel Wood Screw #8 x 1-1/4 in"
type:    "Screws"
tags:    ["wood-screw", "exterior"]
variant: { sku: "WS-08-114-BP", barcode: "", price: 1299, compare_at_price: null }

The rule for brass fittings sits above the rule for wood screws, because the merchant added it first and brass fittings are a real part of the catalog. So brass matches, and the emitter writes this.

<item>
  <g:id>WS-08-114-BP</g:id>
  <title>Brass-Plated Steel Wood Screw #8 x 1-1/4 in</title>
  <g:price>12.99 USD</g:price>
  <g:availability>in stock</g:availability>
  <g:mpn>WS-08-114-BP</g:mpn>
  <g:google_product_category>Hardware &gt; Fasteners &gt; Brass Fittings</g:google_product_category>
  <g:product_type>Fittings &gt; Brass</g:product_type>
  <g:material>Brass</g:material>
</item>

The correct answer is a steel screw in the screws category. Brass-Plated describes the item's finish, not its substance, and the word the rule matched is a modifier on a noun the rule never looked at. The item is now advertised as a brass fitting made of brass, and a shopper filtering the channel for brass fittings is shown a steel screw.

Fixing it properly costs more than the error does. A material field per product is a second source of truth over every SKU, only as good as the last person who added a product in a hurry, and blank often enough that the derivation still has to exist. There would then be two answers that can disagree. A grammar that understands X-plated Y is a hand-rolled tokenizer, because Liquid has no regular expressions, running over every title on every fetch inside a budget that already caps the page at 200 products, and it would need X-coated, X-finish and X-effect added by hand afterwards.

The third option is the one the format is designed around. Move the screws rule above the brass rule and give it brass-plated steel as its first match phrase, and the same product derives steel and the screws category. That is one line, edited in the theme editor, by the person who noticed. First-match-over-an-ordered-list is what makes it possible at all. A scoring scheme or a longest-match rule would take the lever out of the merchant's hands and turn this into a defect only a developer could clear.

A related hazard stays a hazard. Matching is substring containment, so a rule phrase nut matches a walnut-handled mallet. Word boundaries without regular expressions are the same tokenizer problem, so the repository documents the behaviour and tells the merchant to prefer phrases long enough not to collide.

One more thing this replaces, which is easy not to notice: turning the feed on means turning the platform's own channel off. A store running both submits every product twice under two different categorisations. The README says so in its second section.

What the tests cover

The suite renders the template with a Liquid engine in Node against fixture objects shaped like the platform's product and variant drops, then parses the emitted document and asserts on it. No store, no access token, no theme preview. The engine is not the platform's Liquid, so platform-specific filters are registered as documented doubles and no test asserts on what a URL filter returns. The suite holds the template's decisions, not the platform's filter behaviour, and the README says which is which.

Twenty-two properties, of which the ones worth naming are the failure directions. A product with no variants contributes nothing rather than an empty item. A variant priced at or below zero is dropped while its siblings survive. A page past the last renders a valid document with no items, never a 404, because a 404 from a theme renders the store's HTML not-found page and a parse failure against a scheduled fetch can disable the schedule. Every skip decision is made before an item element is opened, so a response cut short can lose items but can never contain half of one.

The test that matters most asserts the wrong answer above.

test('a plated substrate derives the plating, not the substrate', async () => {
  const xml = await renderFeed({
    rules: RULES_WITH_BRASS_ABOVE_SCREWS,
    products: [platedWoodScrew],
  });
 
  const item = parse(xml).items[0];
 
  // Documented-wrong on purpose. The rule set is ordered and `brass` matches
  // first, so the finish wins over the substrate. See "Where it gives the
  // wrong answer" in the README before changing this.
  expect(item['g:google_product_category']).toBe('Hardware > Fasteners > Brass Fittings');
  expect(item['g:material']).toBe('Brass');
});
 
test('and the merchant can fix it by ordering the rules', async () => {
  const xml = await renderFeed({
    rules: RULES_WITH_SPECIFIC_PHRASE_FIRST,
    products: [platedWoodScrew],
  });
 
  const item = parse(xml).items[0];
 
  expect(item['g:google_product_category']).toBe('Hardware > Fasteners > Screws');
  expect(item['g:material']).toBe('Steel');
});

The pair is the point. The first test pins the boundary so that nobody silently "fixes" the simplification without reading why it is there; the second proves the boundary has a door in it. Both assert against the parsed document rather than a helper's return value, because what the channel receives is the only output that exists.

Reading the source

Open the template first. It is short, and the pagination block is the entire argument about what this layer can and cannot be asked to do.

Then read the rule format's documentation before the snippet that consumes it. The snippet is twenty lines of string handling and makes no sense until the ordering contract is clear.

The emitter is the least interesting file and can be skimmed, with one exception worth reading closely: every decision to skip an item happens before the item element is opened, and that ordering is load-bearing rather than tidy.