Presented by David Deppner
Magento Imagine, Las Vegas, NV, May 14, 2019
Fitting Round Pegs Into Square Holes:
Exporting Magento 2 Products to Google Merchant Center
There are many key differences between the Magento product architecture and the Google Shopping Feed Specification. This can create challenges for all Magento merchants. In this presentation, we take a deep dive into the requirements for listing products in Google Shopping Ads, as they relate to key Magento architectural concepts. We will examine the differences between Google’s product and landing page requirements and the core product types in Magento Commerce 2, and explore how to resolve them. Get practical tips on getting your products listed in Google’s systems.
Good afternoon! Like a lot of you, I’ve lived my life at the intersection of technology and business, and these interesting challenges we face with these sorts of business integrations and the tech platforms, how we work it all out and find a way to kind of soldier through the inevitable issues that come up, this stuff really fascinates me because this really is the key to achieving profitable growth in our businesses and for the merchants that we’re helping as we’re on the agency side.
So our business, Psyberware, we work with just all aspects of ad management for Magento merchants and that’s everything from obviously ads and bidding but also dealing with technical integrations with Google Shopping feeds, things like Google Tag Manager implementations. We do a lot of analytics and financial work as well.
So today’s topic is huge. I can barely scratch the surface of everything that goes on between Magento and Google Shopping. So I’m really going to carve out a certain slice of the pie here and focus in on how the Google policy framework interacts with various Magento product types, in order to look at what are the issues with configurable products, what are the issues with group products and try to be a little bit practical, boots on the ground, talk through some of these fundamental issues, because if you can be solid in these fundamental issues, that’s what everything else is built upon.
Right off the bat, I think we need to just have a few words about what is a product. And Magento follows an EAD database structure. Not everybody knows what that is, and it’s pretty complicated in a sense, but it’s also remarkably simple. It allows us to be really flexible about the attributes we define on products, we collect a number of attributes together into some product entity, and it’s a fairly straightforward database structure. You know, we’ve got things like here’s the SKU, here’s the name of the product, other fields like that.
That’s basically what a Magento simple product is. We don’t need to really talk a lot about that because I think everybody pretty much understands that metaphor. But when we get into Magento’s other product types, they’re frequently not just data. They’re not just a data architecture, they’re a user interface. So for example, here we’ve got a configurable product. Configurable products are some that people have the most struggles with with Google Shopping. A configurable product isn’t really a product at all, it’s like some way of containing some other products or selecting between other products. It’s a user interface that allows us to steer towards the simple product we’re really trying to get to. And all the other Magento product types have similar sort of metaphors to them, a similar user interface. A grouped product – well, that’s a group of products, really. Bundled products is several products bundled together and sold collectively.
Google really has one product type and it’s really similar to what the simple product metaphor is in Magento. It’s just an entity with a collection of attributes. But it’s much more rigid. It’s more defined. There’s less flexibility to define whatever you want. It’s basically not customizable at all. So you need to be able to take these attributes from a Magento product and somehow work it into the system, and that’s going to be doing things like, well you’ve got to transform the price – we’ve got to put a currency code on the end of it. Maybe we’re going to do some things for marketing and SEO, like change the name of a brand from a back-end code we’re using on our Magento system to a brand name that is more keyword-rich in the Google Shopping feed. But it’s a pretty straightforward process of just extract the data, map some fields, transform those fields, load that into Google Shopping one way or another, and that’s easy, right? Well, there’s issues that come up, but that basic metaphor is easy.
Where it gets complicated is all the policies and how they interact with everything else we’re doing with other Magento product types.
Just about everybody’s familiar with the product data specification. This is where we go to try to understand what are the different fields, what are the different attributes we can set on the Google Shopping site. So here’s an example of one field, the link field where we’re going to set what is the URL for the landing page that this ad is going to go to. And one thing I want to make a point out of right now is that this is not just a data specification. This also specifies all sorts of various policies. There’s all sorts of little footnotes and “click on here for more information about this”, right? There’s a lot of additional information beyond just data spec. A lot of times, developers read through this and just look at the data side of this and sometimes the policy prescriptions are a little bit – um -- dropped. One of the things that I think is important to note is there’s always a couple key sections – we’ve got the minimum requirements for every field and those are critically important. A requirement is a requirement. You must do it in order to have your products approved. But there’s also a lot of best practices listed down below. The thing about best practices is they’re not required at all. They’re things that you can do to improve the interface on Google’s systems, to improve the findability of your products, but you don’t need to do them. And so when you’re struggling with some of the issues in Google systems and trying to get these integrations connected, you can safely ignore the best practices initially and then come back to them later, once you have everything working, and improve things from there.
There’s also a lot of sources of policy and there’s no index to all of this. So Google has things scattered all over the place. There’s different documents on how to set up different aspects of their system. We’ve got shopping ad policies layered on top of generic advertising policies and we have landing page policies that are specific to shopping ads. All of these things interact in interesting ways and sometimes there’s contradiction. And another thing to note is that sometimes there are differences between what’s enforced and what’s stated in the policy, and that works both ways. Sometimes we have policies that say you must do this and it’s actually not enforced so people get away with it for a long time and then suddenly all their products get disapproved. But then on the flip side, some of the policing of these systems is a little bit too strict and we frequently get things that aren’t in the policy at all that get disapproved. So there is sort of a pragmatic aspect to working with these systems, where sometimes we just have to take a guess-and-check approach on some things that are in the gray area that we don’t know and see if they work.
So let’s get into Magento product types. I said configurable products are the ones that people usually have the most problems with. And I think configurable products also illustrate the problems that are going to come up for everybody else, for the other product types. A configurable product – like I said, it’s a user interface, it’s a front-end that allows people to select a particular simple product that they’re going to ultimately buy. And so it seems like a pretty easy thing, we have a few attributes here that can vary and we’ve got some selections to make and then we can add something to the cart. There’s some key differences in terminology. You’re not going to find any documentation on Google’s site about configuring configurable products. Google calls these “products with variants.” And an individual simple product they call a variant or an individual variant, an individual variation of a product. So as I’m talking through all this, I’m going to go back and forth between this terminology because you can’t keep them straight, I just want you to know what I’m talking about. Configurable equals product with variance. Child/Simple product is a variant. Google’s policy on variants is not in any policy document that talks about variation or configurable products. It’s actually in the product data specification, under the documentation for a single attribute, the item_group_id. Note that Google says this attribute is required in a set of countries but not in some others, so there’s already some interesting variation going on here in how to apply this, depending on what countries you’re selling into and advertising in. So it’s optional for other countries. They say that it is required when your products vary by one or more of six other attributes: color, size, gender, pattern, material, and age group. But if your products vary by other things, for example if you’re selling different brake shoes for cars and they vary by model of car, well then it’s not required, so we’ve got some exceptions here.
Configurable products on the Google side are not supposed to be added to the feed. You’re supposed to add the individual simple products or the individual variations. The parent configurable product is not allowed in the feed. A lot of people add those products to the feed and cause some problems. When you’re adding the individual simple products in Magento, the child products, you need to set the item_group_id in most cases and it’s going to be set typically to the parent SKU. So in this particular case, we’ve got this parent SKU of MT04 and we’ve got a number of child SKUs. We’ve added all those child SKUs to the feed, we tie them all together with an item_group_id of MT04. It’s a common identifier between all the child products. So this is what it would look like. We’ve got all our child products, we’ve got them tied together by the item_group_id, we specify the size so Google knows how they vary and that’s really going to help with findability in the Google user interfaces for selecting different sizes of products. Right now we’ve only got blue in this particular SKU but maybe we’d have some other colors as well or other attributes that these could vary on.
Another key area where people go wrong is in their use of configurable products as landing pages rather than the individual simple products, and there’s some complexity here. Google’s landing page policies are something you need to spend some time with. A minimum requirement that is easy to overlook is that every product landing page must clearly display the price of that product and that price has to be the most prominent price on the page if you have multiple prices, for example you’ve got other recommended products and so on. This causes a lot of issues with products that have variation in price, where it says it’s a price that ranges from this to that until you make selections. So there’s all sorts of ways this can get you. The other thing about best practices here is they say the best practice is to pre-select the correct variant on that configurable product landing page. So let’s look at how we can do that.
Well, first of all, simple product pages -- if the individual simple products are visible on your site, you can side-step the complexity here by just landing on those. But a lot of sites don’t want to list just the individual simple product because, say for SEO purposes they want to funnel all the traffic to the particular product the configurable product that’s going to have the user interface to select the sizes. But if the simple products are visible, this is a good way to go. And there are ways to deal with the SEO issues of having these in there, if that’s a concern for you.
An option that can be used to land on a configurable page and pre-select the correct variant is a poorly documented feature of Magento, that is to use URL suffix as #142=168&93=50. That’s a particular example for this product. 142 is the code for that particular attribute and the small size in the drop-down box has a numeric code as well, and that’s the 168. We’re passing these numbers that allows this to be pre-configured. It’s going to be a little harder for, say, marketing people that don’t have access to the underlying data to figure out what these numbers need to be but there actually are some quick tips you can use if you just want to view source on the page, there’s some common places these are always at and it’s very easy to get to. You can also in code easily get these options – well, maybe it’ll take a little bit of programming work – but you can get these codes in order to dynamically build out URLs for all the products you have in stock based on how things are currently configured.
I did a little bait-and-switch on you here – I changed the blue color swatch to a drop-down and that’s because there’s a little java script bug. This is a poorly used, not very often used feature, so I think it got neglected for a little while, but that bug got fixed for using this method to select swatch attributes but it merged into the 2.3.2 code base. Most people aren’t there yet. We’re on 2.3.1 right now.
So if we’re using this URL suffix method on the configurable parent page as the landing pages for all these products, we can dynamically build some links, include those in the shopping feed and it works just fine. And there are a number of extensions that make this easier, there’s a number of extensions that make selecting the variations for configurable products better, to get your keywords in the URL and if you’re using one of those it’s going to be a much easier process.
Simple products with custom options are much more complicated. They basically function as a user interface to the end user just like a configurable product, but these are actually simple products and they have a slightly different configuration on the back-end. So, it’s a simple product where you go down to the bottom, you specify some attributes that can vary, you might specify different prices for those, which should immediately set some alarm bells off that that would change the pricing on the landing page and cause some problems. But ultimately we’re creating some virtual SKUs, so if you select a “small” it’s going to append an "S" onto the end of the parent SKU, so technically it’s possible to get these guys kinda into the shopping feed if you don’t vary the prices. If the prices of all options are the same, so that the landing page meets Google’s price requirements and if you actually build out some basically fake virtual products in your shopping feed for every one of those SKU variations, you can make it work. It just takes a little bit of added work. The other thing you can do, obviously, is just recreate these as individual simple products and tie them together with a parent configurable and that will work as well.
Grouped products are tough. Google has a policy that you cannot land a shopping ad on a category page. If you just look at this page, we’ve got several different products, several different prices. A grouped product is not really a product. It’s almost like a kinda terminal node of a category structure that allows you to pick between several different products. So there’s an immediate problem there that makes these basically impossible to land on as a landing page and they’re not really a product, they’re a collection of different products. You could list those individual simple products separately in your feed but there are some situations where you might want to use one of these and actually land on it. And the thing I’ll say is that, remember how configurable Magento is, you can change anything, so let’s say you’re in a business that sells a product that takes a battery and this is some expensive, high-end product with an expensive battery and you want people to remember they have to buy the battery as well. So you list two products on your grouped product page, you got your main equipment and you got the battery. If you change the page a little bit to make the main product the more prominent product and to put the price of that very prominent as well, tweak your schema.org markup on the page to correctly have the information about the condition and availability and so on, you can make this work. And then people are landing on the right page and they get that little memory jog to remember to buy the batteries as well. But basically, unless you’re going to make changes like that, you really can’t include these in the feed, you really can’t use these as a landing page. Use simple products instead. But in these cases, those products that are contained on the group page are already simple products somewhere else, so as long as they’re individually visible, you can go that route.
Bundle products have a lot of problems – but not always. So here’s a good example of what you can do with Magento – a lot of configuration, a lot of complexity, ultimately this is going to select between a bunch of different products that can be conditionally added and configured in different ways and they’re all going to get added to your cart and you can check out. But from Google’s perspective, this violates the pricing policy. There’s no single price displayed on the landing page, all the configuration options cause the same issues that we have with configurable products, and there’s really no mechanism for pre-selecting any variants on this. The other thing, though, is if you actually go into the product data specification, there is this documentation about this “is_bundle” attribute. In fact, you can include a bundle of products in Google Shopping. What they say is that if they’re sold together as one package for one price, you can do it. So as long as you create a bundle that just has the same things in it and not all the configuration, it’s all for one price and it’s prominently displayed on the landing page, not a problem at all. You just have to set that is_bundle attribute, it’s just a Boolean yes-no, true-false, and also there’s some particular image requirements on these products as well. You have to show all of the elements in your bundle together in that image, and there’s a couple guidelines about how to do that. Bottom line here – if it’s a dynamic bundle, don’t even attempt it. If it’s a static – well, in that case, just list the individual simple products, just don’t sell them collectively as a bundle through Google Shopping. But if it’s a simple bundle you’ve got a few things that just always come together, for example in the previous example, you’ve got the equipment and you actually sell it with the battery, you can include those together as a bundle in your Google Shopping feed.
Virtual products. These are pretty problematic, but they can work. It depends on how you configure them. Magento’s documentation says these are non-tangible items, such as memberships, services, warranties or subscriptions and digital downloads. The thing is, some of those things can be sold in Google Shopping and some of them can’t. Virtual products also have another problem – they’re kinda like a simple product in the database structure but they don’t have a shipping weight. So that can cause some problems, but other elements here are okay – we’ve got the price on the landing page and so on. Google’s policy documents say that recurring billing products are unsupported. There’s some exceptions to that with certain software subscriptions, so that’s something to note. But in general, recurring billing type products aren’t allowed and they say about services that, in general, labor, time, effort, expertise or actions which don’t involve ownership of a tangible product cannot be sold through Google Shopping. The difficulty here is they don’t define “tangible.” So, for example, they just said in other places you can sell software in certain conditions and certain cases, even software subscriptions. Now software would be considered an intangible in some scenarios, so I’m actually not always sure what Google actually means by “tangible” or “intangible.” I’ve seen some examples on either side of this that have worked or failed. But I think the take-home here is that you’re just going to have to do the research, you’re going to do the best you can, you’re going to find if other people have advertised similar things that you’re trying to advertise, and understand there’s a little bit of ambiguity going on and you might have a situation where you just need to do a test and see if it gets disapproved. If it gets disapproved, they’re going to give you a specific error message and that’s going to help point you to where you need to go to figure out what you can and can’t do. At least you have something also you can talk to support folks over at Google about at that point about the specific disapproval. Don’t forget – that zero dollar shipping needs to be set on products like this, if there is really zero dollar shipping. Everything in Google shopping has to have a shipping rate. So if it’s zero, explicitly state that it’s zero.
Downloadable products in Magento can be configured a few different ways, and some of these cause problems with landing pages and some don’t. This is primarily a landing page issue. You can see, this one looks pretty straightforward. We’ve got a single price, we can add that to our cart, we can check out – we can put this product in our feed. Let’s check out this guy – here’s another one that has several episodes. The episodes can be added or not, and then the price is going to dynamically update based on that. But when Google comes over and tries to check this as a landing page, they’re not going to be able to see a price. So that causes some issues. Think about it like this – make sure that you set the price at the product level. Don’t set prices on individual download links, and if you need to have different prices on individual download links, then think about recreating these products as separate downloadable products and then you can list them that way. And also, don’t forget that shipping issue. This is another one where you could have these products disapproved because you don’t specify how much it costs to ship them.
Gift card products are another feature of Magento commerce but it’s not in Open Source. These again have a lot of flexibility in how they can be configured, where you can have a drop-down with multiple pre-selected gift card amounts or you can have a text input box where you can punch in like I want a $36 gift card, even though it’s a weird amount. This is basically just an issue with landing page policy. These can be sold on Google. A lot of companies advertise gift cards. Remember that Magento has some flexibility around gift cards that can be both downloaded or sent to you as a physical card, so you might have to grapple with some shipping issues with how they’re delivered and specifying that correctly. But if you recreate these guys as individual gift cards in individual denominations, you can get these listed in Google Shopping. Basically just said all that – fixed-price gift cards, you can use fixed-price gift cards as landing pages and make sure that you understand what you’re doing with shipping if you’re actually mailing somebody a card versus having a downloadable card.
I want to talk briefly about some export methods and how things have been evolving. And we’ll kind of steer it towards a wrap-up with this. Everybody’s pretty familiar with marketplace extensions that create shopping feeds. These guys are really common, they’ve been around for a long time, and it’s the default go-to that agencies use when they’re deploying sites. They’re pretty straightforward, they can be rapidly configured, they’re very well supported. Increasingly, though, there are other options and in addition to the pros here, there’s a few cons, as well. One of the key cons that I’ve come across that worries me sometimes, is we have a situation where, maybe the developers that initially set up a shopping feed are not really involved in the ongoing marketing and maintenance of things going on with product disapprovals and that stuff over time. So maybe marketing folks in the business are working on that and they might be using an outside ad agency, they might share access to the feed generation with the outside ad agency, and you can have a lot of thinkers involved in these from people that aren’t developers and don’t know coding. But most of these feed extensions allow you to not just map fields, but to insert arbitrary PHP code to handle some of the transformations that you need to accomplish in order to make the data valid on the Google side. So you’re basically giving some unknown person in a third-party company access to insert PHP code on your Magento site. And that scares me. I’ve spent enough time on the merchant side to not want that at all. It’s potentially a little dangerous. So I don’t know how that’s going to evolve going forward but there’s a lot of really good options here. Obviously, you can mitigate some of those issues just with the more tight roles and so on and involving actual developers in your shopping feed maintenance – pros and cons. Increasingly, though, we’re seeing more people using API methods that don’t even involve running code on your Magento server. This is something we’ve been leaning toward a lot over the last year. With Magento 2 the API coverage just got dramatically better than it was before, and you can get all the data you need out of Magento, the rest API, third-party SAS platforms can do this now, custom code can do this. There’s various companies now with various solutions. And then Google, as well, Google Ads has dramatically improved their APIs recently. They’re in a major shift right now. You can do just everything via the API. So there’s some pros and cons here as well, right? Interacting via API can actually sort of be a little bit slow sometimes, dealing with really large catalogs and waiting for all sorts of queries to return, checking stock status and availability. But then on the flip side, you also have the ability to sometimes pull data from multiple sources, maybe got a different VRP system over here and we actually want to pull inventory from there rather than from the Magento site. You’ve got a lot more control over what’s possible. I’d say also that has led to some of these systems becoming a little bit more sophisticated with data caching. Most of your product data doesn’t change every day -- do we really need to update the image link on a daily basis and go through a complete rebuild of everything with what images that are in the gallery? We don’t really need to do that, so some intelligent caching coupled with external rest, that can be useful. Then also, all this processing isn’t bogging down your server on a site that’s really complicated with a lot of complicated processing and a ton of products. That’s kind of interesting.
Cory, who introduced me, has been working on this product a lot lately. Magento is including the Google Shopping Ads Channel in future releases of Magento, and this new product just got launched a couple weeks ago. This is kind of a little bit of a hybrid between the two, so this runs on the Magento site, it’s very well integrated with the core platform. The Shopping Ads Channel basically uses a simple ETL approach -- we’re mapping some fields together, we’re getting some data from Magento over to Google. It’s interacting with Google completely via the API. And this can actually take you from not advertising at all to advertising in just a few minutes, because in this scenario it actually logs in, uses the Google APIs to create everything you need and then load all the data in and continue to keep it synchronized after that. So it’s interesting. I will say that there’s a lot of things that are pre-configured to use a certain way of doing things so if you already have complicated ad accounts and particular strategies you’re using in your shopping campaigns or you need a high level of customization in some of the data transformations you’re doing in your feed, then out of the box this probably isn’t the right solution. This would be a very good solution for S&Bs that maybe haven’t had as much success in the advertising space or aren’t advertising at all on Google Shopping and want to turn it on easily and try it out without a huge level of commitment and without a lot of need for paying developers for a lot of customization.
So this product just came out a couple weeks ago, it’s available for download now, it’s available for Magento 2.2.4 and above, but later on this year, in 2.3.3 it’s going to be included in the core platform both for Commerce and Open Source. So keep your eyes on that and it’ll be interesting to see how this product develops over time and how it sort of continues to evolve the landscape. I guarantee you, this is going to trigger some other extension providers to improve rapidly. I like that! Competition is good.
Just to wrap up – most sites do the basic process of ETL well. The basic things about exporting simple products and getting the data into Google – that’s usually not that big of a deal. It’s things like the configurable and the grouped and the bundled products that cause people the most issues, and configurables are just so widely used, every site I look at has problems with configurable products. Just remember that anything involving variation, you’ve got to look into what you’re doing with that item_group_id field. Anything using a bundle, go read the docs on the is_bundle attribute, and remember that most of the problems people have are also tied in with that landing page policy, so it really behooves you to go and read it completely and try to understand every little thing. Sometimes there’s just some random little sentence in there that says what you can or can’t do, and that makes all the difference.
If this was at all useful to you, I gave another presentation a few months ago at Mage Titans in Texas and it was Real World Google Shopping Feeds – How Data Quality Improves Profits. That presentation is a video available online at our blog. That presentation covers a lot more about sort of profit-centric approach to thinking about feed optimization and how data quality improvements in the feed can really improve what’s happening with findability and ultimately with profitably in what you’re advertising.
Feel free to stay in touch. I love talking these issues over with people and helping them out. If you’ve got particular things that we could help talk you through or answer some questions or just brainstorm a little bit about, I don’t know all the answers, I haven’t read every Google policy under the sun but generally we can figure out where to look and sometimes just talking about other people that have been in the trenches working on these things can really help.
So I think I’ll wrap it for now and we definitely have a few minutes for some Q&A if you guys have some questions or things you’d like to chat about.