Part Catalog – Magento 2 migration done right

1. The Part Catalog story

We’ve worked with Alan since 2015 when he approached Magently with an unfinished shop on Magento 1. Alan was trying to set up a store with automotive parts and his developers of that time weren’t able to handle the project, so we took over. We sorted the shop out and since then we’ve been working on improving the performance, building new functionalities and fixing bugs. We’ve had a very good relationship with Alan – he’s always been very understanding and easy to communicate with.

At the end of 2018, Alan decided to migrate from Magento 1.7 to Magento 2. At first, he just wanted to work on improving the Magento 1 site’s performance. We had previously done a number of optimization changes and after a productive conversation, it quickly became apparent that further changes to the Magento 1 store would be very time-consuming. Combining that with the fact that official support for this version of Magento is scheduled to end in June 2020, it seemed to be a good idea to get going with the migration right away, instead of waiting till the last moment. It was also a good opportunity to clean things up and work on the performance on a new version. It gave us enough time so that we didn’t have to rush in fear that we won’t manage to finish before June 2020.

Therefore, in December 2018 Magently started working on migrating the entire e-commerce features and data to Magento 2.3.

The original website on Magento 1

2. Concerns before the migration

Our first concern was the version of the original shop which was 1.7 – it’s quite old and we weren’t sure if the migration process would go smoothly. Magento informs that you can migrate the data of Magento versions from 1.6.0.0 up to 1.9.4.1 to any Magento 2 version (data-migration-tool is a separate module for Magento 2, so it is included in the new releases). Moreover, there should be no significant differences in migrating the data from various Magento 1 versions – the process is the same, only the configuration files change. However, whoever worked with this CMS knows that not everything goes according to the initial plan and you get a lot of obstacles on the way, which was the reason for our doubts. 🙂

Furthermore, the shop was very customized with quite a few modules (and we all know that Magento modules like to get into conflict with each other) and a key complex customization that we had to recreate on the new installation. We only had a rough brief with general information and the code from Magento 1, which we had to analyze to understand the functionality in depth. The advantage was that we’ve done some work on the original Magento 1 integration and only had to recall the details. 

The customization was split into 2 parts which were both interconnected and gave a single result – creating a kind of product search based on car’s year, make and model selected by the client. The functionality was necessary because Part Catalog has several million records in its product database – it would be a tough task for the end client to go through all of them to find the product they need and that also matches their car.

We had to analyze the available options and design the final functionality. You’ll find more information on our solutions below.

Our last major concern was the version of Magento 2 that we should migrate to. Two weeks before we started working on the migration, Magento 2.3 was released and we weren’t sure if it’s stable enough to use it. We usually wait a while before updating our clients’ shops to make sure that bugs in new versions are fixed. But this time we had to make a decision quickly taking into account the benefits and potential risk. 2.3 introduced quite a few important changes that would make our work more efficient – if we didn’t go for it, we would have to do the update sooner or later and it would eventually involve more work.

We made a thorough analysis of the new version including the changes and their influence on the required customizations, and we decided in the end that Magento 2.3 will be a good choice.

3. The process

At the very beginning of each project, we try to ask as many questions as possible to make sure we’re on the same page with our client. Understanding the requirements is the key to a successful project and we try to prepare an accurate quotation, if possible.

Sometimes it’s not the case though. Sometimes a client is in a rush and prefers to start work in a more agile way. This is what happened here – we’ve done an initial analysis of the requirements, prepared a simple plan and a very initial estimation. The project was very complex and customizations would require much more time to be properly investigated, therefore, together with Alan, we decided to start it without further delay and carry on without a full upfront quotation. We’ve worked together for a few years now and both parties trusted each other, so it was a safe decision to make.

Our process for managing the project, in this case, was the following:

• The migration first had to be done locally, so that we could work on the
  production database until the final migration – it helped to avoid additional bugs
  at a later stage. We set up a project on our dev server with obfuscated data.
• A development environment on Alan’s server was also set up – we pushed
  ongoing changes to the server, they could be tested in an environment similar
  to live.
• We installed the most important modules, especially the ones that could affect
  the look of the site.
• The real work started here – two developers worked on the major functionalities
  and the rest on the theme.
• Once the shop on Magento 2 was completed and tested (Alan was very helpful
  and complimented the work of our PMs and QMs in terms of verifying most
  of the functionalities and pages), we’ve done another migration from the
  Magento 1 production instance to Magento 2 on the client’s staging.
  At this point, the database from the first migration was outdated.
• After the migration, there were some tests to be done and bugs to be fixed.
• We dumped the database and did the final migration from Magento 1 to
  Magento 2 to create the target live installation. At this moment, it wasn’t
  possible to make changes in the Magento 1 admin panel (this included settings,
  products and categories – the only allowed operations were to be done on
  orders) and on Magento 2 (no changes allowed for orders, users, products,
  inventory or orders – we could only change the configuration in Stores).
• Of course, we all love tests, so there was more testing at this point. 🙂
  Once we made sure that the live-to-be installation works fine, we had to disable
  the shop –    the shop is based in Texas and it only covers the United States,
  so we enabled maintenance mode in the morning of our time (GMT+2) to make
  sure that only a handful of users will be affected by the downtime. We ran the
  delta migration – this is an ongoing migration that transfers all new customers
  and orders to Magento 2. The migration was done the day before, so delta was
  very quick in this case.
• We could now switch the live site from Magento 1 to Magento 2. The downtime
  was less than 30 minutes and during this time the shop was available solely for
  Magently and Alan. We performed tests again and when we were happy with the
  results, we disabled maintenance mode on Magento 2.

The shop migration process is quite complex, but it can be a smooth operation if everyone works together, communicates well and all hands are on board when necessary. 

It has to be remembered that “migration” can mean two things:

1. The whole migration process which also involves building Magento 2 shop from scratch, recreating all integrations, functionalities and layout. This is the main focus of the project because it’s the most time-consuming.
2. Data migration – migrating the database from Magento 1 installation, including orders, customers, products, categories and attributes.

Data migration is very quick compared to the whole process. In this case, it only took around 3,5% of the total project duration. Much more time is needed for related operations or setting up the new shop. Of course, it strongly depends on the database and customizations of a given shop – each case should be treated individually.

4. Challenges

1. Lack of detailed brief or designs – Alan provided us with a short document consisting of an overview of the project, general requirements and the list of used modules. He was flexible, gave us space for improvisation and wanted to keep as much of the Magento default functionalities as possible. We had a general idea on how the customization worked, but we didn’t make a thorough analysis of the code at the very beginning – we would have to spend a significant amount of time on it and Alan wanted to start the project as soon as possible to ensure the shop is relaunched before Easter.
2. Differences between Magento 1 and Magento 2 – some data was migrated correctly, but we found out that there were further problems, for example, styling issues caused by syntax changes.

5. Our solution

The team

The team working on the project on our side was:

• Project Manager, 
• Project Lead, 
• 2-5 Developers depending on the needs during the project and our availability,
• Quality Managers.

Since we know that with this size of the project a client needs to have a good overview of the progress and remaining tasks, and Alan was well-organized and informative, we decided to let him into Redmine which we use internally to manage all our projects. We also set up a channel on Slack where Alan was able to reply directly to our developers. It made our work completely transparent and helped with communication and quick feedback.

It was a very good decision – both Alan and Magently were happy with the quick access and cooperation. Some tasks could be directly assigned to Alan and he could see the whole conversation or work progress in a given thread.

Communication is very important, so we had regular daily meetings with the developers and a catch up call with Alan every 1-2 weeks depending on the progress. We wanted to make sure that everyone has up-to-date information and is aware of any potential risks. This way decisions could be made much quicker.

Magento version

Eventually, we decided to choose Magento 2.3 – unfortunately, the decision has not only had advantages, but also various disadvantages. On the one hand, it saved us time as it contained some functionalities we’d otherwise have to write from scratch. On the other hand, some of the modules we needed to use were not compatible with 2.3 yet. 

Alan was eager to help us at any point during the project, so we asked him to contact all module providers that didn’t have the compatibility information stated on their websites. We had around 15 extensions to install – the list had to change during the process because some modules indeed didn’t support Magento 2.3. Flexibility is a must if you have a complex project and a limited budget (which is often the case 😉 ).

The project took a few months, therefore, we waited with installing some of the modules – otherwise, their free support periods could have finished before we delivered the project. It mainly applied to modules that didn’t affect the theme and with which we didn’t have to worry that we will need to adjust the frontend again after installing them. In the meantime, the compatible versions with Magento 2.3 have been released. Still, we had to skip the non-critical ones and some of the others had to be adjusted by us. 

In the end, though it presented some challenges, Magento 2.3 turned out to be a good choice and didn’t cause too many problems during the development.

Customizations

As mentioned, we’ve had quite a complex customization at hand which we divided into two parts. The first one used an external database with vehicle configuration data (year, make, model, engine type, etc.) and information about the relation of the vehicles and the available parts. It allowed for filtering Magento 2 products by selected vehicle and this was the responsibility of the second part. The functionality was split this way so that we could work simultaneously on both parts and complete it much quicker. 

Customization #1

Here we created a selector for the year of the car, make and model – it involved complex mechanisms – and the next option could be displayed depending on your selection. We had to provide integration with an external database and make sure that Magento treats it as if it was its own. We implemented it next to the default Magento filtering system and then made a connection.

Customization #2

It was implemented as a custom product type with children products from one manufacturer. The work was done mainly from scratch – we decided that we would have the chance to write clean code and that it had to be adjusted to Magento 2 anyway. We had to take many relations into account when coding it, for example how it behaved on the frontend, applied prices, or how it affected configuration options in the admin panel, etc.

Most products in Part Catalog were included in Customization #2 (and there are 200,000 products in total), so we had to make sure that the functionality is quick and we keep in mind its speed during the whole project. The interaction with the database had to be present on the lowest level, so that only the essential operations are executed. Otherwise, processing the data could strongly influence the site speed and cause significant delays.

Why wasn’t it simply created as a category or bundle products? The main reason is its implementation in Magento 1 that was done in a similar way. We analyzed different options, and in the end, decided that changes at this stage would cause much more work than necessary. The data migration would be too time-consuming because we would have to write code translating the current implementation to the new approach – from heavy data mapping changes to special modifications of category pages. When starting the migration, we had a ready database from Magento 1, the structure of which was adjusted for the customization and we could simply use it.

Moreover, this solution was more flexible and gave us better control over the code and future modifications. If we created it as a bundle product, we would have to strongly depend on the core mechanisms and modifying them could mean more work than writing a new product type from scratch.

Regular tests

We believe it’s better to be safe than sorry, therefore, we kept testing the website to make sure that we don’t find out just before going live that there are loads of bugs and we have to significantly move the delivery date. When planning the project, we allowed 8 working days for bug fixing just before the final migration. Eventually, we didn’t need so much time at that point, because some tests had been run much earlier and we’d already addressed some of the issues. However, we did have a small delay at the end of the project, so I believe that it all balanced out quite well.

Setting up a limit

As a shop owner, it might seem that every feature that comes to your mind is essential right away, but you need to prioritize tasks and some of them sometimes have to be paused for a while. Otherwise, you will never go live – that’s the hard truth. The key should be to start using Magento 2 as soon as possible so that you and your clients can benefit from it. Some features can be done post-migration and you have to keep the possibility in mind. 

The same situation happened in our project – when migrating Part Catalog, Alan wanted to do a few checkout changes. This is the most vulnerable part of the shop and it takes a significant amount of time to make modifications even if they seem easy at first glance. Alan decided that going live is more important and we moved the changes to a later stage.

Checklist

The procedure of switching sites and running commands on a live database is always stressful to developers, so we created a checklist for going live – it included not only the standard procedure such as checking if emails work or whether cron is configured but also manual settings that had to be set up in the admin panel.

6. The Effect

Both the client and Magently were happy with the outcome of the work. We had a small delay with delivering the project and a few bumps on the way occurred – but hey, how many Magento projects go exactly according to plan? 🙂 I believe that we managed to handle the problems smoothly and quickly, and Alan was also supportive and very helpful all the time. 

The project took around 5 months and 2 thousand hours to deliver, and we went live 2 weeks after the planned date. Since we didn’t have the detailed scope of the required work, we weren’t able to specify the deadline until the end of the third month. We were completely flexible, but we kept an eye on everything so as not to cause unnecessary delays. This is not a standard procedure, but our relationship with Alan was very good and both parties believed that this is the best way to proceed.

The most time-consuming part was the major customization – you can see a breakdown of the whole project below:

• Customization #1 and #2 – 47% of the whole project
• frontend – 19%, 
• data migration – 3,5%, 
• testing & fixes – 12% 
• modules installation – 2%
• setup, Ops, planning, communication, initial optimization and other custom product types – 17%.

You can see the current website on Magento 2 under this link.

The best would be to use Alan’s words here:

I am very happy with the result. I know it took a lot more time than originally expected, but I also understand that it’s almost impossible to accurately quote a project like this. Having Redmine access for this project was key. I never felt in the dark and could jump in and clarify something when needed. I think the project would have been very frustrating if not for this access, as I wouldn’t be able to easily see issues & progress.

Their communication and professionalism stood out, as did their project managers and all the developers. Some of our requirements were rather difficult, but Magently was professional in helping us develop appropriate solutions.

Because we’re now on Magento 2, we no longer have the threat of the Magento 1 end-of-life. As a result, we’ll be able to maintain the best security practices and patches. Additionally, because Magently developed our website with best practices, we’re able to easily extend it to cover future functionality requirements.

7. Lessons for the future

The shop is up and running now, but that’s not the end – we need to form some conclusions for the future to learn from our mistakes and be better prepared! We decided to share them with you.

1. Never update the Magento version during the process – it’s always better to stick to one version that is carefully chosen when the project starts. I know it might be tempting to do the update just before going live, because the new version introduces nice features or seems completely safe. It’s definitely a bad idea and it should be treated as a separate project, once you finish the current one. You’re not able to predict the potential issues and this way you will avoid the unnecessary risk and postponing the delivery date. Once the migration is done, you can think about the update to the newest version and work on it with peace of mind.
2. Be prepared that some modules might not work out of the box – it’s the case with all Magento projects – sometimes modules might conflict with other modules or simply include new pages or elements that haven’t been styled. That’s typical.
   But now imagine that some modules change the way they work for Magento 2 – that actually happened with Alan’s extension for the sitemap which caused some problems even though it was compatible with Magento 2.3. It turned out, eventually, that in Magento 1 the sitemap used an attribute called exclude from sitemap and in Magento 2 they changed it to include in sitemap. Doesn’t seem like a big change, but the migrated data simply showed wrong results.
3. Don’t be afraid of research and planning –  developers like to do their job – I think that for most of them it would be best to just sit down and start coding right away. However, if you create a functionality, but ignore it in the context of the whole project and its future, you are more prone to compatibility issues and might end up rewriting certain parts at some point. The process is sometimes time-consuming, but it helps you avoid unpleasant surprises.
   The whole project should be at least initially planned so that you have the general picture of what’s needed. Additionally, all major functionalities should be analyzed and thought out just before you commence the work on them, and it’s the best to do it with the Project Lead who is usually the only person engaged in all the technical discussions in the project. You should analyze the old version of the shop if there is no detailed brief and check the modules in terms of compatibility, customizations and code quality. 
4. Predict the unpredictable – estimating in Magento is – to a certain extent – just guessing based on previous experience and knowledge. But every shop is unique and so is its code and set of modules. Therefore, we always add some extra hours on top of our estimations to reflect the risk factor that should be taken into account – we believe it’s better to be prepared for a bigger budget than to find out at the end of the project that you’ve run out of funds and the site is not finished!
   Migration is usually not a quick project and you might have new ideas in the meantime that were not included in the estimation. Some features might not have been described in enough detail or there might be bugs that will have to be fixed and will occur for sure. It’s also good to remember about optimizing the speed after going live. And that’s how the time and budget is growing relentlessly!
   
5. Testing, and testing again – we knew that testing is key and we included a lot of time for it, but what we didn’t think of is that we would not be able to test everything on staging, because the API for some products only works with the live installation and we can’t simply change it. This is something to remember about the next time. In this case, we didn’t have any major problems – there were a few minor bugs that were quickly handled, but it gave us the creeps when we found them!

In general, the project went quite smoothly and everyone is happy with the results.

Have some questions? Write them in the comments.

Need a migration? Contact us to get a free quote. We’ll be happy to guide you through the process.