SaaS Shipment Tracking System
Stanislav Stankov
This is why developing a shipment tracking SaaS is so hard!
Rush is a shipment tracking app that provides post-purchase solutions. This is my experience as a product manager with Rush. First things first, developing a parcel tracking solution is a subject that can’t and shouldn’t be approached casually; which we did a couple of years back when we started. Before going too much into that, let’s cover some basics:
What is a parcel tracking solution?
A sophisticated software system where you input your parcel number (with optional additional details such as delivery country, and order number, order date) and as output, you get carrier information, categorized parcel checkpoints, and the current status of the shipment it is in.
Input example:
Output example:
How did we end up here?
Just to give you some context, we are currently on version 3 of the tracking solution. This is the 3rd version that we rebuilt from the ground up. With the expansion of our business model, the architecture we developed at the start became unable to support new requirements, so we needed to start from scratch.
The first version (v1)
Initially, the tracking system, v1, was part of the monolith. It produced a limited output and was able to process only 3 carriers’ APIs:
USPS
ChinaPost
YunExpress
Because of its simplicity — presenting carrier messages on a tracking page, the system had a good performance.
Very soon, we realized that keeping the tracking processing codebase inside the monolith code restricts us to release faster patches to needed places. Not being able to release because of dependencies on other parts of the system, degraded the quality of the service. There was also an increasing need to support more carriers and the code was failing to streamline all different carrier scraping needs, not to mention the code was already spaghetti.
The second version (v2)
Coming to the 2nd version, v2, we decided to build a tracking API system that basically handles all logic of getting carrier response (by API or scraping), processing the carrier response, and returning standardized information. It was also encapsulating all carrier processing logic in a single repository, so the learning curve getting developers to start with it was smaller, plus we were able to release it independently. It was more of a poorly designed microservice that acted as a proxy, encapsulating the carrier domain. Simply put, it received tracking numbers and produced outputs with related information, on request. On its own, the service did not do anything.
After a year with this system, it started showing limitations performance-wise, e.g., queries started to take more time, occasional SQL locks, scaling Kubernetes pods was inefficient, code architecture was outgrown based on the new learnings, and it was hard to trace the root cause of an issue. The system was also failing to measure its ability to map tracking numbers to carriers, and carriers’ messages to be categories like Pending, In Transit, and Delivered… to name a few. There were constant complaints about wrong carrier mappings and incorrect mapping of shipment checkpoints statuses from clients. The biggest problem of all was that we really did not know how well we were doing. Are we mapping correctly 80% or maybe 90%? How can we track that we are moving in the right direction?
At this point, we were processing over 350K shipments per month, while requesting between 4 and 6 times each undelivered shipment per day. So we had roughly ~ 1.5M calls to external systems, requesting new shipment information. You need to keep in mind that there are just a handful of carriers that will push updates to your system, once shipment data changes. For 99% of carriers, you need to constantly pull the information and compare it with the old one if there are new updates.
To add fuel to the fire, the system also started to hit limits that we had never experienced before. It was hard to scale service effectively while hitting 429 limits or getting IP blocked. Meanwhile, the carriers needed different strategies for overcoming limitations and the system was not designed to process shipments that had next-mile carriers and in turn, were unable to return valuable information to the customer queries. It became a crazy game with all work and no play.
The third version (v3)
This series of never quite hitting the mark brings us to the third version, v3. This time, we built the system with what we learned along the way. This should be noted here that the failures we saw with the previous versions allowed us to eliminate errors in v3 and with v3, now we have a fully standalone microservice that manages all shipments, carriers, and address domains. It has workers, cron jobs, and more. It encapsulated better the information we needed in the Tracking service to make decisions, and also was able to scale independently in an efficient manner.
We also decided to increase the developer team from 1 person to 3, constantly improving the quality and scale of the service. Along the way, I learned that if you want to cover 500 carriers with minimum errors and have your software support them; you need to have at least 2 teams working with roughly 7 developers in each. Now, one team is focused on running the business (keeping the carriers up to date and responsive), and the other has been working on keeping infrastructure and architecture up to the mark.
V3 infrastructure and capabilities
Our current infrastructure setup:
AWS EKS
Kubernetes with Helm templates
Bitbucket CI/CD pipelines (I’m a fanboy of Atlassians)
Codebase Laravel 9 with Horizon
Databases: RDS (Aurora PostgreSQL) and Redis for queues
Kafka — In process to serve us for broadcast shipping events
At the time of writing this article:
We support over 1100 Shopify stores;
We process over 500k tracking numbers on a monthly basis. (We can process up to 500k on an hourly basis based on our performance testing)
We are doing ~ 2,7M daily calls to grab new carrier information (this is without the webhooks ~ 50k daily injections).
We process over 90 different carriers — using both API and page scraping.
We rotate API keys, and proxies to overcome connection limitations.
We also integrated a dashboard showing us important tracking system qualitative and quantitative metrics. With it, we are able to quickly point out areas that need improvement. To not share all the secrets, I blurred the data, but you can see how many things we watch on a daily basis. Also note that this is just application-level data, and infrastructure, we track it in another Grafana dashboard.
What are the problems we hit so far and how did we solve them?
We can group the problems we faced into multiple categories, part of them are closely related but the major areas are:
Big data problems
Demanding infrastructure
Getting It Right First Time (GIRFT)
Carriers integrations
1. Big data problems
These are related to the velocity, variety, and volume of information. The biggest change in the tracking v3 service was the implementation of the pipeline for the tracking parcel. Rather than a single process that gets the tracking number, processes it, returns carrier information, and standardizes it; we have broken it down into multiple steps with multiple validations at the end of each one. This gives us the ability to track, monitor, and validate that each step is completed before moving forward. Simplified steps of our pipeline include:
tracking injection
carrier assignment
information retrieval
information processing
information storing
notifications broadcast
Here are a few points to keep in mind when you are creating such a system:
Being able to process 5k shipments per hour and 500k per hour — is a totally different system, with a different architecture design — you should break down the process into steps and move through it. You should be mindful of the lead and cycle times of the pipeline and monitor it closely.
Track how many shipments are processed correctly — you need to have a built-in system for monitoring it and if the percentage of correctly assigned shipments starts to drop, it should be fixed on a priority basis.
Track how many carrier messages you map correctly. A carrier message or checkpoint is a single line of shipment status. Example: “Your item arrived at our NASHVILLE TN DISTRIBUTION CENTER ANNEX origin facility on June 25, 2022 at 10:11 pm. The item is currently in transit to the destination.” We need to create a rule to take out the date (2022–05–25 10:11:00), location (Nashville, Tennessee, United State), and status (In transit) of this message. Historically speaking, at one point we were not able to recognize and classify over 5M unique checkpoints. The complexity comes from each carrier having a different message in a different language, so you need to think about a maintainable codebase to support mappings.
Frequent writes/deletes from the database-based carrier updates — database architecture needs to support it and be optimized for writing. We are applying the CQRS Design Pattern (read from replicate and write on master), but the system should be designed with such in mind. Keep on reading to learn a bit more about it.
Automation tools/scripts — as we can’t manually process or see every error that happens, so we use scripts that help us validate input data, output data, verify the ordering, type, and the content of information among other things, and if there are errors, we want to group them and solve the ones with the highest volume or impact.
Handling multiple carriers with inconsistent APIs requests, response schemas, and checkpoints messages mapping is a must. You also need to streamline the process as much as possible, to support the variety of carriers, but adding and maintaining carriers to not take a lot of time. There is not a single carrier that has a similar system to another one. USPS uses XML, FedEx uses SOAP, YunExpress uses JSON, and for Yanwen, we scrape the tracking page directly (HTML). Each carrier has its own data structure and message patterns, and they also group data differently. Some of the carriers also change their schema frequently. We are seeing more and more carriers’ changes, as their IT systems evolve and cope with the demand.
Each carrier has its own usage protection — some use rate limits by API key or IP, some use captchas, and some use leaky buckets. Not being able to overcome them, you will not be able to get timely updates. There are different strategies to handle them, but you need to take into account several workers sending a sequential request to the same carrier. You do not have a single machine that is doing subsequent requests, you can have parallel. We sometimes have up to 180 workers at a time and they coordinate in between which system they can request without hitting limits. So you need some orchestration between those not to break the limits, but also a way to scale up and down, and be very efficient with the server resources.
Carriers changes — a lot of carriers change APIs without further notice, so you need to implement monitoring on the heath of the carrier. You should also stop requesting it (abusing it), if the carrier is down, and let the carrier enter the cool-off period. Carrier API/web services also can go offline or return invalid information (due to development error, or high traffic) and you need to handle those cases, but also apply updates swiftly. That is why we track all our HTTP status and body responses meticulously.
You may need another system just tracking errors, logs, and responses. We use mix — an external system to track errors and the volume, as well as log part of the errors in the operational database. Of course, all logs are cleaned after some time, and there is a database performance hit because of those write operations, but we are able to join those operational logs with other data that would help us in the debug process.
When you have large data sets and you have implemented pipelines for tracking number processing, you need to track to optimize for latencies between different steps in it. You need to know your cycle time (from entering till exit) and know where the shipment just waits for processing.
2. Demanding infrastructure
Service availability and scaling
You need to be aware that we have 2 services that we are scaling:
One service is the API request that we are processing — when other systems need to get shipment information, or when they are requesting shipment to get an instant update, or inserting a new shipment for processing. This service scales based on how many FPM children (those are PHP specifics) we are running and how many we are utilizing. The more API calls, the more FPM children start, and if they are over a certain threshold, we lift a new node to serve those calls.
We want our endpoints to be efficient and less time-consuming as much as possible, and leave the heavy lifting to the workers (scripts that execute based on queues).
The second service is the workers part — scripts that execute jobs based on queue. To organize this, we use Laravel Horizon, the screenshot below:
Here scaling is also done horizontally, the more jobs you have in the queue, the more workers you need to create. We scale horizontally by adding new nodes with additional workers available. We also scale down, where there are not many jobs in the queue, to keep costs down.
The major point is that you want to be able to scale web-facing calls and workers independently from one another. You might want to have 2 nodes serving web requests, and 6 nodes with workers. This should be planned and taken into consideration.
Database
Now that you are able to answer all web requests (thus providing higher availability service) and have a queue that handles peaks, and workers that process them, your throughput will be based on the database performance.
There are lots of ways to optimize databases, and SQL queries time, but I want to focus more on how to design your application from start, and not let it get into the optimization phase quickly. We design endpoints based on CQRS Design Pattern so where we need to read — we read from the Slave database, and where we write — we write on the Master database. So when you are designing the API, you need to consider whether you need a Master or a Slave database connection, and if you need a mix of those — maybe you should further break down the endpoint into multiple.
The second thing is the database architecture. We used to have a `shipments` table that holds all the information about shipments — information, current status, statistics, etc. This led on one side to larger tables, but we also started hitting database locks, for example: when one worker is updating shipment information related to new statistics (for how long is it in transit) with new current location information based on carrier updates, it hit a deadlock. So, designing tables with usage in mind is mandatory — knowing what consumer systems and what data will use it, how often will we write over them, as well as will you need to join the information frequently is quite important.
3. Getting It Right First Time (GIRFT)
This is probably the hardest part to handle — making sure the right carrier is assigned to the tracking number — from the first time. As well as the right status is mapped to the checkpoint message. Most of the store owners do not know what carriers they use in 90% of the cases (or at least for us). Or which carriers their warehouse or fulfillment services use. For those cases, you need to have functionality that recognizes and maps the right carrier to the provided tracking number.
Keep in mind that usually clients are not happy about 98% of the carriers that you do map correctly, but they will keep complaining about the 2% that you did not map. This is at least our reality. So, we are in pursuit to hit 99.5% of all — correctly mapped carriers, currently, we are at 99.1%.
Carrier mapping & recognition
Usually carrier mapping is straightforward. You get the tracking number, you create regexp and you map it to the carrier for example 92001901755477000996406140 is USPS tracking number, and overtime we were able to identify what all of the following tracking numbers can be processed by them:
Problem is that you cannot find that information on the USPS website or in their document. You need to figure out most of the cases by looking at the tracking numbers that you are not able to map, and manually look at which carriers can process them. This is a neverending process, so you need to have a dedicated person/team on it.
What about carriers that have the same tracking number patterns? For example tracking number 303367017250 with pattern ^\d{12}$ can be 4PX, DHLExpress, GLS US, FedEx, Australia Post, and 10 more carriers. Now problems become a bit more complex. What we do in this case is to add a further check on the destination country. Based on that, we can directly eliminate part of those careers. In our example, if the destination country is the USA, we eliminate Australia Post as a possible carrier but keep DHLExpress, GLS US, FedEx, and 4PX. We use multiple other strategies, but I won’t go into them. The main point is you want to eliminate as much as possible, to keep your quota for numbers that you are 100% sure are correct.
Now we call all the carriers and see which will provide the best response for further mapping. Usually, all carriers respond most of the time. So you need to further pick one based on the checkpoint date it has and the information of order creation date or fulfillment date. Not easy, and not 100% but further improves your carrier mapping percentage.
Here is another case, you need to handle if you have the top case implemented. Complexity can come from eventual consistency in carrier tracking systems or how they are simply architected. Some carriers issue tracking numbers, but they are still not trackable immediately or until the first event. This means that there is a gap where if you request information too soon you might think this is not the right carrier, as the right one will return no results. So you should not be hasty to eliminate a carrier as an option if they do not return a result for the tracking number.
The real-life case can be that a merchant buys a tracking number, assigns it to an order, and we get the order update immediately, and thus the tracking number, we check the carrier, but the carrier responds that the tracking number is not available.
Tracking numbers variations
For example one of the FedEx tracking number patterns is a 12-digits pattern
^\d{12}$ however they can also start with 0or 00. Tracking number information is the same, no matter how many 0 you put in front, but those are completely different patterns that you need to account for as well ^0*\d{12}$. So you need to be aware that some carriers do not really handle tracking numbers as STRINGS to match but as numbers.Carriers like R+L Carriers, however skipping the
0upfront can be the difference between returning the tracking information and not.Other carriers like the RoyalMail tracking page support getting tracking inputs as
0213-AF64-0411-EBDE but redirect to 0213AF640411EBDE. See https://www.royalmail.com/track-your-item#/tracking-results/0213-AF64-0411-EBDE and https://www.royalmail.com/track-your-item#/tracking-results/0213AF640411EBDE .
Not understanding that some carriers need to apply tracking number pre-processing, was one issue that we got with a client who complained about it. Actually, RoyalMail Strips the
— and uses what is left from the tracking number to map it. However, knowing this information and adjusting it for all possibilities, increases the chance for multiple carriers to map to the same tracking number pattern. As well as unneeded complexity is added to your services.Message categorizations
All checkpoint message categorization is key to knowing what’s going on with shipment.
Each message should be mapped to date, status, optionally sub status, and geo-location/address. This can be quite hard to be maintained as new carrier messages appear. There are also nuances that you need to make sure your team understands to be able to map them accordingly.
Our current system supports regExp mapping for each carrier message. However, we are on a path towards refactoring the system, unifying them for all carriers, and actually setting prioritization for different categories of regex, creating a single place and system to map the messages.
Here again, you need to be aware of the schemes. Some carrier keeps all data in a string message, so you need to pull out dates, locations, etc. Others provide you a JSON, with the message, and additional nodes with location, status category, and more.
Next-mile tracking number
Example: YunExpress is an international carrier, that uses last-mile — local to country mail services. For example, dropshippers ship from China via YunEpxress, and once the parcel is inside the USA, USPS delivers it to the home address. This means that YunExpress buys tracking labels from USPS, and as soon as the parcel is in the USA it handles it to the USPS.
Understanding which is the last-mile carrier has huge advantages for store customers — they know to call USPS in case of missed delivery rather than YunExpress. They feel more secure, knowing that the delivery will be done with a carrier that they already dealt with.
This hands-off between multiple carriers till final delivery can be done domestic-wise. We are seeing up to far up to 4 hops in the USA domestic deliverables, mostly done with USPS <>DHL pair.
Sometimes however carriers do not share to which carrier they transfer the package, or share the wrong name. So being able to rely on your internal system to find the right carriers is important.
We are sharing this, because it was a major flaw in our v2 architecture design — to be able to process a chain of tracking numbers, rather than just a single shipment. This requires extra planning, different code, and database architecture.
From one tracking number to multiple
Are you aware that one tracking number can actually be multiple tracking numbers? Take for example DHL eCommerce tracking number MYDBJWHF19013. Actually, this tracking number is related to 5 other shipments, that are processed and delivered separately.
Does your system handle these cases? I will leave you to decide how you can handle this case, where not only shipments can be a chain of tracking numbers with a different carrier, but each one can be broken down into multiple :)
Tracking numbers expires
You might want to check the top tracking number on the following link: https://ecommerceportal.dhl.com/track/?ref=MYDBJWHF19013 just to see that it is expired.
You need to be aware that some carrier expires their tracking numbers after 1–2 months of delivery. Some do not, like YunExpress — you can check back years. Other like FedEx, UPSP expire their tracking numbers and rotates them. The rotation includes let’s say tracking numbers from 10000 and go 99999, and after the carrier reaches the end, they start again using 10000 and cycle repeat endlessly. Failure to know that may result in not assigning the right carriers to the tracking numbers, or not being able to handle clients that want to sync tracking numbers months backward.
In summary, to map carriers correctly, automatically to 100% is quite hard. You can see the Pareto rule applies here fully. Getting the first 80% correct rate is quite easy, but the last 20% is quite hard, time-consuming, and requires business dedication on it. To add a few more points, some carrier requires order numbers, order date, or postcode of the delivery destination, so you do not need just tracking number, you need quite a lot of data to be able to pull data based on different carrier requirements.
4. Carrier integrations
If you read so far, you already started to sense that different carriers are processed differently, and there is quite a bit of variability that needs to be taken into account with each carrier implementation, and still be streamlined. Here I want to point out some cornerstones that you want to think about.
Carrier API specifications
While integrating USPS, FedEx, DHL — you start to get the feeling that every carrier has well-documented API and responsive support. However, the reality cannot be more different. Most of the carriers that we implement do not provide to connect over API, or they are not documented, or you need to register a Chinese company to get access (4PX) :) Most of the carriers that we integrated are based on web scaping as the API they provide is with limited information.
Carrier tracking pages can differ from what they return on API
We had this case with the YunExpress tracking page returning one information, while the YunExpress API returns another. The biggest difference was how the status of the shipment was handled — API was in an exception state, where on the tracking page the shipment was marked as delivered.
Batch or single tracking number processing
Some carriers support batching like USPS up to 30 numbers in a single request, and others — you can only request one-by-one. Making your pipeline to support both single and batch requires a bit more planning and effort, but it is a good spend, as you will hit less of limits when your tracking number for the specific carrier starts to increase.
Eventual consistency between print and track
The biggest carriers implement eventual consistency between their print system and tracking capabilities. There are some delays between purchasing a label, and when this label becomes trackable. This can result in checking a new tracking number, and the carrier returns that this tracking number does not exist.
In conclusion,
I hope this article helps you understand more about the pitfalls of tracking carriers, and how across the industry — there is no standardization.
Like this project
Posted May 13, 2025
Iterating over the Core Product Features for Shipment tracking of Rush SaaS, a Shopify App.
Likes
0
Views
2
Timeline
Feb 1, 2020 - Jan 1, 2024
Clients
Rush
