Piloting a Canadian route to Microsoft Azure: lessons learned

Public clouds, such as Amazon AWS and Microsoft Azure, are increasingly becoming a popular choice for hosting an organization’s IT infrastructure and services. They can often provide managed and semi-managed services more efficiently than an average in-house IT department. With more Cybera members leveraging public cloud services such as these, we are looking at ways to ensure they have the best network access possible to the public clouds' data centres.

When leveraging a public cloud, you often have the ability to choose the geographical data centre to host your service. Years ago, it was rare — sometimes impossible — to find a public cloud with a Canadian-based data centre. (When CANARIE partnered with Compute Canada and Cybera in 2011 to create the DAIR cloud for Canadian small-to-medium sized businesses, it was one of the first Infrastructure as a Service clouds in the country). Fortunately, times have changed, and all major public cloud providers now have a presence in Canada.

As great as this is, getting the best possible network path to these Canadian data centres can be a challenge. Focusing on Microsoft Azure, all traffic, regardless of the data centre, is first routed west from Cybera to Seattle. This means that traffic to Azure's Canadian data centres in Toronto and Quebec City first travel west before going east. This results in a round trip of approximately 59 to 68 milliseconds. While not bad, it's not great, either.

Route Path from Cybera to Azure Canada East (Quebec City):

traceroute to 40.69.97.22 (40.69.97.22), 50 hops max, 60 byte packets
 1  * * * * *
 2  162.246.156.1 (162.246.156.1) [AS15296]  1.009 ms  0.978 ms  0.949 ms  0.918 ms  0.895 ms
 3  host-199.116.233.45.cybera.ca (199.116.233.45) [AS15296]  0.767 ms  0.783 ms  0.756 ms  0.727 ms  0.706 ms
 4  clgr2rtr2.canarie.ca (199.212.24.66) [AS6327/AS6509]  0.711 ms  9.107 ms  0.598 ms  9.032 ms  8.934 ms
 5  * * * * *
 6  sttl1rtr2.canarie.ca (206.81.80.189) [*]  15.760 ms  15.745 ms  15.463 ms  15.368 ms  15.323 ms
 7  microsoft-1-lo-std-707.sttlwa.pacificwave.net (207.231.242.7) [*]  14.981 ms  15.058 ms  15.030 ms  14.995 ms  14.912 ms
 8  ae28-0.ear01.pdx31.ntwk.msn.net (104.44.233.84) [AS8075]  19.164 ms *  19.190 ms * *
 9  be-21-0.ibr02.pdx31.ntwk.msn.net (104.44.21.61) [AS8075]  69.633 ms  69.320 ms  69.381 ms be-20-0.ibr01.pdx31.ntwk.msn.net (104.44.21.59) [AS8075]  69.589 ms  69.866 ms
10  104.44.16.70 (104.44.16.70) [AS8075]  69.920 ms  69.884 ms  69.846 ms  70.806 ms 104.44.16.72 (104.44.16.72) [AS8075]  69.761 ms
11  be-7-0.ibr02.cys04.ntwk.msn.net (104.44.18.224) [AS8075]  69.491 ms be-8-0.ibr01.cys04.ntwk.msn.net (104.44.18.222) [AS8075]  70.083 ms  71.049 ms be-7-0.ibr02.cys04.ntwk.msn.net (104.44.18.224) [AS8075]  69.461 ms  69.284 ms
12  be-8-0.ibr02.dsm05.ntwk.msn.net (104.44.18.151) [AS8075]  69.375 ms be-5-0.ibr01.dsm05.ntwk.msn.net (104.44.19.87) [AS8075]  71.407 ms  69.775 ms  69.780 ms be-8-0.ibr02.dsm05.ntwk.msn.net (104.44.18.151) [AS8075]  69.295 ms
13  be-7-0.ibr01.ch2.ntwk.msn.net (104.44.19.250) [AS8075]  69.842 ms be-4-0.ibr02.ch2.ntwk.msn.net (104.44.19.252) [AS8075]  69.720 ms be-7-0.ibr01.ch2.ntwk.msn.net (104.44.19.250) [AS8075]  69.705 ms be-4-0.ibr02.ch2.ntwk.msn.net (104.44.19.252) [AS8075]  69.235 ms *
14  be-5-0.ibr02.yto20.ntwk.msn.net (104.44.17.146) [AS8075]  69.766 ms  69.830 ms  69.793 ms be-8-0.ibr01.yto20.ntwk.msn.net (104.44.17.144) [AS8075]  69.830 ms  69.933 ms
15  be-7-0.ibr02.yqb20.ntwk.msn.net (104.44.28.102) [AS8075]  69.486 ms be-4-0.ibr01.yqb20.ntwk.msn.net (104.44.28.100) [AS8075]  69.992 ms  78.365 ms  69.817 ms  69.635 ms
16  ae24-0.yqb20-96cbe-1a.ntwk.msn.net (104.44.11.198) [AS8075]  68.855 ms  69.067 ms 

Route Path from Cybera to Azure Canada Central (Toronto):

traceroute to 52.237.21.246 (52.237.21.246), 50 hops max, 60 byte packets
 1  * * * * *
 2  162.246.156.1 (162.246.156.1) [AS15296]  1.627 ms  1.638 ms  1.629 ms  1.610 ms  1.597 ms
 3  host-199.116.233.45.cybera.ca (199.116.233.45) [AS15296]  1.599 ms  1.583 ms  1.559 ms  1.562 ms  1.544 ms
 4  clgr2rtr2.canarie.ca (199.212.24.66) [AS6327/AS6509]  0.880 ms  0.572 ms  0.815 ms  0.790 ms  0.784 ms
 5  * * * * *
 6  sttl1rtr2.canarie.ca (206.81.80.189) [*]  15.227 ms  15.148 ms  15.174 ms  15.132 ms  15.341 ms
 7  six1.microsoft.com (206.81.80.30) [*]  14.818 ms  15.107 ms  15.022 ms  15.026 ms  15.103 ms
 8  * * ae27-0.ear01.pdx31.ntwk.msn.net (104.44.236.18) [AS8075]  18.740 ms * *
 9  be-20-0.ibr01.pdx31.ntwk.msn.net (104.44.21.59) [AS8075]  60.141 ms be-21-0.ibr02.pdx31.ntwk.msn.net (104.44.21.61) [AS8075]  60.472 ms  59.710 ms be-20-0.ibr01.pdx31.ntwk.msn.net (104.44.21.59) [AS8075]  59.965 ms be-21-0.ibr02.pdx31.ntwk.msn.net (104.44.21.61) [AS8075]  59.630 ms
10  104.44.16.70 (104.44.16.70) [AS8075]  60.126 ms 104.44.16.72 (104.44.16.72) [AS8075]  59.877 ms  59.650 ms  59.696 ms  59.677 ms
11  be-7-0.ibr02.cys04.ntwk.msn.net (104.44.18.224) [AS8075]  59.730 ms be-8-0.ibr01.cys04.ntwk.msn.net (104.44.18.222) [AS8075]  60.088 ms be-7-0.ibr02.cys04.ntwk.msn.net (104.44.18.224) [AS8075]  59.528 ms  59.516 ms  59.925 ms
12  * be-8-0.ibr02.dsm05.ntwk.msn.net (104.44.18.151) [AS8075]  59.810 ms be-5-0.ibr01.dsm05.ntwk.msn.net (104.44.19.87) [AS8075]  60.049 ms * be-8-0.ibr02.dsm05.ntwk.msn.net (104.44.18.151) [AS8075]  59.385 ms
13  be-4-0.ibr02.ch2.ntwk.msn.net (104.44.19.252) [AS8075]  59.482 ms  59.443 ms  59.397 ms  59.836 ms  59.331 ms
14  be-5-0.ibr02.yto20.ntwk.msn.net (104.44.17.146) [AS8075]  59.555 ms  59.741 ms be-8-0.ibr01.yto20.ntwk.msn.net (104.44.17.144) [AS8075]  60.213 ms be-5-0.ibr02.yto20.ntwk.msn.net (104.44.17.146) [AS8075]  59.482 ms  60.872 ms
15  ae102-0.icr02.yto20.ntwk.msn.net (104.44.20.150) [AS8075]  59.790 ms be-1-0.ibr02.yto30.ntwk.msn.net (104.44.7.162) [AS8075]  59.889 ms be-1-0.ibr01.yto30.ntwk.msn.net (104.44.7.156) [AS8075]  60.158 ms  60.535 ms ae122-0.icr02.yto20.ntwk.msn.net (104.44.20.166) [AS8075]  59.074 ms
16  * ae122-0.icr02.yto30.ntwk.msn.net (104.44.20.178) [AS8075]  59.743 ms * * 

In each of the above traces, you can see the traffic leaving Calgary at hop 4 and arriving in Seattle at hop 6, before entering Microsoft's network and ultimately going back east.

The Pilot

Last month, in collaboration with CANARIE, Cybera began a pilot to more efficiently route network traffic destined to Microsoft Azure's Canadian data centres. Instead of going west to Seattle, traffic was instead sent east to Winnipeg, then Toronto, and then optionally on to Quebec City.

Route Path from Cybera to Azure Canada East (Quebec City):

traceroute to 52.229.126.237 (52.229.126.237), 50 hops max, 60 byte packets
 1  * * * * *
 2  162.246.156.1 (162.246.156.1) [AS15296]  0.743 ms  0.826 ms  0.793 ms  0.770 ms  0.739 ms
 3  host-199.116.233.45.cybera.ca (199.116.233.45) [AS15296]  1.239 ms  1.189 ms  1.075 ms  1.082 ms  1.065 ms
 4  clgr2rtr2.canarie.ca (199.212.24.66) [AS6327/AS6509]  0.950 ms  0.923 ms  0.897 ms  0.864 ms  0.783 ms
 5  wnpg2rtr2.canarie.ca (205.189.33.199) [AS6509]  37.139 ms  36.964 ms  37.180 ms  37.165 ms  37.148 ms
 6  205.189.33.182 (205.189.33.182) [AS6509/AS53904]  37.106 ms  37.082 ms  37.231 ms  37.186 ms  37.164 ms
 7  peer-as6509.pr03.yyz1.tfbnw.net (103.4.99.3) [AS32934]  37.110 ms  37.261 ms  37.224 ms  37.071 ms  37.037 ms
 8  canarie.yto01-96cbe-1a.ntwk.msn.net (207.46.219.84) [AS8075]  36.860 ms  36.842 ms  36.823 ms  36.533 ms  36.523 ms
 9  ae21-0.icr01.yto30.ntwk.msn.net (104.44.237.155) [AS8075]  37.364 ms ae22-0.icr01.yto20.ntwk.msn.net (104.44.237.151) [AS8075]  37.415 ms  37.440 ms ae21-0.icr01.yto30.ntwk.msn.net (104.44.237.155) [AS8075]  37.550 ms ae22-0.icr01.yto20.ntwk.msn.net (104.44.237.151) [AS8075]  37.328 ms
10  be-120-0.ibr02.yto20.ntwk.msn.net (104.44.20.165) [AS8075]  47.681 ms be-100-0.ibr01.yto30.ntwk.msn.net (104.44.20.161) [AS8075]  47.449 ms  47.424 ms be-100-0.ibr01.yto20.ntwk.msn.net (104.44.20.149) [AS8075]  47.344 ms be-100-0.ibr01.yto30.ntwk.msn.net (104.44.20.161) [AS8075]  47.432 ms
11  be-5-0.ibr01.yqb20.ntwk.msn.net (104.44.28.7) [AS8075]  47.614 ms be-3-0.ibr02.yqb20.ntwk.msn.net (104.44.28.79) [AS8075]  47.364 ms be-4-0.ibr01.yqb20.ntwk.msn.net (104.44.28.100) [AS8075]  47.256 ms be-3-0.ibr02.yqb20.ntwk.msn.net (104.44.28.79) [AS8075]  47.219 ms be-5-0.ibr01.yqb20.ntwk.msn.net (104.44.28.7) [AS8075]  47.516 ms
12  ae23-0.yqb20-96cbe-1b.ntwk.msn.net (104.44.11.202) [AS8075]  47.025 ms 

Route Path from Cybera to Azure Canada Central (Toronto):

traceroute to 52.228.17.116 (52.228.17.116), 50 hops max, 60 byte packets
 1  * * * * *
 2  162.246.156.1 (162.246.156.1) [AS15296]  0.653 ms  0.771 ms  0.757 ms  0.772 ms  0.739 ms
 3  host-199.116.233.45.cybera.ca (199.116.233.45) [AS15296]  0.844 ms  1.101 ms  1.105 ms  1.094 ms  1.044 ms
 4  clgr2rtr2.canarie.ca (199.212.24.66) [AS6327/AS6509]  1.018 ms  0.731 ms  0.922 ms  0.654 ms  0.654 ms
 5  wnpg2rtr2.canarie.ca (205.189.33.199) [AS6509]  37.467 ms  37.497 ms  37.203 ms  37.168 ms  37.212 ms
 6  205.189.33.182 (205.189.33.182) [AS6509/AS53904]  37.771 ms  37.778 ms  37.510 ms  37.487 ms  37.450 ms
 7  peer-as6509.pr03.yyz1.tfbnw.net (103.4.99.3) [AS32934]  36.876 ms  37.296 ms  37.230 ms  37.215 ms  37.237 ms
 8  canarie.yto01-96cbe-1a.ntwk.msn.net (207.46.219.84) [AS8075]  36.809 ms  36.798 ms  36.538 ms  36.545 ms  36.691 ms
 9  ae22-0.icr01.yto20.ntwk.msn.net (104.44.237.151) [AS8075]  36.728 ms 

The above traces are quite different than before. Here, we can see traffic leaving Calgary at hop 4 and arriving in Winnipeg (instead of Seattle) at hop 5. In addition, you can see the round trip times have dropped from 59-68 milliseconds to 36-47 milliseconds — that's quite an improvement!

Lessons Learned

However, this wouldn't be a true pilot unless there were complications. 

For members who were leveraging Microsoft Azure prior to this pilot, it made sense for them to pick the data centre that had the best network performance. And since all their traffic was travelling to the western United States, the best data centre choices were in… the western United States.

Now that we’ve redirected all Azure traffic to go east, traffic destined for the western US data centre has to travel east before going west. We've effectively flipped the problem, and that's not great for some use cases. Fortunately, members who find themselves in this situation are able to easily opt out of our pilot and return to the original network path.

While it would be perfect if we could provide the best of both worlds — western US traffic go west, and eastern Canada traffic go east — it's unfortunately not that simple. When we route this traffic, we have to think of Azure as a single entity, rather than as a distributed public cloud with several data centres. As this pilot progresses, we will continue to investigate how to best handle this situation.

This pilot is also notable because it raises the issue of accessible data sovereignty. What conditions are needed for Canadian institutions to leverage more cloud services from Canadian-based data centres? As we look to grow the value and utility of Canada's National Research and Education Network (NREN) in the future, how can we ensure those requirements are met?

Piloting new solutions and ideas is a core value of what we do, especially when it can provide our members with better access to new technologies. Working through the difficult problems that are identified during a pilot is just another day in the office for us. We'll keep you updated as to how this pilot progresses.