[email protected] 1-222-2333-33

Category Archives: Using AWS Snowball Edge

  1. Home / 
  2. Archive by category "Using AWS Snowball Edge"
Aug 19, 2024
DNIs – Addressing Disconnected Scenarios with AWS Snow Family

DNIs were introduced to AWS Snow Family devices to support advanced network use cases. DNIs provide layer 2 network access without any translation or filtering, enabling features such as multicast streams, transitive routing, and load balancing. This direct access enhances network performance and allows for customized network configurations.

DNIs support VLAN tags, enabling network segmentation and isolation within the Snow Family device. Additionally, the MAC address can be customized for each DNI, providing further flexibility in network configuration:

Figure 4.18 – AWS Snowball Edge device with one DNI

DNIs and security groups

It’s important to note that traffic on DNIs is not protected by security groups, so additional security measures need to be implemented at the application or network level.

Snowball Edge devices support DNIs on all types of physical Ethernet ports, with each port capable of accommodating up to seven DNIs. For example, RJ45 port #1 can have seven DNIs, with four DNIs mapped to one EC2 instance and three DNIs mapped to another instance. RJ45 port #2 could simultaneously accommodate an additional seven DNIs for other EC2 instances.

Note that the Storage Optimized variant of AWS Snowball Edge does not support DNIs:

Figure 4.19 – AWS Snowball Edge network flows with DNIs

Looking at Figure 4.19, we can see that al2-1 has two Ethernet ports configured inside Linux. One is on the typical 34.223.14.128/25 subnet, but the other is directly on the 192.168.100.0/24 RFC 1918 space. A configuration such as this is the only time an interface on an EC2 instance on an AWS Snow Family device should be configured for any subnet but 34.223.14.128/25.

Figure 4.20 shows what a DNI looks like from the perspective of the EC2 instance that has one attached:

Figure 4.20 – DNI details under Amazon Linux 2

Storage allocation

All AWS Snowball Edge device variants work the same way with respect to storage allocation. Object or file storage can draw from the device’s HDD storage capacity, while block volumes used by EC2 instances can be drawn from either the device’s HDD or SDD capacity. Figure 4.21 shows an example of this:

Figure 4.21 – Storage allocation on AWS Snowball Edge

S3 buckets on a device can be thought of as being thin-provisioned in the sense that they start out consuming 0 bytes, and as objects are added, they only take the amount needed for those objects from the HDD capacity.

Block volumes for EC2 instances, on the other hand, can be thought of as thick-provisioned. When the volume is created, a capacity is specified, and it is immediately removed from the HDD pool for any other use.

More Details
Apr 29, 2024
Other considerations – Addressing Disconnected Scenarios with AWS Snow Family

Let us assume the following conditions for a migration using an AWS Snowball Edge device:

A SAN array has two servers as clients

Each server utilizes two Logical Unit Numbers (LUNs) on the SAN

One server runs Windows Server 2019

One server runs Red Hat Enterprise Linux 8 (RHEL 8)

The Windows server exposes its data for copying through a CIFS share

The Linux server exposes its data using an NFS export

The desktop is going to act as a data mover for the AWS Snowball Edge device

Figure 4.9 – Hypothetical data movement paths

Looking at Figure 4.9, we can see several places where the throughput could get slowed down:

The disk groups/pools on the SAN array

The controllers/I/O ports on the SAN array

The Fibrechannel fabric connecting the SAN array to the servers

The hardware configuration of either server

The OS and file-serving configuration of either server

Whether either server is dedicated to this task or is running other apps

Differences in the CIFS and NFS protocols or their versions

The network between the servers and the desktop

Hardware and software configuration of the desktop

An even worse possibility is that the servers and the desktop can pull the data from the SAN at maximum speed of all devices and links involved, only to discover this causes the SAN controller to queue I/O requests for a third client you weren’t aware of.

It turns out this third server is running a large Microsoft SQL Server that consumes LUNs from the same disk pool on the SAN array, and it also shares the same pair of SAN controllers on the frontend. The 10 Gbit/s of sequential reads causes head thrashing on the disk pool and overruns the shared cache on the controllers.

As a result, the mission-critical application that depends on this database suffers performance degradation – or worse, an outage. Anyone who has overseen many data center migrations – to the cloud or otherwise – has probably witnessed such a situation. Figuring out how fast you can possibly move data onto a device is important, but it is even more important to determine the maximum non-impactive speed for the source of the data.

More Details
Dec 20, 2023
Using AWS Snowball Edge – Addressing Disconnected Scenarios with AWS Snow Family

There is no longer a division between AWS Snowball and AWS Snowball Edge. Now, all such devices fall under the AWS Snowball Edge line, even if their intended use case is a straightforward data migration to S3.

There are four configurations with which an AWS Snowball Edge device can be ordered (see Figure 4.1):

   Storage Optimized w/80 TBCompute Optimized Type 1Compute Optimized Type 2 1 1 At the time of writing, this variant is limited to US-based regions onlyCompute Optimized w/GPU
HDD in TB8039.539.539.5
SSD in TB17.6807.68
NVME in TB00280
VCPUs245210452
VRAM in GB80208416208
GPU typeNoneNoneNoneNVIDIA V100
10 Gbit RJ451222
25 Gbit SFP1111
100 Gbit QSFP1111
Volume (in3)5381538153815381
Weight (lbs)47474747
Power draw (avg)304 w304 w304 w304 w
Power draw (max)1200 w1200 w1200 w1200 w
Voltage range100-240 v100-240 v100-240 v100-240 v

Table 4.1 – Comparison of AWS Snowball Edge variants

The AWS Snowball Edge Storage Optimized variant is now used for data migrations in place of the old AWS Snowball. There is a local S3 endpoint to which files can be directly copied using AWS OpsHub, the AWS Command Line Interface (AWS CLI), or direct API commands from a script.

The local compute capacity can be used to host an AWS DataSync instance, an AWS Tape Gateway instance, an AWS File Gateway instance, or another instance that provides a different type of loading interface of your choosing.

Migrating data to the cloud

Table 4.2 illustrates how long migrations of varying sizes would take depending upon the network throughput:

   50 Mbps100 Mbps1 Gbps2 Gbps5 Gbps10 Gbps25 Gbps40 Gbps100 Gbps
50 Terabytes3.3 months1.7 months5 days2.5 days1 day12 hours5 hours3 hours1 hour
500 Terabytes2.8 years1.4 years1.7 months25 days10 days5 days2 days1.25 days12 hours
5 Petabytes28.5 years14.3 years1.4 years8.5 months3.4 months1.7 months20 days12 days5 days
10 Petabytes57 years28.5 years2.8 years1.4 years6.8 months3.4 months1.3 months24 days10 days

Table 4.2 – Comparison of migration times

Many organizations don’t have high-throughput internet connections that could be fully dedicated to migration. Nor do they have access to/familiarity with the techniques needed to fully utilize said connection once the latency gets above a few milliseconds.

This is why loading one or more devices connected to a local network and physically shipping to AWS is so popular – despite the days on either end the devices spend on a truck:

Figure 4.2 – An AWS Snowball Edge device being loaded with data

More Details
Sep 1, 2023
GEOMETRIC DILUTION OF PRECISION (GDOP) – Understanding Network and Security for Far-Edge Computing

GDOP is a calculated value that combines the impact of several factors related to the angle at which the ground station can reach the satellites into a single coefficient that expresses how accurate a calculated position is.

Referring back to the previous figure, we can see an example of good geometry of the satellites involved. They are spread across the sky in all three axes. Contrast that with the following situation. In this case, the user is in an area surrounded by mountains. The terminal has no choice but to use samples from satellites that are closer together in the sky, and the calculated position will be less accurate as a result:

Figure 3.43 – Poor geometry due to obstructions

Other sources of GNSS inaccuracy

Atmospheric refraction is when a satellite’s signal is bent a little while traveling through the upper layers of the atmosphere. Sunspot activity can cause interference. Lower-quality receivers are more susceptible to measurement noise, which can happen even under perfect environmental conditions. A clock error of 1 nanosecond (a billionth of a second) can introduce as much as half a meter (1.5 feet) of imprecision.

Urban environments pose a particular challenge to GNSSs. Not only is the geometry compromised by buildings, but the signals the user can receive are often reflected off of them – causing unwanted multipath propagation as previously discussed. If you’ve ever requested a ride from an app on your phone and wondered why the driver thinks you’re at a restaurant two streets away, these are likely culprits.

Global Positioning System (GPS)

The first satellite for what we now know as GPS was launched in 1978 by the United States Air Force. At first, only the US military had access to the system.

In 1983, pilots of a commercial flight from Alaska to Korea made a navigational error that took their aircraft over the Kamchatka Peninsula near Japan. In response, a Soviet SU-15 interceptor shot down the Boeing 747, killing all 269 civilians onboard. To prevent future incidents, the US opened GPS for civilian use.

As of 2020, GPS is operated by the United States Space Force and remains open for anyone to use. At the time of writing, it has 32 satellites in a semi-synchronous21 medium Earth orbit (MEO) with an altitude of 20,200 kilometers (12,600 miles). Each orbit has a different inclination, providing global ground coverage.

21 A semi-synchronous orbit is one in which the spacecraft passes over a given point on the Earth twice per day.

More Details
Jul 26, 2023
Satellite orbits – Understanding Network and Security for Far-Edge Computing

Geostationary orbit (GEO)

GEO satellites are positioned in orbit around the Earth at an altitude of about 35,786 kilometers (22,236 miles). They are designed to remain in a fixed location relative to a point on the Earth’s surface as they orbit the Earth at the same rate that the Earth rotates.

This makes things easy for ground-based users. There are mobile apps that will tell you exactly where in the sky to point your antenna, and then you’re done:

Figure 3.41 – GEO satellite distance

The downside is the high latency incurred when signals have to travel that far. The speed of light is fast, but it is finite. ~200 milliseconds are required for light to go from one spot on the earth up to the GEO satellite and another 200 to go down to another spot. Factor in the latency of any ground segment and a 600ms RTT is considered typical.

Here are some typical GEO-based SATCOM data services:

Broadband Global Area Network (BGAN): This is an L-band service from Inmarsat. It can achieve speeds up to 492kbps for standard IP data traffic and up to 800kbps for streaming data (usually video), although this depends heavily upon the terminal involved. Six geostationary satellites are involved in providing global coverage (including polar regions) for this service. It is extremely reliable, supporting a 99.9% uptime SLA.

Global Xpress (GX): This is a Ka-band service from Inmarsat. It can achieve download speeds up to 50mbps and 5mbps speeds for upload. Five geostationary satellites provide near-global coverage.

European Aviation Network (EAN): This is a hybrid service comprised of a single Inmarsat S-band satellite in geostationary orbit above Europe and Vodafone’s terrestrial 4G/LTE network. Specifically built to provide data services onboard aircraft in European airspace, data rates as high as 100mbps are supported. Aircraft use the terrestrial network below 10,000 feet and switch to the S-band service above this altitude.

ViaSat-3: This is a Ka-band service that uses a constellation of three geostationary satellites operated by ViaSat. Each satellite serves a specific region (AMER, EMEA, or APAC), and has a total network capacity greater than 1 terabit per second. Typical consumer plans are 100mbps, while contracts for defense and commercial entities can be higher.

GEO HTS: This is a Ku-band service from SES that can achieve speeds up to 10mbps. It has near-global coverage using four satellites in geostationary orbit.

FlexGround: This is a Ku-band service from Intelsat that supports download speeds up to 10mbps and 3mbps upload speeds. Being one of the pioneers in SATCOM19, Intelsat has over 50 satellites in geostationary orbit.

19 Intelsat launched its first satellite in 1965.

More Details
Apr 2, 2023
LoRaWAN device classes – Understanding Network and Security for Far-Edge Computing

One of the primary design parameters for LoRaWAN devices is low power consumption. LoRaWAN devices don’t leverage any special battery technology. Some of them use simple AAA or AA batteries you can purchase at the supermarket. Rather, it’s because they try to spend as much time as possible doing as little as possible.

LoRaWAN device batteries are measured in terms of milliamp-hours (mAh), just the same as a power bank you might use to recharge your mobile phone. In the LoRaWAN specification, end devices/nodes can operate in three different modes: Class A, Class B, and Class C.

All end devices support Class A [14]. These spend most of their time in sleep mode. Because LoRaWAN is not a scheduled protocol, end devices can communicate any time there is a change in a sensor reading or when a local timer on the device goes off:

Figure 3.37 – Class A LoRaWAN temperature and humidity sensor

These devices can wake up and talk to the server at any random moment. After the device sends an uplink, it listens for a message from the network one and two seconds after the uplink (receive windows) before going back to sleep. Class A is the most energy efficient and results in the longest battery life. A 5,000mAh power bank for your phone could keep the average class A device running for 30 years 17.

17 Do not attempt this – it is likely such a power bank would self-discharge long before 30 years..

Examples of Class A devices include LoRaWAN-enabled pushbuttons that transmit alarm information in case of an emergency. There are such buttons on the market with a 600mAh capacity that can sustain 70,000 pushes of the button (and associated message transmission).

Class B devices are designed for use in applications where the device needs to transmit data more frequently, but still has relatively low power requirements. They are allowed to transmit data at regular intervals, and they listen for a response from the network after each transmission. This allows them to transmit data more frequently than Class A devices, and the part where they listen for a response ensures more reliability, but they still have a low power consumption:

Figure 3.38 – Class B LoRaWAN barometric pressure sensor

Devices in this class might include a smart meter that needs to reliably collect the kilowatt-hour utilization of a power circuit at regular intervals or an environmental sensor that needs to be sure it collects a windspeed sample at prescribed intervals for the dataset to be valid.

Class C devices are used in applications where the device needs to transmit data continuously. They are allowed to transmit data at any time and are always listening for a response from the network. They never go to sleep. This makes them the least power-efficient of the three classes:

Figure 3.39 – Class C LoRaWAN manhole sensor

An example might be a sensor in a manufacturing plant that ensures something dangerous remains within a specific temperature range. Another might be a device that’s used for real-time asset tracking, where we want to be actively alerted the moment something leaves the area it is supposed to be in.

More Details
Nov 26, 2022
WiFi and MIMO – Understanding Network and Security for Far-Edge Computing

As discussed previously, MIMO is a method for increasing effective throughput by deliberately exploiting multipath propagation. The different generations of WiFi make use of this in varying ways.

802.11n (Wi-Fi-4)

This supported the more limited Single User MIMO (SU-MIMO). As its name suggests, SU-MIMO means the access point can only be sent to one client at a time.

802.11ac (Wi-Fi-5)

This added MU-MIMO (d). The (d) stands for downlink. With MU-MIMO (d), only one station can transmit, but multiple stations can receive at any given time.

802.11ax (Wi-Fi-6)

This was extended to MU-MIMO (u/d). Now, multiple devices can both transmit and receive simultaneously.

MU-OFDMA

Basic OFDM has been supported since 802.11a (Wi-Fi-2). 802.11ax (Wi-Fi-6) has extended this to now support multiple users.

You could think of the older style of OFDM as a sequence of trucks, each delivering boxes from one vendor at a set time every day. MU-OFDMA allows each truck to be loaded with multiple vendor’s boxes. It also allows the delivery schedule of those trucks to happen only when there’s a full load.

Older Wi-Fi specifications were designed for web browsing and checking email. Congestion emerged as video streaming, AR/VR, and gaming became common. This, combined with more and more client devices transmitting at the same time, meant that the queuing caused by simple OFDM increased latency.

Perhaps most importantly, MU-OFDMA allows priorities to be set not only per client but per protocol/traffic type. In other words, the access point could prioritize video streaming at one level, IoT messages at another, and mission-critical VOIP at the highest.

802.11p (DSRC)

An amendment to the broader IEEE 802.11 Wireless LAN (WLAN) standard, 802.11p is tailored for high-speed, short-range communication in a vehicular environment. The standard operates in the 5.9 GHz frequency band and utilizes the Dedicated Short-Range Communications (DSRC) protocol to ensure low latency and reliable data exchange.

The primary advantage of DSRC over 4G/LTE or 5G for V2X is that it can provide some value in the absence of any infrastructure. If two V2X-equipped cars come within range of each other, they will exchange information in a peer-to-peer fashion. This would function even in the middle of the Sahara.

In 2016, Toyota became the first automaker to introduce cars equipped with V2X systems, followed by GM in 2017. Both of these used DSRC as opposed to 4G/LTE or 5G. While DSRC was the first standard the automotive industry adopted, that is changing for several reasons. Compared to 4G/LTE or 5G for V2X, DSRC suffers from the following limitations:

Limited capacity and scalability: DSRC operates in a narrow frequency band (5.9GHz), which limits its capacity to support a high number of simultaneous connections in dense traffic scenarios. 5G offers broader bandwidth and improved spectral efficiency, allowing it to handle more devices and users concurrently.

Lower data rates: DSRC offers lower data rates compared to 5G, which hinders its ability to support advanced V2X applications that require higher throughput, such as high-definition video streaming for autonomous vehicles. 5G, with its enhanced data rates, can better accommodate these demanding use cases.

Latency: Although DSRC provides relatively low latency communication, 5G has the potential to achieve even lower latencies, especially with the implementation of 5G Ultra-Reliable Low-Latency Communication (URLLC). URLLC can enable mission-critical applications and real-time control systems that demand near-instantaneous response times.

Network slicing: 5G supports network slicing, a feature that allows the creation of virtual networks tailored to specific use cases or applications. This enables the allocation of dedicated resources for V2X communications, ensuring the desired performance levels. DSRC, on the other hand, does not offer this level of customization and flexibility.

Global harmonization: While DSRC has been adopted in some regions, it has not achieved global harmonization, leading to inconsistencies in spectrum allocation and regulation across different countries. 5G has a more unified approach, with global standardization and broader adoption, making it more attractive for V2X implementations across various regions.

Keeping all of this in mind, automakers have begun to include both in their chipsets. The idea is that cellular networks are the primary communication path, and when those are not available, the chipset will leverage DSRC for peer-to-peer vehicle communication when and where it can.

More Details
Oct 8, 2022
Spatial streams – Understanding Network and Security for Far-Edge Computing

The term for beamforming as it is implemented within Wi-Fi is spatial streams.

While some vendors of 802.11n (Wi-Fi-4) devices did implement beamforming, it was through proprietary mechanisms that were specific to each product line. 802.11ac (Wi-Fi-5) was the first to include it as part of the specification.

When a Wi-Fi access point has beamforming enabled, it first estimates the angle of arrival of each client by comparing small differences in arrival times of a signal across multiple antennas that are close together. Once it knows the direction in which it needs to steer the beam, it will have those antennas broadcast the signal at slightly different times. The pattern that’s used is known as a steering matrix.

This deliberately introduces interference because the waves now overlap a little bit. However, not all interference is the same. Some are constructive interference, which makes the signal stronger in one direction, while destructive interference makes it weaker in another:

Figure 3.32 – Beamforming with 802.11ac (Wi-Fi-5)

The net effect of all this is to maximize the signal strength on a per-client basis. This means the signal effectively travels farther and penetrates obstacles better. With older Wi-Fi specifications, all you could do is increase the power output of an omnidirectional signal or add Wi-Fi repeaters.

This is one of the reasons for a seemingly endless multiplication of antennas on even consumer-grade access points. More antennas on both the AP and the clients are better for Wi-Fi throughput – up to a point13. Regardless of the number of antennas, the 802.11ac (Wi-Fi-5) specification supports a maximum of four spatial streams to be active at once.

13 Two antennas are the minimum for beamforming to function at all, while three is recommended.

802.11ax (Wi-Fi-6) increased this to eight and also enhanced it by including client-side modifications that help the AP figure out where a given client is instead of leaving all the work on the AP.

More Details
Aug 22, 2022
Modulation and coding schemes (MSCs) – Understanding Network and Security for Far-Edge Computing

The speeds provided in the preceding table are best-case scenarios. They assume an optimal SNR, which, in turn, allows the use of a modulation and encoding scheme that gets a higher data rate. Each generation of Wi-Fi has a different matrix of MCSs. The following is the MCS index table for 802.11ac (Wi-Fi-5):

   ModulationFEC Coding RateData Rate
MCS0BPSK1/2 
MCS1QPSK1/22x faster than MCS0
MCS2QPSK3/43x faster than MCS0
MCS316-QAM1/24x faster than MCS0
MCS416-QAM3/46x faster than MCS0
MCS564-QAM2/38x faster than MCS0
MCS664-QAM3/49x faster than MCS0
MCS764-QAM5/610x faster than MCS0
MCS8256-QAM3/412x faster than MCS0
MCS9256-QAM5/613.3x faster than MCS0

Figure 3.30 – 802.11ac modulation and coding schemes

Each of the MCSs shown has two parameters:

Modulation: In this context, modulation refers to the particular 802.11x modulation type in use. Some modulation types are very sensitive to noise while others tolerate it well. However, the robustness of a modulation type is achieved by reducing how sensitive it is – and this means a lower bit rate.

FEC coding rate: This describes how many bits transfer data, and how many are used for forward error correction. A coding rate of 5/6 means for every 5 bits of useful information, the coder sends 6 bits of data. In other words, there’s one error bit for every 5 data bits:

Figure 3.31 – Impact of MCS on data rate for 802.11ac

A Wi-Fi-5 or Wi-Fi-6 access point will negotiate the best MCS that it can, given the interference it is experiencing. Wi-Fi devices tend to express the SNR as a single number in dB, which represents the amount of signal above whatever noise is present.

A laptop 1 meter away from an access point with no obstructions would have an SNR of ~50 dB, and be able to operate at MCS9 (100% max speed). A second laptop far away or in a different room might only see an SNR of ~25 dB and be stuck at MCS3 (30% max speed).

Here are some practical steps that can help your device negotiate a faster MCS to its access point:

Reduce devices per AP: Try to have only 3-4 devices per AP where possible

Change Wi-Fi channels: Utilities such as NetSpot can help with this

Increase AP signal power: Some APs default to a lower power level than they are legally able to use

More Details
Jul 24, 2022
Optimizing Wi-Fi (802.11x)-based connectivity – Understanding Network and Security for Far-Edge Computing

Wi-Fi was designed to allow laptops, smartphones, and tablets to connect to the internet and/or communicate with each other on a local area network (LAN). It uses RF to transmit data over relatively short distances, typically within a home or office – although permutations intended for outdoor use are becoming more common.

Wi-Fi is based on the IEEE 802.11 standards, which operate at Layer 1 of the OSI model (physical). Introduced in the late 1990s, it was the first commercially successful wireless networking technology that was designed to work seamlessly with Ethernet (IEEE 802.3) – which almost all LANs use at Layer 2.

Wi-Fi-1 through Wi-Fi-6

The following table shows us the comparison of 802.11a/b/g/n/ac/ax:

   802.11 (b) Wi-Fi-1802.11 (a) Wi-Fi-2802.11 (g) Wi-Fi-3802.11 (n) Wi-Fi-4802.11 (ac) Wi-Fi-5802.11 (ax) Wi-Fi-6
Max Speed11 Mbps54 Mbps54 Mbps600 Mbps10 10 Requires the use of vendor-specific proprietary beamforming/spatial streams.1.3 Gbps11 11 Refers to per-station throughput. The whole network theoretical maximum is 6.9 Gbps.1.7 Gbps12 12 Refers to per-station throughput. The whole network theoretical maximum is 9.6 Gbps.
Range Indoor (2.4)35 mN/A45 m60 mN/A60 m
Range Indoor (5)N/A30 m30 m45 m45 m45 m
Range Outdoor (2.4)70 mN/A90 m120 mN/A120 m
Range Outdoor (5)N/A60 m75 m90 m90 m90 m
2.4 GHz BandYesNoYesYesNoYes
5 GHz BandNoYesYesYesYesYes
OFDMNoYesYesYesYesYes
MU-OFDMANoNoNoNoNoYes
SU-MIMONoNoNoYesYes8×8
MU-MIMO (d)NoNoNoNo4×48×8
MU-MIMO (u/d)NoNoNoNoNo8×8
Spatial StreamsNoNoNoNo48

Figure 3.29 – Comparison of 802.11a/b/g/n/ac/ax

More Details