Aug 19, 2024
DNIs – Addressing Disconnected Scenarios with AWS Snow Family

DNIs were introduced to AWS Snow Family devices to support advanced network use cases. DNIs provide layer 2 network access without any translation or filtering, enabling features such as multicast streams, transitive routing, and load balancing. This direct access enhances network performance and allows for customized network configurations.

DNIs support VLAN tags, enabling network segmentation and isolation within the Snow Family device. Additionally, the MAC address can be customized for each DNI, providing further flexibility in network configuration:

Figure 4.18 – AWS Snowball Edge device with one DNI

DNIs and security groups

It’s important to note that traffic on DNIs is not protected by security groups, so additional security measures need to be implemented at the application or network level.

Snowball Edge devices support DNIs on all types of physical Ethernet ports, with each port capable of accommodating up to seven DNIs. For example, RJ45 port #1 can have seven DNIs, with four DNIs mapped to one EC2 instance and three DNIs mapped to another instance. RJ45 port #2 could simultaneously accommodate an additional seven DNIs for other EC2 instances.

Note that the Storage Optimized variant of AWS Snowball Edge does not support DNIs:

Figure 4.19 – AWS Snowball Edge network flows with DNIs

Looking at Figure 4.19, we can see that al2-1 has two Ethernet ports configured inside Linux. One is on the typical 34.223.14.128/25 subnet, but the other is directly on the 192.168.100.0/24 RFC 1918 space. A configuration such as this is the only time an interface on an EC2 instance on an AWS Snow Family device should be configured for any subnet but 34.223.14.128/25.

Figure 4.20 shows what a DNI looks like from the perspective of the EC2 instance that has one attached:

Figure 4.20 – DNI details under Amazon Linux 2

Storage allocation

All AWS Snowball Edge device variants work the same way with respect to storage allocation. Object or file storage can draw from the device’s HDD storage capacity, while block volumes used by EC2 instances can be drawn from either the device’s HDD or SDD capacity. Figure 4.21 shows an example of this:

Figure 4.21 – Storage allocation on AWS Snowball Edge

S3 buckets on a device can be thought of as being thin-provisioned in the sense that they start out consuming 0 bytes, and as objects are added, they only take the amount needed for those objects from the HDD capacity.

Block volumes for EC2 instances, on the other hand, can be thought of as thick-provisioned. When the volume is created, a capacity is specified, and it is immediately removed from the HDD pool for any other use.

More Details
Mar 31, 2024
Client-side mechanisms for loading data onto AWS Snowball Edge – Addressing Disconnected Scenarios with AWS Snow Family

With the exception of AWS DataSync, file loading is a push operation from your data loader workstation. Thus, you will need to use an appropriate client application to communicate with the target you have selected.

Performance tip – batching

Regardless of the client-side mechanism you use to copy data, there is a certain amount of per-file overhead incurred by operations such as encryption. This is why copying a thousand 1 KB files is slower than copying one 1,000 KB file. If the data you are loading consists of many small files spread across many subdirectories, you will probably save time by batching them up into one large archive with utilities such as zip, gzip, or tar. This is true even if you obtain zero compression by doing so.

AWS OpsHub for Snow Family

The simplest thing to do is use the drag-and-drop interface in the AWS OpsHub application. Customers who prefer a GUI interface download and use this anyway to unlock the device and make configuration changes to it. This option also requires no special target configuration.

While it might be convenient, as you can see from Figure 4.8, it is also quite slow with a maximum speed of around 0.3 Gbps:

Figure 4.8 – Uploading files via drag and drop in AWS OpsHub

NFS client

When using the NFS endpoint, your data loader workstation must have an NFS client installed. This is usually installed by default on macOS or Linux. While Windows does offer an NFS client, it is not installed by default, and the performance tends to be lower.

AWS CLI

The AWS CLI should be installed anyway on your data loader workstation. It can be used to target the locally running S3 endpoint on the AWS Snowball Edge device. Using the aws s3 sync command, you can do bulk data transfer operations the same way you would with an S3 bucket in an AWS region.

s5cmd

The AWS CLI is a general-purpose utility written in Python. It wasn’t explicitly designed to maximize file transfer speed. This means it can’t usually push as fast as the S3 endpoint can receive. Fortunately, s5cmd can. It is an open source project available on GitHub. It is written in Go and focuses on maximum parallelization. The more CPU cores your data loader has, the faster it can move data. However, given most laptops or even desktops don’t have 128 cores and 25 Gbps interfaces, this option tends to be used when the loader itself is a server in the customer’s data center.

More Details
Nov 19, 2023
Introduction to the AWS Snow Family – Addressing Disconnected Scenarios with AWS Snow Family

In today’s interconnected world, reliable connectivity is often taken for granted. However, there are numerous scenarios where maintaining a consistent network connection is a challenge, such as remote locations, disaster-stricken areas, or environments with limited or intermittent network access. In these disconnected scenarios, organizations require a solution that can ensure data availability, enable efficient data processing, and one that will support critical operations. This is where the AWS Snow Family comes into play, providing a range of robust and versatile solutions designed specifically to address the unique requirements of disconnected environments.

In this chapter, we will explore how the AWS Snow Family empowers organizations to overcome the limitations of disconnected scenarios and seamlessly bridge the gap between on-premises infrastructure and the cloud. We will delve into the features and capabilities of AWS Snow Family offerings and discuss their use cases, benefits, and considerations. Whether it’s securely transferring large amounts of data, performing on-site data processing and analysis, or extending cloud services to the edge, the AWS Snow Family offers reliable, scalable, and cost-effective solutions that cater to the needs of disconnected environments. Join us as we discover the power of AWS Snow to enable data-driven decision-making and unlock new possibilities in disconnected scenarios.

Here are the main headings:

Introduction to the AWS Snow Family

Using AWS Snowball Edge

Using AWS Snowcone

Introduction to the AWS Snow Family

The original AWS Snowball service was introduced in 2015. It started out as a mechanism to move large amounts of data when doing so over the network wasn’t reasonable. In the ensuing years, customer demand for new capabilities has driven the expansion of this line into different variants with use-case-specific capabilities:

Figure 4.1 – AWS Snow Family devices

All offer an interface and operating model that is consistent with Amazon EC2 and Amazon S3, and they are all designed to run autonomously. All AWS Snow Family devices operate their own local control, management, and data planes. Thus, they do not require a consistent network connection back to the AWS cloud to operate.

AWS Snow Family devices can all host local object storage buckets that utilize the same API/CLI interface as Amazon S3 buckets. When a customer orders one, it is sent to them, they copy their data to these local buckets, and then they ship the unit back to AWS. This is facilitated by an e-ink display on the unit that eliminates the need to pack it in a box or obtain a shipping label separately. When the device is received by AWS, the data is uploaded to the relevant “real version” of the Amazon S3 bucket in question.

Additionally, AWS Snow Family devices do not have the same restrictive environmental requirements as most off-the-shelf compute and storage hardware. AWS Snow Family devices are found operating in a wide variety of field situations that would be impractical with standard off-the-shelf servers. First responders heading to the site of a disaster can even check them in as luggage.

More Details
Aug 7, 2023
LOW-EARTH ORBIT (LEO) – Understanding Network and Security for Far-Edge Computing

LEO satellites are positioned in orbit around the Earth at an altitude of up to 2,000 kilometers (1,200 miles). Because of this, they are in constant motion relative to an observer.

LEO satellites are known for their ability to provide coverage over a large area of the Earth’s surface since they orbit the Earth relatively quickly (compared to GEO satellites). This allows them to provide communication and other services to a large number of users, as well as to track the movement of objects on the surface of the Earth.

The primary technical advantage of LEO-based SATCOM systems is their much lower latency than GEO (as low as ~20ms RTT). The main disadvantage is caused by the fact that they are in constant motion concerning any given point on the ground. They must use mechanisms such as motorized tracking antennas (or complex phased-array antennas) and constellations of a sufficient size to ensure users on the ground can always reach at least one satellite.

Here are some examples of LEO-based SATCOM services:

Certus 700: An L-band service from Iridium that supports speeds as high as 704 Kbps. It is served by 66 cross-linked satellites in LEO.

Starlink Roam: A Ka/Ku-band service from Starlink that supports speeds up to 200 Mbps. It is served by over 3,50020 cross-linked satellites in LEO, with plans to grow to as many as 12,000.

20 As of February, 2023.

Global Navigation Satellite System (GNSS)

GNSS is an overarching term that includes all of the systems that use timing signals from satellite constellations to determine a position on the ground for navigation purposes.

GNSS for positioning

Trilateration

All satellite-based navigation systems discussed in this section determine a terminal’s position using trilateration. Unlike triangulation, it measures distance – not angles. Satellites in these systems repeatedly broadcast their current position and local time, derived from multiple onboard atomic clocks.

The following figure demonstrates a point on the ground receiving the same broadcast from four satellites:

Figure 3.42 – Trilateration using four satellites

From these four pieces of data, a terminal can calculate its position within a margin of error that varies from centimeters to hundreds of meters, depending on the circumstances.

More Details
May 15, 2023
Integrating SATCOM – Understanding Network and Security for Far-Edge Computing

Satellite communication (SATCOM)

SATCOM is the use of satellites to provide communication services, such as telephone, television, and internet connectivity. SATCOM systems use a network of satellites in orbit around the Earth to transmit and receive signals between two or more points on the surface of the Earth, or between the Earth and another body in space (such as a spacecraft).

There are two main types of SATCOM systems: fixed and mobile. Fixed SATCOM systems are typically used to provide communication services to a specific location, such as a remote village or a ship at sea. Mobile SATCOM systems are designed to provide communication services to mobile users, such as aircraft, vehicles, or portable devices.

SATCOM systems are used in a wide range of applications, including military and government communications, emergency and disaster response, and commercial telecommunications. They are particularly useful in areas where it is difficult or impossible to install terrestrial communication infrastructure, such as in remote or inaccessible locations, or disaster-stricken areas.

SATCOM terminal18

18 Some SATCOM operators refer to terminals as antennas or modems, which is technically inaccurate as a terminal is the overall system the end user needs to connect to.

In the context of satellite communications, a terminal is the user equipment that acts as an interface between the user’s network and the satellite constellation. SATCOM terminals vary in cost, size, and complexity, ranging from small handheld devices to larger installations used in industries such as aviation, rail, maritime, and the military. Terminals typically consist of antennas, transceivers, modems, and associated electronics that facilitate satellite communication for voice, data, video, or other forms of communication.

SATCOM frequency bands

For the most part, SATCOM takes place within the SHF or VHF bands, as defined by the ITU. However, SATCOM has its own frequency band definitions, which are more granular:

  Band StartFrequency (GHz)Wavelengthn
StopStartStop 
Classical L-Band0.9501.450316207
Extended L-Band0.9502.150316140
S-band1.7003.000176100
Extended C-BandDownlink3.4004.2008871
Uplink5.8506.7255145
LMI C-BandDownlink3.7004.0008175
Uplink5.7256.0255250
Russian C-BandDownlink3.6504.1508272
Uplink5.9506.4755046
Standard C-BandDownlink3.7004.2008171
Uplink5.9256.4255147
X-BandDownlink7.2507.7504139
Uplink7.9008.4003836
Ku-BandDownlink10.00013.0003023
Uplink14.00017.0002118
K-Band18.00026.5001711
Ka-BandnDownlink18.00021.0001714
Uplink27.00031.0001110

Figure 3.40 – SATCOM frequency bands

More Details
Apr 2, 2023
LoRaWAN device classes – Understanding Network and Security for Far-Edge Computing

One of the primary design parameters for LoRaWAN devices is low power consumption. LoRaWAN devices don’t leverage any special battery technology. Some of them use simple AAA or AA batteries you can purchase at the supermarket. Rather, it’s because they try to spend as much time as possible doing as little as possible.

LoRaWAN device batteries are measured in terms of milliamp-hours (mAh), just the same as a power bank you might use to recharge your mobile phone. In the LoRaWAN specification, end devices/nodes can operate in three different modes: Class A, Class B, and Class C.

All end devices support Class A [14]. These spend most of their time in sleep mode. Because LoRaWAN is not a scheduled protocol, end devices can communicate any time there is a change in a sensor reading or when a local timer on the device goes off:

Figure 3.37 – Class A LoRaWAN temperature and humidity sensor

These devices can wake up and talk to the server at any random moment. After the device sends an uplink, it listens for a message from the network one and two seconds after the uplink (receive windows) before going back to sleep. Class A is the most energy efficient and results in the longest battery life. A 5,000mAh power bank for your phone could keep the average class A device running for 30 years 17.

17 Do not attempt this – it is likely such a power bank would self-discharge long before 30 years..

Examples of Class A devices include LoRaWAN-enabled pushbuttons that transmit alarm information in case of an emergency. There are such buttons on the market with a 600mAh capacity that can sustain 70,000 pushes of the button (and associated message transmission).

Class B devices are designed for use in applications where the device needs to transmit data more frequently, but still has relatively low power requirements. They are allowed to transmit data at regular intervals, and they listen for a response from the network after each transmission. This allows them to transmit data more frequently than Class A devices, and the part where they listen for a response ensures more reliability, but they still have a low power consumption:

Figure 3.38 – Class B LoRaWAN barometric pressure sensor

Devices in this class might include a smart meter that needs to reliably collect the kilowatt-hour utilization of a power circuit at regular intervals or an environmental sensor that needs to be sure it collects a windspeed sample at prescribed intervals for the dataset to be valid.

Class C devices are used in applications where the device needs to transmit data continuously. They are allowed to transmit data at any time and are always listening for a response from the network. They never go to sleep. This makes them the least power-efficient of the three classes:

Figure 3.39 – Class C LoRaWAN manhole sensor

An example might be a sensor in a manufacturing plant that ensures something dangerous remains within a specific temperature range. Another might be a device that’s used for real-time asset tracking, where we want to be actively alerted the moment something leaves the area it is supposed to be in.

More Details
Mar 13, 2023
LoRaWAN network topology – Understanding Network and Security for Far-Edge Computing

Notice that, unlike Wi-Fi, LoRaWAN inserts gateways as intermediaries between the devices and the network. While some large enterprise Wi-Fi networks have a similar topology, it is something manufacturers bolted on later for scale-related reasons and is not part of the original 802.11x specification.

With Wi-Fi, a device is only ever associated with one access point at a time, and when they move around, their session is cut over between them. LoRaWAN, on the other hand, sends its traffic to all of the gateways it can see simultaneously. If the server needs to send a message back to the device, it will choose the best gateway to use for that purpose:

Figure 3.36 – LoRaWAN network topology

This architecture is known as star-on-star. It yields advantages that are relevant to typical LoRaWAN use cases:

Redundancy: If a gateway fails or needs to be taken offline for maintenance, devices in the network are not affected.

Affordability: Because this form of redundancy is a fundamental part of the LoRaWAN specification, it is cheaper to implement both in terms of hardware and deployment effort.

Scalability: The number of gateways a network server can manage is limited only by the processing power of that server. When that is exhausted, additional servers can be added to scale the system horizontally. There are LoRaWAN networks with 40,000 gateways that support many millions of devices 15.

15 https://www.thethingsnetwork.org/map

Direct communication between devices

The LoRaWAN protocol does not support direct communication between end nodes. This can be confusing because LoRaWAN-capable devices exist that communicate without involving the gateways. However, this is done using a different protocol such as RadioHead16 or something proprietary to that manufacturer.

16 https://www.airspayce.com/mikem/arduino/RadioHead/

Geolocation

All battery-powered LoRaWAN devices such as tags or sensors can move while they are communicating without increasing the power budget. Additionally, it is not always practical to track physical coordinates when deploying stationary devices.

Fortunately, the LoRaWAN protocol provides two inbuilt methods for determining the position of devices. Nothing needs to be added to existing LoRaWAN-capable endpoints for these to work, making it a lower-cost alternative to adding GNSS to all devices. In some cases, even when the devices do have GNSS positioning, LoRaWAN geolocation is used as a check on that position:

Received Signal Strength Indication (RSSI): This measures the received signal power in milliwatts, and is measured in dBm. This method works for coarse positioning in the 1,000-to-2,000-meter range.

Time Difference of Arrival (TDOA): Each gateway must have a tightly synchronized time source for this method. Usually, this is obtained from a GNSS network such as GPS. The network server converts the timestamp of when messages were received by each gateway into a distance. It then plots those distances and estimates the devices’ location at the intersection. If a device can reach three or more gateways, its position can be calculated to be between 20 and 200 meters.

Regardless of which method is used, a good rule of thumb is that rural deployments will see accuracies toward the lower end of the range while accuracy in urban environments will be toward the higher end. Both methods will benefit from higher gateway density.

More Details
Feb 7, 2023
Long range wide area network (LoRaWAN) – Understanding Network and Security for Far-Edge Computing

Long range wide area network (LoRaWAN)

This is a protocol that sits on top of LoRA. It operates at Layer 2 (data link) and Layer 3 (network) of the OSI model. LoRaWAN does the same job that Ethernet and IP do for typical computer networks. It is possible to use LoRaWAN on top of a different Layer 1 radio technology, but this is uncommon.

Figure 3.34 – Examples of LoRaWAN gateways

LoRaWAN is an open standard that is supported by the LoRa Alliance, a non-profit organization that promotes the adoption of the technology. It is widely used, having been adopted by many major telcos around the world. LoRaWAN networks are used for applications that require long-range communication, low power consumption, and a low data rate, such as smart metering, asset tracking, and environmental monitoring.

The technology is well suited for non-video IoT applications because it allows rapid deployment of inexpensive sensors and relatively little infrastructure compared, to, say, 5G:

Figure 3.35 – Smart agriculture with LoRaWAN

A LoRaWAN network consists of the following elements:

End devices: These are also called nodes. They are the actual sensors, actuators, cameras, and the like in an IoT deployment. They communicate with gateways over the LoRa protocol.

Gateways: These are also called concentrators. These are similar to Wi-Fi extenders in that they act as a bridge from the end device/node to the network. Unlike Wi-Fi, however, a given device can talk to multiple gateways at once, and all a gateway does is gather those device messages and forward them to the network server. It is up to the network server to handle duplicate messages.

You usually want your devices to talk to a minimum of three gateways.

They also have an IP connection of some sort – it could be wired or wireless – so that they can communicate with the network server. That link is not LoRaWAN, because it is an aggregation point and needs higher throughput.

Network server: These could be thought of as similar to the AP controllers some enterprise Wi-Fi networks use to manage multiple access points. They receive messages from the gateways/concentrators and forward them to the application – both over an IP network.

They are also responsible for deduplication of messages. This is because multiple gateways can receive the same message from a given device, and they will simply forward them along and let the network server figure out if it is unique or not.

Note that LoRaWAN devices are not paired to a gateway – they are paired to a network server. The gateways are just a transport mechanism.

Application server: This is the final stop of a LoRaWAN message’s journey. The application server handles message encryption, data storage, and authentication of new nodes into the network.

More Details
Dec 8, 2022
Connecting to low-powered devices with LoRaWAN – Understanding Network and Security for Far-Edge Computing

First, let’s understand what Long Range (LoRa) is, as well as its benefits and drawbacks.

LoRa

LoRa operates at Layer 1 (physical) of the OSI model. You could think of it as the equivalent of a Cat-6 RJ-45 cable. You can’t do much with that on its own.

LoRa

LPWAN radio technology was developed by Semtech and designed for use in IoT. It is based on a spread spectrum technique called Chirp Spread Spectrum (CSS), which allows data to be transmitted over long distances (up to several kilometers) with low power consumption.

Benefits of LoRa

The following are some of the benefits of LoRa:

Long range14: 40 kilometers/25 miles (rural environment) and 5 kilometers/3 miles (urban environment)

14 These figures represent best-case line-of-sight range, or the radius of coverage with an omnidirectional antenna. Further, they can vary depending on the design of the gateway.

Low power: Designed to consume minimal energy, with some LoRa-capable devices having built-in batteries that last 20 years. The asynchronous connection model allows the device to sleep when there is no data to transmit or receive.

Reliable: Built-in forward error correction improves resilience against interference.

High penetration: Depending on the region, LoRa operates between 863-928MHz. This frequency range is less than half of the lowest 802.11x band. Due to this, LoRa signals penetrate obstacles approximately twice as well as Wi-Fi.

License-free: See the following table for the unlicensed frequency ranges for LoRa:

  RegionBand
North AmericaUS915 (902-928MHz)
EuropeEU868 (863-873MHz)
South AmericaAU915 (915-928MHz)
IndiaIN865 (865-867MHz)
AsiaAS923 (915-928MHz)

Figure 3.33 – License-free LoRa bands by region

Drawbacks of LoRa

Here are some of the drawbacks of LoRa:

Low throughput: The max bitrate is around 50 kilobits a second

Uncontrolled spectrum usage: License-free operation helps with deployment, but if you have ever had problems with your neighbor’s Wi-Fi router stepping on your signal, the potential is apparent.

More Details
Nov 26, 2022
WiFi and MIMO – Understanding Network and Security for Far-Edge Computing

As discussed previously, MIMO is a method for increasing effective throughput by deliberately exploiting multipath propagation. The different generations of WiFi make use of this in varying ways.

802.11n (Wi-Fi-4)

This supported the more limited Single User MIMO (SU-MIMO). As its name suggests, SU-MIMO means the access point can only be sent to one client at a time.

802.11ac (Wi-Fi-5)

This added MU-MIMO (d). The (d) stands for downlink. With MU-MIMO (d), only one station can transmit, but multiple stations can receive at any given time.

802.11ax (Wi-Fi-6)

This was extended to MU-MIMO (u/d). Now, multiple devices can both transmit and receive simultaneously.

MU-OFDMA

Basic OFDM has been supported since 802.11a (Wi-Fi-2). 802.11ax (Wi-Fi-6) has extended this to now support multiple users.

You could think of the older style of OFDM as a sequence of trucks, each delivering boxes from one vendor at a set time every day. MU-OFDMA allows each truck to be loaded with multiple vendor’s boxes. It also allows the delivery schedule of those trucks to happen only when there’s a full load.

Older Wi-Fi specifications were designed for web browsing and checking email. Congestion emerged as video streaming, AR/VR, and gaming became common. This, combined with more and more client devices transmitting at the same time, meant that the queuing caused by simple OFDM increased latency.

Perhaps most importantly, MU-OFDMA allows priorities to be set not only per client but per protocol/traffic type. In other words, the access point could prioritize video streaming at one level, IoT messages at another, and mission-critical VOIP at the highest.

802.11p (DSRC)

An amendment to the broader IEEE 802.11 Wireless LAN (WLAN) standard, 802.11p is tailored for high-speed, short-range communication in a vehicular environment. The standard operates in the 5.9 GHz frequency band and utilizes the Dedicated Short-Range Communications (DSRC) protocol to ensure low latency and reliable data exchange.

The primary advantage of DSRC over 4G/LTE or 5G for V2X is that it can provide some value in the absence of any infrastructure. If two V2X-equipped cars come within range of each other, they will exchange information in a peer-to-peer fashion. This would function even in the middle of the Sahara.

In 2016, Toyota became the first automaker to introduce cars equipped with V2X systems, followed by GM in 2017. Both of these used DSRC as opposed to 4G/LTE or 5G. While DSRC was the first standard the automotive industry adopted, that is changing for several reasons. Compared to 4G/LTE or 5G for V2X, DSRC suffers from the following limitations:

Limited capacity and scalability: DSRC operates in a narrow frequency band (5.9GHz), which limits its capacity to support a high number of simultaneous connections in dense traffic scenarios. 5G offers broader bandwidth and improved spectral efficiency, allowing it to handle more devices and users concurrently.

Lower data rates: DSRC offers lower data rates compared to 5G, which hinders its ability to support advanced V2X applications that require higher throughput, such as high-definition video streaming for autonomous vehicles. 5G, with its enhanced data rates, can better accommodate these demanding use cases.

Latency: Although DSRC provides relatively low latency communication, 5G has the potential to achieve even lower latencies, especially with the implementation of 5G Ultra-Reliable Low-Latency Communication (URLLC). URLLC can enable mission-critical applications and real-time control systems that demand near-instantaneous response times.

Network slicing: 5G supports network slicing, a feature that allows the creation of virtual networks tailored to specific use cases or applications. This enables the allocation of dedicated resources for V2X communications, ensuring the desired performance levels. DSRC, on the other hand, does not offer this level of customization and flexibility.

Global harmonization: While DSRC has been adopted in some regions, it has not achieved global harmonization, leading to inconsistencies in spectrum allocation and regulation across different countries. 5G has a more unified approach, with global standardization and broader adoption, making it more attractive for V2X implementations across various regions.

Keeping all of this in mind, automakers have begun to include both in their chipsets. The idea is that cellular networks are the primary communication path, and when those are not available, the chipset will leverage DSRC for peer-to-peer vehicle communication when and where it can.

More Details