Listing Supported Availability Zones (AZs) for Instance Types in AWS

Availability Zones

In Amazon Web Services (AWS), you generally spread your nodes over multiple availability zones for high availability. Unfortunately, not every node type is available in every availability zone, and in general, it is hard to know which zones one is available in in advance.

Error Types

If you are provisioning a single EC2 instance or you are only provisioning EC2s in an Auto Scaling Group (ASG) in a single-zone, you will obviously notice if you chose an incompatible zone for your instance type as it just won’t work.

It can be more nefarious when you have an ASG with multiple zones though. For example, our large scale airflow service runs in kubernetes, and the main ASG goes over 3 zones. Today, we ran out of IPs in two zones and realized that the third was not even being utilized. When hunting down why, this message was in the “activity” tracker page for the ASG.

Launching a new EC2 instance. Status Reason: Your requested instance type (r5.2xlarge) is not supported in your requested Availability Zone (us-east-1e). Please retry your request by not specifying an Availability Zone or choosing us-east-1a, us-east-1b, us-east-1c, us-east-1d, us-east-1f. Launching EC2 instance failed.

This is a very helpful message, but it’s unfortunate that we had to do the wrong thing in order to get the supported zones list.

Getting the Correct Zones in Advance

You can use this AWS CLI (V2) command to check the list of zones supported for an instance type in advance.

% aws ec2 describe-instance-type-offerings --location-type availability-zone --filters="Name=instance-type,Values=r5.2xlarge" --region us-east-1 --output table
-------------------------------------------------------
|            DescribeInstanceTypeOfferings            |
+-----------------------------------------------------+
||               InstanceTypeOfferings               ||
|+--------------+--------------+---------------------+|
|| InstanceType |  Location    |    LocationType     ||
|+--------------+--------------+---------------------+|
||  r5.2xlarge  |  us-east-1f  |  availability-zone  ||
||  r5.2xlarge  |  us-east-1c  |  availability-zone  ||
||  r5.2xlarge  |  us-east-1b  |  availability-zone  ||
||  r5.2xlarge  |  us-east-1d  |  availability-zone  ||
||  r5.2xlarge  |  us-east-1a  |  availability-zone  ||
|+--------------+--------------+---------------------+

Sources

You can find some information on this from AWS at this link.

Google Drive API v3 + Sheets + Shared Drives in Java

There are plenty of examples of how to use the Google Drive API online. A ton are for old versions though, and most are basic cases (not good with restricted sharing options/etc). Also, virtually none show you how to do things with shared drives.

I had to do all of this recently, so I hope this helps someone else avoid the pain I went through =). The only thing this assumes is that you have a valid credentials file generated from the developer console.

Defining Scopes

These scopes should all be enabled for your credentials on the consent screen part of the developer console. Also list them in your code.

static {
SCOPES = new ArrayList<>();
SCOPES.add(SheetsScopes.DRIVE);
SCOPES.add(SheetsScopes.DRIVE_FILE);
SCOPES.add(SheetsScopes.SPREADSHEETS);
}

Get Credentials

private HttpRequestInitializer getCredentials(NetHttpTransport httpTransport) {
    GoogleCredential credential = null;
    try {
        credential = GoogleCredential.fromStream(new FileInputStream(credentialsFilePath), httpTransport, JSON_FACTORY)
                .createScoped(SCOPES)
                .createDelegated(svcAccount);
    } catch (IOException e) {
        logger.error("ERROR Occurred while Authorization using the credentials provided...!!!");
    }
    return setHttpTimeout(credential);
}

Get Sheet and Drive Services

private Sheets getSheetService(String applicationName, NetHttpTransport httpTransport) throws FileNotFoundException {
return new Sheets.Builder(
httpTransport,
JSON_FACTORY,
getCredentials(httpTransport)
).setApplicationName(applicationName).build();
}

private Drive getDriveService(String applicationName, NetHttpTransport HTTP_TRANSPORT) throws FileNotFoundException {
return new Drive.Builder(HTTP_TRANSPORT,
JSON_FACTORY,
getCredentials(HTTP_TRANSPORT))
.setApplicationName(applicationName)
.build();
}

Create a Spreadsheet and Control Permissions

You can create a sheet easily with the sheet service. But, if you want to put your sheet in a specific parent folder and change permissions/control sharing settings, then you need to create it with the drive service setting a mime-type of sheet.

You can find your folder ID by navigating to your folder in google drive and getting the ID from the URL. Since we set “supports all drives”, we can create this file in a folder in our share drive. Without this setting, share drives fail with some kind of auth error.

private File createSpreadSheet(Drive driveService, String sheetTitle, String userFolderId) {
try {
File fileSpec = new File();
fileSpec.setName(sheetTitle);
fileSpec.setParents(Collections.singletonList(userFolderId));
fileSpec.setMimeType("application/vnd.google-apps.spreadsheet");

File sheetFile = driveService.files()
.create(fileSpec)
.setSupportsAllDrives(true) //Share drives don't work without this parameter.
.execute();

sheetFile.setViewersCanCopyContent(false);
sheetFile.setCopyRequiresWriterPermission(true);
sheetFile.setWritersCanShare(false);
driveService.files().update(sheetFile.getId(), sheetFile);

return sheetFile;
} catch (IOException e) {
throw new RuntimeException("Error occurred while creating the sheet.\n" + e);
}
}

Write Data to a Spreadsheet

private void writeToSpreadSheet(Sheets service, String spreadSheetId, String json) {
    final String range = "Sheet1";
    ValueRange body = new ValueRange()
            .setValues(getJsonData(json));
    UpdateValuesResponse response;
    try {
        response = service
                .spreadsheets()
                .values()
                .update(spreadSheetId, range, body)
                .setValueInputOption(VALUE_INPUT_OPTION)
                .execute();
    } catch (IOException e) {
        throw new RuntimeException("ERROR Occurred while insert / updating the values in Google Spread Sheet : " + spreadSheetId + "\n" + e);
    }
    logger.info(response.getUpdatedCells() + " cells updated.");
}

Find a Folder in Another Folder

private String getFolderIdIfExists(Drive driveService, String folderName) throws IOException {

    FileList folders = driveService.files().list()
            .setSupportsAllDrives(true)
            .setIncludeItemsFromAllDrives(true)
            .setQ(String.format("'%s' in parents and mimeType = 'application/vnd.google-apps.folder' and name = '%s'",
                    mainFolderId, folderName))
            .execute();

    return folders.getFiles().size() == 1 ? folders.getFiles().get(0).getId() : null;
}

Create a Folder In a Specific Folder

private String createUserFolderAndGetId(Drive driveService, String folderName) throws IOException {

    File fileSpec = new File();
    fileSpec.setName(folderName);
    fileSpec.setParents(Collections.singletonList(mainFolderId));
    fileSpec.setMimeType("application/vnd.google-apps.folder");

    File targetFolder = driveService.files()
            .create(fileSpec)
            .setSupportsAllDrives(true) //Share drives don't work without this parameter.
            .execute();

    return targetFolder.getId();
}

Helm 3 / GitLab Uninstall If Exists

Helm 3 does not seem to have a good way to “uninstall if exists” unfortunately. So, we had to find a way around that to make sure we could wipe out a previous deployment reliably, in CI/CD (in cases where we had to change a deployment version, which is rare).

As we use GitLab, we found this trick in the docs:

If any of the script commands return an exit code different from zero, the job will fail and further commands won’t be executed. This behavior can be avoided by storing the exit code in a variable:

job:
script:
- false || exit_code=$?
- if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi;

Using this, you can do:

helm uninstall -n your-namespace some-deployment-0-0-5 || exit_code=$?

And, while you’ll receive a note that it didn’t work on any release after it’s gone, the pipeline will continue on fine.

Presto / Hive find parquet files touched/referenced by a query/predicate.

We had a use case where we needed to find out which parquet files were touched by a query/predicate.  This was so that we could rewrite certain files in a special way to remove specific records.   In this case, presto was not mastering the data itself.

We found this awesome post -> https://stackoverflow.com/a/44011639/857994 on stack overflow which shows this pseudo-column:

select "$path" from table
This correctly shows you the parquet file a row came from, which is awesome!  I also found this  MR  which shows work has been merged to add $file_size  and $file_modified_time properties which is even cooler.
So, newer versions of presto-sql have even more power here.