Skip to content

Commit b227dc3

Browse files
authored
Merge pull request #53 from daos-stack/develop
DAOSGCP-63 Merge changes from develop to main
2 parents ff3cf49 + 2b62168 commit b227dc3

File tree

87 files changed

+3335
-1959
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

87 files changed

+3335
-1959
lines changed

.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,13 @@ terraform.tfvars
2020

2121
# Ignore terraform/examples/io500 active configuration symlink
2222
terraform/examples/io500/config/active_config.sh
23+
terraform/examples/io500/login
2324

2425
# Ignore other files
2526
id_rsa*
2627
hosts*
2728
*.flag
2829
keys.txt
30+
31+
# addlicense binary for pre-commit
32+
tools/autodoc/addlicense

.pre-commit-config.yaml

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,30 @@
1+
# Copyright 2022 Intel Corporation
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
115
---
216
repos:
17+
- repo: local
18+
hooks:
19+
- id: addlicense
20+
name: addlicense
21+
entry: tools/autodoc/addlicense.sh
22+
language: script
23+
types: ['text']
24+
exclude: ^(\.terraform\/.*$|\..*|README.md)
25+
exclude_types: ['json']
26+
pass_filenames: true
27+
require_serial: true
328
- repo: https://github.com/antonbabenko/pre-commit-terraform
429
rev: v1.64.0
530
hooks:
@@ -18,10 +43,3 @@ repos:
1843
exclude: \.terraform\/.*$
1944
pass_filenames: true
2045
require_serial: true
21-
- id: packer-readme
22-
name: packer-readme
23-
entry: tools/autodoc/terraform_docs.sh
24-
language: script
25-
files: ^.*\.pkr\.hcl$
26-
pass_filenames: true
27-
require_serial: true

.tflint.hcl

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
plugin "google" {
22
enabled = true
3-
version = "0.12.1"
3+
version = "0.16.1"
44
source = "github.com/terraform-linters/tflint-ruleset-google"
55
}
66
rule "terraform_deprecated_index" {

README.md

Lines changed: 68 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -12,16 +12,16 @@ This repository contains:
1212

1313
```
1414
.
15-
├── docs Miscellaneousc documentation and Cloud Shell tutorials
15+
├── docs Documentation and Cloud Shell tutorials
1616
├── images Cloud Build config files and Packer templates
17-
│ └── scripts Scripts that Packer runs to configure images
17+
│ └── scripts Scripts that Packer runs to build images
1818
├── terraform Terraform content
1919
│ ├── examples Examples that demonstrate how to use the DAOS Terraform modules
2020
│ └── modules Terraform modules for deploying DAOS server and client instances
2121
└── tools Tools used by pre-commit
2222
```
2323

24-
### Prerequisites
24+
## Prerequisites
2525

2626
In order to deploy DAOS on GCP you will need
2727

@@ -37,61 +37,109 @@ In order to deploy DAOS on GCP you will need
3737

3838
The documentation in this repository assumes that you will use [Cloud Shell](https://cloud.google.com/shell).
3939

40-
If you use [Cloud Shell](https://cloud.google.com/shell), you do not need to install any software on your system.
40+
With [Cloud Shell](https://cloud.google.com/shell), there is no need to install any software on your system.
4141

4242
If you do not want to use Cloud Shell, you will need to install
4343
- [Git](https://git-scm.com/)
4444
- [Google Cloud CLI](https://cloud.google.com/sdk/docs/install)
4545
- [Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli)
4646

47-
### Deploy DAOS on GCP
47+
## Deploying DAOS on GCP
4848

49-
Steps to deploy DAOS on GCP
49+
### Pre-Deployment Steps
50+
51+
The following pre-deployment steps are required
5052

5153
1. **Set defaults for Google Cloud CLI (```gcloud```)**
5254

5355
Only needs to be done once in your shell (Cloud Shell or local shell).
5456

55-
2. **Create a Packer image in your GCP project**
57+
2. **Enable service APIs and grant permissions**
58+
59+
Enabling APIs and granting service account permissions only needs to be done once for a GCP project.
60+
61+
3. **Create a Packer image in your GCP project**
5662

5763
In order to build DAOS images with Cloud Build your GCP project must contain a Packer image.
5864

59-
Building the Packer images only needs to be done once for a GCP project.
65+
Building the Packer image only needs to be done once for a GCP project.
6066

61-
3. **Build DAOS Server and Client images**
67+
4. **Build DAOS Server and Client images**
6268

6369
DAOS Server and Client instances are deployed using images that have DAOS pre-installed.
6470

65-
Therefore, the images need to be built prior to running Terraform.
71+
Therefore, the images need to be built prior to running Terraform to deploy a DAOS cluster.
72+
73+
Click the button below to open a Cloud Shell tutorial which will guide you through the pre-deployment steps listed above. If you lose your Cloud Shell session you can always come back to this README and click the button again.
74+
75+
[![DAOS on GCP Pre-Deployment](http://gstatic.com/cloudssh/images/open-btn.png)](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/daos-stack/google-cloud-daos&cloudshell_git_branch=main&shellonly=true&tutorial=docs/tutorials/pre-deployment.md)
76+
77+
### Deploy a DAOS Cluster with Terraform
78+
79+
After completing the pre-deployment steps listed above, you will need to write your own Terraform configuration for your particular use case.
80+
81+
The [terraform/modules](terraform/modules) in this repo can be used in your Terraform configuration to deploy DAOS server and client instances.
82+
83+
The [terraform/examples/daos_cluster](terraform/examples/daos_cluster/README.md) example serves as both a reference and a quick way to deploy a DAOS cluster.
84+
85+
Click the button below to open a Cloud Shell tutorial that will walk you through using the [terraform/examples/daos_cluster](terraform/examples/daos_cluster/README.md) example to deploy a DAOS cluster.
86+
87+
[![DAOS Cluster Example](http://gstatic.com/cloudssh/images/open-btn.png)](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/daos-stack/google-cloud-daos&cloudshell_git_branch=main&shellonly=true&tutorial=docs/tutorials/example_daos_cluster.md)
88+
89+
### Deploy a DAOS Cluster with the Google HPC Toolkit
6690

67-
> Click the button to open an interactive walk-through in Cloud Shell which will guide you through the steps listed above.
68-
>
69-
> [![DAOS on GCP Setup](http://gstatic.com/cloudssh/images/open-btn.png)](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/daos-stack/google-cloud-daos&cloudshell_git_branch=main&shellonly=true&tutorial=docs/tutorials/daosgcp_setup.md)
91+
The [HPC Toolkit](https://github.com/GoogleCloudPlatform/hpc-toolkit) is an open-source software offered by Google Cloud which makes it easy for customers to deploy HPC environments on Google Cloud.
7092

71-
4. **Use DAOS Terraform modules in your Terraform code**
93+
The HPC Toolkit allows customers to deploy turnkey HPC environments (compute, networking, storage, etc) following Google Cloud best-practices, in a repeatable manner. It is designed to be highly customizable and extensible, and intends to address the HPC deployment needs of a broad range of customers.
7294

73-
You will need to write your own Terraform code for your particular use case.
95+
The HPC Toolkit includes the following community examples which use the Terraform modules in this repository.
7496

75-
Your Terraform code can use the modules in ```terraform/modules``` to deploy DAOS server and client instances.
97+
| HPC Toolkit Community Example | Description |
98+
| ----------------------------- | ----------- |
99+
| [DAOS Cluster](https://github.com/GoogleCloudPlatform/hpc-toolkit/tree/main/community/examples/intel#daos-cluster) | Use the HPC Toolkit to deploy a standalone DAOS cluster |
100+
| [DAOS Server with Slurm cluster](https://github.com/GoogleCloudPlatform/hpc-toolkit/tree/main/community/examples/intel#daos-server-with-slurm-cluster) | Use the HPC Toolkit to deploy a set of DAOS servers for storage and a Slurm cluster in which the compute nodes are DAOS clients. The example demonstrates how to use DAOS storage in a Slurm job. |
76101

77-
The example Terraform configurations provided in ```terraform/examples``` can be used as a reference.
102+
If you are just getting started with deploying DAOS on GCP, it is highly recommended to use the HPC Toolkit as it can save you a lot of time as opposed to developing your own Terraform configuration.
78103

79-
See the [DAOS Cluster](terraform/examples/daos_cluster/README.md) example to learn more about how to use the ```terraform/modules```.
104+
## Support
105+
106+
Content in the [google-cloud-daos](https://github.com/daos-stack/google-cloud-daos) repository is licensed under the [Apache License Version 2.0](LICENSE) open-source license.
107+
108+
[DAOS](https://github.com/daos-stack/daos) is being distributed under the BSD-2-Clause-Patent open-source license.
109+
110+
Intel Corporation provides several ways for the users to get technical support:
111+
112+
1. Community support is available to everybody through Jira and via the DAOS channel for the Google Cloud users on Slack.
113+
114+
To access Jira, please follow these steps:
115+
116+
- Navigate to https://daosio.atlassian.net/jira/software/c/projects/DAOS/issues/
117+
118+
- You will need to request access to DAOS Jira to be able to create and update tickets. An Atlassian account is required for this type of access. Read-only access is available without an account.
119+
- If you do not have an Atlassian account, follow the steps at https://support.atlassian.com/atlassian-account/docs/create-an-atlassian-account/ to create one.
120+
121+
To access the Slack channel for DAOS on Google Cloud, please follow this link https://daos-stack.slack.com/archives/C03GLTLHA59
122+
123+
> This type of support is provided on a best-effort basis, and it does not have any SLA attached.
124+
125+
2. Commercial L3 support is available on an on-demand basis. Please get in touch with Intel Corporation to obtain more information.
126+
127+
- You may inquire about the L3 support via the Slack channel (https://daos-stack.slack.com/archives/C03GLTLHA59)
80128

81129
## Links
82130

83131
- [Distributed Asynchronous Object Storage (DAOS)](https://docs.daos.io/)
84132
- [Google Cloud Platform (GCP)](https://cloud.google.com/)
133+
- [Google HPC Toolkit](https://github.com/GoogleCloudPlatform/hpc-toolkit)
85134
- [Google Cloud CLI (gcloud)](https://cloud.google.com/cli)
86135
- [Google Cloud Build](https://cloud.google.com/build)
87136
- [Cloud Shell](https://cloud.google.com/shell)
88137
- [Packer](https://www.packer.io/)
89138
- [Terraform](https://www.terraform.io/)
90139

91-
92140
## Development
93141

94-
If you are contributing to the code in this repo, see [Development](docs/development.md)
142+
If you are contributing to this repo, see [Development](docs/development.md)
95143

96144
## License
97145

docs/development.md

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,6 @@ You will also need to install the dependencies that are required for the pre-com
1515

1616
Instructions can be found at the [pre-commit website](https://pre-commit.com/#install).
1717

18-
1918
2. Install [TFLint](https://github.com/terraform-linters/tflint)
2019

2120
See the [installation instructions](https://github.com/terraform-linters/tflint#installation)
@@ -31,8 +30,23 @@ You will also need to install the dependencies that are required for the pre-com
3130

3231
See [https://github.com/terraform-docs/terraform-docs](https://github.com/terraform-docs/terraform-docs)
3332

33+
4. Add `ADDLICENSE_COMPANY_NAME` environment variable to your `~/.bashrc` file
34+
35+
When pre-commit runs for the first time it will download the [google/addlicense](https://github.com/google/addlicense/releases/tag/v1.0.0) binary into the `tools/autodoc/` directory. The `addlicense` binary is excluded in the `.gitignore` file so it does not get checked into the repo.
36+
37+
The `addlicense` pre-commit hook will ensure that files have the proper license header.
38+
39+
The company name that is used in the license header is specified in the `ADDLICENSE_COMPANY_NAME` environment variable.
40+
41+
If the `ADDLICENSE_COMPANY_NAME` environment variable is not present, the company name in the license header will be set to **Intel Corporation**
42+
43+
If you do not work for Intel be sure to export the `ADDLICENSE_COMPANY_NAME` environment variable with the name of your company as it should appear in the license header of files.
44+
45+
```bash
46+
export ADDLICENSE_COMPANY_NAME="your_company_name_here"
47+
```
3448

35-
4. MacOS only
49+
5. MacOS only
3650

3751
MacOS users will need to install `findutils` and `coreutils`.
3852

0 commit comments

Comments
 (0)