Skip to main content Link Menu Expand (external link) Document Search Copy Copied
DL Docs

Installilling TAO5

Created : 26/05/2022
Status: Draft

Writeup

Checklist

  • k8 secret is created
  • storage provisionser is created and default storage class is set to the storage class of the helm chart (e.g. nfs-client)
  • ingress controller is installed

my instll feedback

helm upgrade  --install  tao-toolkit-api tao-toolkit-api/  --values tao-toolkit-api/values.yaml -n tao-gnet 
Release "tao-toolkit-api" does not exist. Installing it now.
W0728 15:27:40.208231  302145 warnings.go:70] path /tao-gnet/openapi.json cannot be used with pathType Prefix
W0728 15:27:40.208223  302145 warnings.go:70] path /tao-gnet/openapi.yaml cannot be used with pathType Prefix
W0728 15:27:40.208231  302145 warnings.go:70] path /tao-gnet/api/v1/user(/|$)(.*) cannot be used with pathType Prefix
W0728 15:27:40.208236  302145 warnings.go:70] path /tao-gnet/api/v1/login(/|$)(.*) cannot be used with pathType Prefix
NAME: tao-toolkit-api
LAST DEPLOYED: Fri Jul 28 15:27:39 2023
NAMESPACE: tao-gnet
STATUS: deployed
REVISION: 1
TEST SUITE: None

I got following warnings (from the second install)

Release "tao-toolkit-api" does not exist. Installing it now.
W0808 08:29:30.920884 2848700 warnings.go:70] path /tao-gnet/api/v1/user(/|$)(.*) cannot be used with pathType Prefix
W0808 08:29:30.920962 2848700 warnings.go:70] path /tao-gnet/openapi.yaml cannot be used with pathType Prefix
W0808 08:29:30.953083 2848700 warnings.go:70] path /tao-gnet/api/v1/login(/|$)(.*) cannot be used with pathType Prefix
W0808 08:29:30.954079 2848700 warnings.go:70] path /tao-gnet/openapi.json cannot be used with pathType Prefix
NAME: tao-toolkit-api
LAST DEPLOYED: Tue Aug  8 08:29:30 2023
NAMESPACE: tao-gnet
STATUS: deployed
REVISION: 1
TEST SUITE: None

check for various probes

liveness probe:

Exec into the pod (e.g. kubectl exec -it -n tao-gnet tao-toolkit-api-app-pod-5ffc48cd57-nc2qp -- /bin/bash)

curl -X GET "http://localhost:8000/api/v1/health/liveness"

readiness

curl -X GET "http://localhost:8000/api/v1/health/readiness

Errors:

Whenever I restart the k8 master The API login fails by timing out: I keep getting this in error logs

potential fix

use the stable version of ingress nginx

helm uninstall ingress-nginx -n tao-gnet helm repo list remove ingress-nginx helm repo add nginx-stable https://helm.nginx.com/stable helm install ingress-nginx nginx-stable/nginx-ingress -n tao-gnet

TAO files:

TAO Notebooks

Unauthorized: Credentials error: HTTPSConnectionPool(host='authn.nvidia.com', port=443): Max retries exceeded with url: /token?service=ngc (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9605168a90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
Wed Aug  9 11:10:55 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /api/v1/login/LOL (ip 192.168.251.163) !!!
Wed Aug  9 11:10:55 2023 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET /api/v1/login/LOL (192.168.251.163)

Notes:

I wanted to modify the yaml templates that take overriding values from the values.yaml so i made a backup file (e.g. ingress.yaml to ingress.yaml.backup) But I noticed that the created ingresses had the incorrect class name. then i renamed the backup to ingress-yaml.backup then it all worked. this means it only looks for the filename.yaml tag


Check the next topic

Click here to report Errors, make Suggestions or Comments!