-
Notifications
You must be signed in to change notification settings - Fork 51
Release Testing
Erik Jaegervall edited this page Dec 14, 2023
·
22 revisions
This page intends to list the tests that we shall/must do before we do a release for this repository. The intention of the release test is to do a reasonable regression test for components listed as Beta/Production/Stable/Mature in the KUKSA Component Maturity Overview. It is assumed that new features of individual components have been thoroughly tested, the focus of the release test is to verify that there are no regressions. For capacity reasons the tests have no intention to cover all possible use cases.
- No outstanding Pull Requests that may affect component behavior.
- CI Docker builds on latest master has succeeded
- Pre-release of kuksa-client exists on PyPI
erik@debian3:~/kuksa.val/kuksa_databroker$ cargo run --bin databroker -- --metadata ../data/vss-core/vss_release_4.0.json --insecure
(kuksa-client) erik@debian3:~/kuksa.val/kuksa-client$ kuksa-client grpc://127.0.0.1:55555
Test Client> setValue Vehicle.Speed 33
OK
Test Client> getValue Vehicle.Speed
- NOTE: Verify that correct version number is shown in the "hello message!*
erik@debian3:~/kuksa.val/kuksa_databroker$ cargo run --bin databroker-cli
...
sdv.databroker.v1 > feed Vehicle.Speed 44
[feed] OK
sdv.databroker.v1 > get Vehicle.Speed
[get] OK
Vehicle.Speed: 44.00
Start Client
erik@debian3:~/kuksa.val/kuksa_go_client$ go run .
Start Python client (or Databroker CLI) and verify that DTCList has been set by Go client:
Test Client> getValue Vehicle.OBD.DTCList
{
"path": "Vehicle.OBD.DTCList",
"value": {
"value": {
"values": [
"dtc1, dtc2",
"dtc2",
"dtc3, dtc3",
"dtc4"
]
},
"timestamp": "2023-07-20T13:03:46.314802+00:00"
}
}
Set Vehicle.Speed in Python client and verify that Go client gets a notification
Test Client> setValue Vehicle.Speed 65
2023/07/20 15:04:14 Vehicle.Speed Subscribed: timestamp:{seconds:1689858254 nanos:509989582} float:65
They rely on the client, so some testing is necessary. Two options exist, either build and install kuksa-client locally, or upload an alpha package to pypi. Testing with an uploaded package is preferable as that also tests that Pypi packaging works as expected.
Start DBC Feeder using default settings
erik@debian3:~/kuksa.val.feeders/dbc2val$ ./dbcfeeder.py
Let it run for a while, verify with Python client (or databroker CLI) that Vehicle.Speed change now and then (it takes some time to see the first change).
Test Client> getValue Vehicle.Speed
{
"path": "Vehicle.Speed",
"value": {
"value": 77.0,
"timestamp": "2023-07-20T13:17:26.946733+00:00"
}
}
Test Client> getValue Vehicle.Speed
{
"path": "Vehicle.Speed",
"value": {
"value": 54.0,
"timestamp": "2023-07-20T13:17:33.924064+00:00"
}
}
This is somewhat tricky. the trick seems to be to make sure that you use the default Python version of your distribution, i.e. not something you have sideloaded. Also using a different port is needed. To test:
sudo apt install gpsd-clients gpsd-tools
# Then if needed change python3 to point at "default system python", typically `/usr/bin/python3.11` or similar
sudo update-alternatives --config python3
Some distros have gpsd which maybe is sufficient
Run gpsfake:
erik@debian3:~/kuksa.val.feeders/gps2val$ gpsfake -t -P 2949 simplelog_example.nmea
Processing simplelog_example.nmea
gpsfake: log cycle of simplelog_example.nmea begins.
gpsfake: log cycle of simplelog_example.nmea begins.
Run gps2val:
Make sure python points at wanted version, or possible specify it manually like /usr/local/bin/python3.10
erik@debian3:~/kuksa.val.feeders/gps2val$ python gpsd_feeder.py --protocol grpc --port 55555 --insecure true --gpsd_port 2949
2023-07-20 15:26:32,525 INFO gpsfeeder: No configuration file found. Using default values.
2023-07-20 15:26:32,526 INFO gpsfeeder: Init kuksa client...
2023-07-20 15:26:32,578 INFO gpsfeeder: No token information provided, subsequent errors expected if Server/Databroker requires authentication!
2023-07-20 15:26:32,579 INFO gpsfeeder: Connection status is: False
2023-07-20 15:26:32,579 INFO kuksa_client.grpc: No Root CA present, it will not be posible to use a secure connection!
2023-07-20 15:26:32,579 INFO kuksa_client.grpc.aio: Establishing insecure channel
2023-07-20 15:26:32,579 INFO gpsfeeder: Init gpsd client...
2023-07-20 15:26:32,580 INFO gpsfeeder: Trying to connect gpsd at 127.0.0.1 port 2949
2023-07-20 15:26:32,580 INFO gpsfeeder: Using mapping
2023-07-20 15:26:32,582 INFO gpsfeeder: gpsd receive loop started
gRPC channel connected.
2023-07-20 15:26:32,601 INFO gpsfeeder: Number of VSS messages sent so far: 1
2023-07-20 15:26:32,620 INFO gpsfeeder: Number of VSS messages sent so far: 2
2023-07-20 15:26:33,640 INFO gpsfeeder: Number of VSS messages sent so far: 4
verify with KUKSA Client that lat and/or lon is updated
Test Client> getValue Vehicle.CurrentLocation.Latitude
{
"path": "Vehicle.CurrentLocation.Latitude",
"value": {
"value": 48.779966667,
"timestamp": "2023-07-20T13:27:09.618384+00:00"
}
}
Test Client> getValue Vehicle.CurrentLocation.Latitude
{
"path": "Vehicle.CurrentLocation.Latitude",
"value": {
"value": 48.780183333,
"timestamp": "2023-07-20T13:27:13.012102+00:00"
}
}
erik@debian3:~/kuksa.val/kuksa_databroker$ cargo run --bin databroker -- --metadata ../data/vss-core/vss_release_4.0.json --tls-cert ../kuksa_certificates/Server.pem --tls-private-key ../kuksa_certificates/Server.key --jwt-public-key ../kuksa_certificates/jwt/jwt.key.pub
Verify with Kuksa-client that connection works
erik@debian3:~/kuksa.val/kuksa-client$ kuksa-client grpcs://127.0.0.1:55555 --cacertificate ../kuksa_certificates/CA.pem --tls-server-name Server
...
Secure gRPC channel connected.
Test Client> authorize ../jwt/provide-all.token
"Authenticated"
Test Client> setValue Vehicle.Speed 12
OK
Test Client> getValue Vehicle.Speed
{
"path": "Vehicle.Speed",
"value": {
"value": 12.0,
"timestamp": "2023-07-21T09:12:42.968598+00:00"
}
}
Test Client>
erik@debian4:~/kuksa-python-sdk/kuksa-client$ kuksa-client grpcs://127.0.0.1:55555 --cacertificate /home/erik/kuksa-common/tls/CA.pem --tls-server-name Server
⢀⣤⣶⣾⣿⢸⣿⣿⣷⣶⣤⡀
⣴⣿⡿⠋⣿⣿ ⠈⠙⢿⣿⣦
⣾⣿⠋ ⣿⣿ ⣶⣿ ⠙⣿⣷
⣸⣿⠇ ⣿⣿⠠⣾⡿⠃ ⠸⣿⣇ ⣶ ⣠⡶⠂ ⣶ ⢰⡆ ⢰⡆⢀⣴⠖ ⢠⡶⠶⠶⡦ ⣰⣶⡀
⣿⣿ ⠿⢿⣷⣦⡀ ⣿⣿ ⣿⢾⣏ ⣿ ⢸⡇ ⢸⡷⣿⡁ ⠘⠷⠶⠶⣦ ⢠⡟⠘⣷
⢹⣿⡆ ⣿⣶⠈⢻⣿⡆ ⢰⣿⡏ ⠿ ⠙⠷⠄ ⠙⠷⠶⠟⠁ ⠸⠇⠈⠻⠦ ⠐⠷⠶⠶⠟ ⠠⠿⠁ ⠹⠧
⢿⣿⣄ ⣿⣿ ⠿⣿ ⣠⣿⡿
⠻⣿⣷⡄⣿⣿ ⢀⣠⣾⣿⠟ kuksa-client CLI
⠈⠛⠇⢿⣿⣿⣿⣿⡿⠿⠛⠁ 0.0.0
Default tokens directory: /home/erik/kuksa-python-sdk/submodules/kuksa.val/kuksa_certificates/jwt
Connecting to VSS server at 127.0.0.1 port 55555 using KUKSA GRPC protocol.
TLS will be used.
INFO 2023-11-16 10:36:56,850 kuksa_client.grpc Using TLS with Root CA from /home/erik/kuksa-common/tls/CA.pem
INFO 2023-11-16 10:36:56,855 kuksa_client.grpc Using client private key and certificates, mutual TLS supported if supported by server
INFO 2023-11-16 10:36:56,855 kuksa_client.grpc.aio Establishing secure channel
INFO 2023-11-16 10:36:56,855 kuksa_client.grpc.aio Using TLS server name Server
INFO 2023-11-16 10:36:56,909 kuksa_client.grpc.aio Unauthenticated channel started
Secure gRPC channel connected.
Test Client> authorize /home/erik/kuksa-common/jwt/actuate-provide-all.token
"Authenticated"
Test Client> setValue Vehicle.Speed 33
OK
Test Client> getValue Vehicle.Speed
{
"path": "Vehicle.Speed",
"value": {
"value": 33.0,
"timestamp": "2023-11-16T09:37:49.769857+00:00"
}
}
erik@debian3:~/kuksa.val/kuksa_databroker$ cargo run --bin databroker-cli -- --token-file ../jwt/provide-all.token --ca-cert ../kuksa_certificates/CA.pem
Finished dev [unoptimized + debuginfo] target(s) in 0.20s
Running `/home/erik/kuksa.val/target/debug/databroker-cli --token-file ../jwt/provide-all.token --ca-cert ../kuksa_certificates/CA.pem`
...
sdv.databroker.v1 > feed Vehicle.Speed 5
[feed] OK
sdv.databroker.v1 > get Vehicle.Speed
[get] OK
Vehicle.Speed: 5.00
sdv.databroker.v1 >
NOTE: Go Client does not support TLS, test skipped
Verify first that Databroker:master is recently built and there is no pending/queued builds.
erik@debian3:~/kuksa.val/kuksa_databroker$ docker pull ghcr.io/eclipse/kuksa.val/databroker:master
erik@debian3:~/kuksa.val/kuksa_databroker$ docker run --rm -it -p 55555:55555/tcp -v /home/erik/kuksa.val/kuksa_certificates:/certs ghcr.io/eclipse/kuksa.val/databroker:master --tls-cert /certs/Server.pem --tls-private-key /certs/Server.key --jwt-public-key /certs/jwt/jwt.key.pub
Then repeat test for local build
Test as above for TLS + Token
Note: Verify that correct version is shown!
erik@debian3:~/kuksa.val/kuksa-client$ docker run --rm -it -v /home/erik/kuksa.val/kuksa_certificates:/certs -v /home/erik/kuksa.val/jwt:/jwt --net=host kuksa-client:latest --cacertificate /certs/CA.pem --tls-server-name Server grpcs://127.0.0.1:55555
Welcome to Kuksa Client version 0.1.6
...
Secure gRPC channel connected.
Test Client> authorize /jwt/provide-all.token
"Authenticated"
Test Client> setValue Vehicle.Speed 63
OK
Test Client> getValue Vehicle.Speed
{
"path": "Vehicle.Speed",
"value": {
"value": 63.0,
"timestamp": "2023-07-21T11:31:28.461371+00:00"
}
}
Test Client>
erik@debian4:~/kuksa-python-sdk/kuksa-client$ docker run --rm -it -v /home/erik/kuksa-common/tls:/tls -v /home/erik/kuksa-common/jwt:/jwt --net=host kuksa-client:latest --cacertificate /tls/CA.pem --tls-server-name Server grpcs://127.0.0.1:55555
⢀⣤⣶⣾⣿⢸⣿⣿⣷⣶⣤⡀
⣴⣿⡿⠋⣿⣿ ⠈⠙⢿⣿⣦
⣾⣿⠋ ⣿⣿ ⣶⣿ ⠙⣿⣷
⣸⣿⠇ ⣿⣿⠠⣾⡿⠃ ⠸⣿⣇ ⣶ ⣠⡶⠂ ⣶ ⢰⡆ ⢰⡆⢀⣴⠖ ⢠⡶⠶⠶⡦ ⣰⣶⡀
⣿⣿ ⠿⢿⣷⣦⡀ ⣿⣿ ⣿⢾⣏ ⣿ ⢸⡇ ⢸⡷⣿⡁ ⠘⠷⠶⠶⣦ ⢠⡟⠘⣷
⢹⣿⡆ ⣿⣶⠈⢻⣿⡆ ⢰⣿⡏ ⠿ ⠙⠷⠄ ⠙⠷⠶⠟⠁ ⠸⠇⠈⠻⠦ ⠐⠷⠶⠶⠟ ⠠⠿⠁ ⠹⠧
⢿⣿⣄ ⣿⣿ ⠿⣿ ⣠⣿⡿
⠻⣿⣷⡄⣿⣿ ⢀⣠⣾⣿⠟ kuksa-client CLI
⠈⠛⠇⢿⣿⣿⣿⣿⡿⠿⠛⠁ 0.0.0
Default tokens directory: /kuksa-client/_internal/kuksa_client/kuksa_server_certificates/jwt
Connecting to VSS server at 127.0.0.1 port 55555 using KUKSA GRPC protocol.
TLS will be used.
INFO 2023-11-16 09:43:54,901 kuksa_client.grpc Using TLS with Root CA from /tls/CA.pem
INFO 2023-11-16 09:43:54,902 kuksa_client.grpc Using client private key and certificates, mutual TLS supported if supported by server
INFO 2023-11-16 09:43:54,902 kuksa_client.grpc.aio Establishing secure channel
INFO 2023-11-16 09:43:54,902 kuksa_client.grpc.aio Using TLS server name Server
INFO 2023-11-16 09:43:54,966 kuksa_client.grpc.aio Unauthenticated channel started
Secure gRPC channel connected.
Test Client> authorize /jwt/provide-all.token
"Authenticated"
Test Client> setValue Vehicle.Speed 63
OK
Test Client> getValue Vehicle.Speed
{
"path": "Vehicle.Speed",
"value": {
"value": 63.0,
"timestamp": "2023-11-16T09:44:11.296047+00:00"
}
}
Test Client>
erik@debian4:~/kuksa.val$ docker run --rm -it --net=host -v /home/erik/kuksa.val/kuksa_certificates:/certs -v /home/erik/kuksa.val/jwt:/jwt databroker-cli:latest --token-file /jwt/provide-all.token --ca-cert /certs/CA.pem
This is the default setting if using default config
./kuksa-val-server --vss ./vss_release_4.0.json
Do as below, checks given as comments with #
Note: On newer versions tokens seems to be in Default tokens directory: /kuksa-client/_internal/kuksa_certificates/jwt
erik@debian4:~/kuksa.val$ docker run --rm -it --net=host kuksa-client:latest wss://127.0.0.1:8090
Welcome to Kuksa Client version 0.4.0a5.post9+git.77a1d363
# Verify that version is as expected
Websocket connected securely.
Test Client> getValue Vehicle.Speed
# Shall give authorization error
Test Client> authorize /kuksa-client/kuksa_certificates/jwt/super-admin.json.token
# Shall be accepted
Test Client> setValue Vehicle.Speed 54
# OK
Test Client> getValue Vehicle.Speed
# OK
Test Client>
erik@debian4:~/kuksa-python-sdk$ docker run --rm -it --net=host kuksa-client:latest wss://127.0.0.1:8090
⢀⣤⣶⣾⣿⢸⣿⣿⣷⣶⣤⡀
⣴⣿⡿⠋⣿⣿ ⠈⠙⢿⣿⣦
⣾⣿⠋ ⣿⣿ ⣶⣿ ⠙⣿⣷
⣸⣿⠇ ⣿⣿⠠⣾⡿⠃ ⠸⣿⣇ ⣶ ⣠⡶⠂ ⣶ ⢰⡆ ⢰⡆⢀⣴⠖ ⢠⡶⠶⠶⡦ ⣰⣶⡀
⣿⣿ ⠿⢿⣷⣦⡀ ⣿⣿ ⣿⢾⣏ ⣿ ⢸⡇ ⢸⡷⣿⡁ ⠘⠷⠶⠶⣦ ⢠⡟⠘⣷
⢹⣿⡆ ⣿⣶⠈⢻⣿⡆ ⢰⣿⡏ ⠿ ⠙⠷⠄ ⠙⠷⠶⠟⠁ ⠸⠇⠈⠻⠦ ⠐⠷⠶⠶⠟ ⠠⠿⠁ ⠹⠧
⢿⣿⣄ ⣿⣿ ⠿⣿ ⣠⣿⡿
⠻⣿⣷⡄⣿⣿ ⢀⣠⣾⣿⠟ kuksa-client CLI
⠈⠛⠇⢿⣿⣿⣿⣿⡿⠿⠛⠁ 0.4.2.post0+git.10f61d1d.dirty
Default tokens directory: /kuksa-client/_internal/kuksa_client/kuksa_server_certificates/jwt
Connecting to VSS server at 127.0.0.1 port 8090 using VISS protocol.
TLS will be used.
connect to wss://127.0.0.1:8090
Websocket connected. Negotiated subprotocol None
Test Client> authorize /kuksa-client/_internal/kuksa_client/kuksa_server_certificates/jwt/super-admin.json.token
{
"TTL": 1767225599,
"action": "authorize",
"requestId": "f94ca2a2-6133-4f0a-b8b1-39ce6ee07a99",
"ts": "2023-11-16T10:04:02.1700129042Z"
}
Test Client> setValue Vehicle.Speed 5
{
"action": "set",
"requestId": "6c6ac3df-6be9-4e9d-b216-f21a7e455416",
"ts": "2023-11-16T10:04:14.1700129054Z"
}
Test Client> getValue Vehicle.Speed
{
"action": "get",
"data": {
"dp": {
"ts": "2023-11-16T10:04:14.173862849Z",
"value": "5.0"
},
"path": "Vehicle.Speed"
},
"requestId": "0ee1fca1-c898-4416-88e5-b578ce988364",
"ts": "2023-11-16T10:04:21.1700129061Z"
}
docker pull ghcr.io/eclipse/kuksa.val/kuksa-val:master
docker run -it --rm -p 127.0.0.1:8090:8090 -e LOG_LEVEL=ALL ghcr.io/eclipse/kuksa.val/kuksa-val:master
Test with kuksa-client like above
Change config to connect to KUKSA.val server, typically use:
[general]
server_type = kuksa_val_server
port = 8090
tls = True
tls_server_name=localhost
token=../../kuksa.val/kuksa_certificates/jwt/super-admin.json.token
Run as ./dbcfeeder.py
Verify that Vehicle.Speed changes
Start databroker as above. Install kuksa-client and run it. Make sure you do not run from the directory where kuksa-client resides
erik@debian3:~$ pip install -U kuksa-client
...
Successfully installed kuksa-client-0.4.0
erik@debian3:~$ kuksa-client --ip 127.0.0.1 --port 55555 --protocol grpc --cacertificate kuksa.val/kuksa_certificates/CA.pem --tls-server-name Server
...
Welcome to Kuksa Client version 0.4.0
Test Client> authorize kuksa.val/jwt/provide-all.token
"Authenticated"
Test Client> setValue Vehicle.Speed 43
OK
Test Client> getValue Vehicle.Speed
{
"path": "Vehicle.Speed",
"value": {
"value": 43.0,
"timestamp": "2023-07-26T09:21:14.432932+00:00"
}
}
Test Client>
In general, download containers:
docker pull ghcr.io/eclipse/kuksa.val/databroker:0.4
erik@debian3:~/kuksa.val/kuksa_databroker$ docker run --rm -it -p 55555:55555/tcp -v /home/erik/kuksa.val/kuksa_certificates:/certs ghcr.io/eclipse/kuksa.val/databroker:0.4 --tls-cert /certs/Server.pem --tls-private-key /certs/Server.key
Test with Python Client and CLI. Check that correct version is shown
erik@debian3:~/kuksa.val$ docker run --rm -it --net=host -v /home/erik/kuksa.val/kuksa_certificates:/certs ghcr.io/eclipse/kuksa.val/kuksa-client:0.4 --ip 127.0.0.1 --port 55555 --protocol grpc --cacertificate /certs/CA.pem --tls-server-name Server
Welcome to Kuksa Client version 0.4.0
Secure gRPC channel connected.
Test Client> setValue Vehicle.Speed 3
OK
Test Client> getValue Vehicle.Speed
{
"path": "Vehicle.Speed",
"value": {
"value": 3.0,
"timestamp": "2023-07-27T07:56:44.898774+00:00"
}
}
And Databroker CLI
erik@debian3:~/kuksa.val$ docker run --rm -it --net=host -v /home/erik/kuksa.val/kuksa_certificates:/certs -v /home/erik/kuksa.val/jwt:/jwt ghcr.io/eclipse/kuksa.val/databroker-cli:master --token-file /jwt/provide-all.token --ca-cert /certs/CA.pem
⠀⠀⠀⢀⣤⣶⣾⣿⢸⣿⣿⣷⣶⣤⡀
⠀⠀⣴⣿⡿⠋⣿⣿⠀⠀⠀⠈⠙⢿⣿⣦⠀
⠀⣾⣿⠋⠀⠀⣿⣿⠀⠀⣶⣿⠀⠀⠙⣿⣷
⣸⣿⠇⠀⠀⠀⣿⣿⠠⣾⡿⠃⠀⠀⠀⠸⣿⣇⠀⠀⣶⠀⣠⡶⠂⠀⣶⠀⠀⢰⡆⠀⢰⡆⢀⣴⠖⠀⢠⡶⠶⠶⡦⠀⠀⠀⣰⣶⡀
⣿⣿⠀⠀⠀⠀⠿⢿⣷⣦⡀⠀⠀⠀⠀⠀⣿⣿⠀⠀⣿⢾⣏⠀⠀⠀⣿⠀⠀⢸⡇⠀⢸⡷⣿⡁⠀⠀⠘⠷⠶⠶⣦⠀⠀⢠⡟⠘⣷
⢹⣿⡆⠀⠀⠀⣿⣶⠈⢻⣿⡆⠀⠀⠀⢰⣿⡏⠀⠀⠿⠀⠙⠷⠄⠀⠙⠷⠶⠟⠁⠀⠸⠇⠈⠻⠦⠀⠐⠷⠶⠶⠟⠀⠠⠿⠁⠀⠹⠧
⠀⢿⣿⣄⠀⠀⣿⣿⠀⠀⠿⣿⠀⠀⣠⣿⡿
⠀⠀⠻⣿⣷⡄⣿⣿⠀⠀⠀⢀⣠⣾⣿⠟ databroker-cli
⠀⠀⠀⠈⠛⠇⢿⣿⣿⣿⣿⡿⠿⠛⠁ v0.4.0
Successfully connected to http://127.0.0.1:55555/
sdv.databroker.v1 >
docker run -it --rm -p 127.0.0.1:8090:8090 -e LOG_LEVEL=ALL ghcr.io/eclipse/kuksa.val/kuksa-val:0.4.1
docker run --rm -it --net=host -v /home/erik/kuksa.val/kuksa_certificates:/certs ghcr.io/eclipse/kuksa.val/kuksa-client:0.4.1 wss://127.0.0.1:8090 --cacertificate /certs/CA.pem --tls-server-name Server