Skip to content

Commit 726babb

Browse files
jlperlajlperla
authored
Updated difference equations unit tests (#106)
* Updating the differenceequations related unit tests * Stripped out old sequential stuff * Use new DifferenceEquations tag * Shutting off some of the slower CI * Finalized unit test cleanup * Updated benchmarks
1 parent 8ce721d commit 726babb

21 files changed

+523
-1370
lines changed

‎.github/workflows/CI.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@ jobs:
1010
fail-fast: false
1111
matrix:
1212
version:
13-
- '1.6'
13+
- "1.6"
14+
- "1.7"
1415
os:
1516
- ubuntu-latest
1617
arch:
@@ -44,7 +45,7 @@ jobs:
4445
- uses: actions/checkout@v2
4546
- uses: julia-actions/setup-julia@v1
4647
with:
47-
version: '1'
48+
version: "1"
4849
- run: |
4950
julia --project=docs -e '
5051
using Pkg

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,4 @@ JuliaSysimage.dll
2727
Manifest.toml
2828
.ipynb_checkpoints
2929
test/symbolics_sketch/rbc_observables*
30+
**/tune.json

‎Project.toml‎

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,10 @@ version = "0.4.9"
55

66
[deps]
77
ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
8+
DifferenceEquations = "e0ca9c66-1f9e-11ec-127a-1304ce62169c"
89
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
910
DocStringExtensions = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
1011
GeneralizedSylvesterSolver = "3b00829b-373b-4a35-afa6-09f888b04ecd"
11-
Infiltrator = "5903a43b-9cc3-4c30-8d17-598619ec4e9b"
1212
LaTeXStrings = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f"
1313
Latexify = "23fbe1c1-3f47-55db-b15f-69d7ec21a316"
1414
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
@@ -22,16 +22,13 @@ SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"
2222
StructArrays = "09ab397b-f2b6-538f-b94a-2f83cf4a842a"
2323
SymbolicUtils = "d1185830-fcd6-423d-90d6-eec64667417b"
2424
Symbolics = "0c5d862f-8b57-4792-8d23-62f2024744c7"
25-
TensorCast = "02d47bb6-7ce6-556a-be16-bb1710789e2b"
26-
Tullio = "bc48ee85-29a4-5162-ae0b-a64e1601d4bc"
27-
Zygote = "e88e6eb3-aa80-5325-afca-941959d7151f"
2825

2926
[compat]
3027
ChainRulesCore = "1"
28+
DifferenceEquations = "0.4.10"
3129
Distributions = "0.25"
3230
DocStringExtensions = "0.8"
3331
GeneralizedSylvesterSolver = "0.1"
34-
Infiltrator = "1"
3532
LaTeXStrings = "1"
3633
Latexify = "0.15"
3734
MacroTools = "0.5"
@@ -42,7 +39,4 @@ RecursiveFactorization = "0.2"
4239
StructArrays = "0.6"
4340
SymbolicUtils = "= 0.19.2"
4441
Symbolics = "4"
45-
TensorCast = "0.4"
46-
Tullio = "0.3"
47-
Zygote = "0.6"
48-
julia = "1.6"
42+
julia = "1.6, 1.7"

‎benchmark/Project.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
[deps]
2+
DifferenceEquations = "e0ca9c66-1f9e-11ec-127a-1304ce62169c"
3+
DifferentiableStateSpaceModels = "beacd9db-9e5e-4956-9b09-459a4b2028df"
4+
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
5+
LaTeXStrings = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f"
6+
Parameters = "d96e819e-fc66-5662-9728-84c9c7592b0a"
7+
SymbolicUtils = "d1185830-fcd6-423d-90d6-eec64667417b"
8+
Zygote = "e88e6eb3-aa80-5325-afca-941959d7151f"

‎benchmark/benchmarks.jl‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,15 @@ using Test, LinearAlgebra
66

77
# The BLAS threads is still an issue in Julia 1.7
88
# This has no effect with MKL
9-
DifferentiableStateSpaceModels.set_blas_threads()
9+
# See https://github.com/JuliaLang/julia/issues/33409
10+
# Default even lower?
11+
function set_blas_threads(openblas_threads = min(4, Int64(round(Sys.CPU_THREADS / 2))))
12+
if (BLAS.vendor() == :openblas64)
13+
println("Setting openblas64 threads = $openblas_threads")
14+
BLAS.set_num_threads(openblas_threads)
15+
end
16+
end
17+
set_blas_threads()
1018

1119
println("Running Testsuite with Threads.nthreads() = $(Threads.nthreads()) BLAS.vendor = $(BLAS.vendor()), and BLAS.num_threads = $(BLAS.get_num_threads()) \n")
1220

‎benchmark/linear_algebra.jl‎

Lines changed: 0 additions & 115 deletions
This file was deleted.

‎benchmark/tune.json‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎development.md‎

Lines changed: 61 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,25 @@
33

44
## Setup
55
One time setup:
6-
1. First, setup your environment for [VS Code](https://julia.quantecon.org/software_engineering/tools_editors.html), [github](https://julia.quantecon.org/software_engineering/version_control.html) and [unit testing](https://julia.quantecon.org/software_engineering/testing.html).
7-
2. In your global environment, (i.e. start julia without `--project` or use `]activate` to deactivate the current project) add in
8-
```julia
9-
] add BenchmarkTools Infiltrator TestEnv PkgBenchmark
10-
```
11-
6+
1. Setup your environment for [VS Code](https://julia.quantecon.org/software_engineering/tools_editors.html), [github](https://julia.quantecon.org/software_engineering/version_control.html) and [unit testing](https://julia.quantecon.org/software_engineering/testing.html).
7+
2. First start up a Julia repl in vscode this project
8+
3. Activate the global environment with `] activate` instead of the project environment
9+
4. Add in global packages for debugging and benchmarking
10+
```julia
11+
] add BenchmarkTools Infiltrator TestEnv PkgBenchmark
12+
```
13+
5. Activate the benchmarking project
14+
```julia
15+
] activate benchmark
16+
```
17+
6. Connect it the current version of the DSSM package,
18+
```julia
19+
] dev .
20+
```
21+
7. Instantiate all benchmarking dependencies,
22+
```julia
23+
] instantiate
24+
```
1225
## Formatting Code
1326
Within VS Code, format code before committing with `> Format Document` or `> Format Document With...` choices. This uses the `.JuliaFormatter.toml` file with [JuliaFormatter.jl](https://github.com/domluna/JuliaFormatter.jl).
1427

@@ -24,6 +37,14 @@ using TestEnv; TestEnv.activate()
2437
```
2538
At that point, you should be able to edit as if the `test/Project.toml` package was activated. For example, `include("test/runtests.jl")` should be roughly equivalent to `]test`.
2639

40+
If you wish to test the integration with a local version of a package, then after each time you do this step you will need to update the manifest accordingly. For example, if you have downloaded a `DifferenceEquations.jl` in a parallel folder to this one, then you can do
41+
```
42+
] dev ../DifferenceEquations.jl/.
43+
```
44+
And it will use that version until you restart julia. As `]test` will only use the pinned version, you can replicate the full regression test in that state with `include("test/runtests.jl")`
45+
46+
### Infiltrator and Debugging
47+
2748
A useful trick for debugging is with `Infiltrator.jl`. Put in a `@exfiltrate` in the code, (e.g. inside of a DSSM function) and it pushes all local variables into a global associated with the module.
2849

2950
For example, if `call_the_dssm_function_with_exfiltrate` was a function in the DSSM package with `@exfiltrate` in it, then you could do th following in the REPL or a unit test
@@ -38,22 +59,45 @@ Do not forget to remove the exfiltrate when committing to the server though, or
3859
This assumes you are running the repository in VS Code (and hence have the project file activated). If not, then you will need to activate it accordingly (e.g. `--project` when running Julia).
3960

4061
### Running the Full Benchmarks
41-
In your terminal
42-
```julia
62+
63+
Always start with the benchmarks activated, i.e. `] activate benchmark`
64+
A few utilities
65+
```julia
4366
using DifferentiableStateSpaceModels, PkgBenchmark
44-
data = benchmarkpkg(DifferentiableStateSpaceModels; resultfile = joinpath(pkgdir(DifferentiableStateSpaceModels),"benchmark/baseline.json"))
45-
export_markdown(joinpath(pkgdir(DifferentiableStateSpaceModels),"benchmark/trial.md"), data) # can export as markdown
67+
function save_benchmark(results_file = "baseline")
68+
data = benchmarkpkg(DifferentiableStateSpaceModels; resultfile = joinpath(pkgdir(DifferentiableStateSpaceModels),"benchmark/$results_file.json"))
69+
export_markdown(joinpath(pkgdir(DifferentiableStateSpaceModels),"benchmark/trial_$results_file.md"), data)
70+
end
71+
function generate_judgement(new_results, old_results = "baseline", judge_file = "judge")
72+
return export_markdown(joinpath(pkgdir(DifferentiableStateSpaceModels), "benchmark/$judge_file.md"),
73+
judge(PkgBenchmark.readresults(joinpath(pkgdir(DifferentiableStateSpaceModels),
74+
"benchmark/$new_results.json")),
75+
PkgBenchmark.readresults(joinpath(pkgdir(DifferentiableStateSpaceModels),
76+
"benchmark/$old_results.json"))))
77+
end
78+
79+
```
80+
In your terminal
81+
```julia
82+
save_benchmark("test") # default is "baseline"
83+
84+
# Or manually:
85+
# data = benchmarkpkg(DifferentiableStateSpaceModels; resultfile = joinpath(pkgdir(DifferentiableStateSpaceModels),"benchmark/baseline.json"))
86+
# export_markdown(joinpath(pkgdir(DifferentiableStateSpaceModels),"benchmark/trial.md"), data) # can export as markdown
4687
```
4788

48-
If you wanted to run a comparison with different parameters, or after modifications
89+
To compare against different parameters or after modifications, load the existing baseline and use the `judge` function to compare
90+
4991
```julia
50-
data = PkgBenchmark.readresults(joinpath(pkgdir(DifferentiableStateSpaceModels),"benchmark/baseline.json"))
51-
data_2 = benchmarkpkg(DifferentiableStateSpaceModels, BenchmarkConfig(
52-
env = Dict("JULIA_NUM_THREADS" => 8, "OPENBLAS_NUM_THREADS" => 1),
53-
juliacmd = `julia -O3`))
54-
data_judge = judge(data_2, data) # compare data_2 vs. data baseline
55-
export_markdown(joinpath(pkgdir(DifferentiableStateSpaceModels),"benchmark/judge.md"), data_judge)
92+
generate_judgement("test") # defaults to generate_judgement("test", "baseline", "judge")
93+
# Or manually
94+
# data = PkgBenchmark.readresults(joinpath(pkgdir(DifferentiableStateSpaceModels),"benchmark/baseline.json"))
95+
# data_2 = benchmarkpkg(DifferentiableStateSpaceModels, BenchmarkConfig(
96+
# env = Dict("JULIA_NUM_THREADS" => 4, "OPENBLAS_NUM_THREADS" => 1),
97+
# juliacmd = `julia -O3`))
98+
# export_markdown(joinpath(pkgdir(DifferentiableStateSpaceModels),"benchmark/judge.md"), judge(data_2, data))
5699
```
100+
57101
Note: Regenerate the tuning file `tune.json` with `benchmarkpkg(DifferentiableStateSpaceModels, retune = true)`
58102

59103
Also, you can comparing full benchmarks between commit hashes with (although benchmarks need to be in both since it will )

‎src/DifferentiableStateSpaceModels.jl‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,6 @@ using RecursiveFactorization # Or perhaps MKL will dominate this on all hardware
1616
using StructArrays
1717
using Symbolics
1818
using SymbolicUtils
19-
using TensorCast
20-
# using TimerOutputs
21-
using Zygote
2219

2320
# Flip off when not debugging
2421
# using Infiltrator
@@ -36,7 +33,6 @@ include("make_perturbation_model.jl")
3633
include("types.jl")
3734
include("generate_perturbation.jl")
3835
include("generate_perturbation_derivatives.jl")
39-
include("sequence.jl")
4036
include("Examples/Examples.jl")
4137

4238
end # module

0 commit comments

Comments
 (0)