Need help to run the tests in specific order

[AdarshdeepCheema]

test suite

• books_suite_test.go

package books_test

import (
	"testing"

	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
)

func TestBooks(t *testing.T) {
	RegisterFailHandler(Fail)
	RunSpecs(t, "Books Suite")
}

test spec

• 01_install_test.go

package books_test

import (
	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
)

var _ = Describe("install:", func() {
	Context("", func() {
		It("case1", func() {
			Expect(nil).NotTo(HaveOccurred())
		})
	})
})

• 02_read_test.go

package books_test

import (
	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
)

var _ = Describe("read:", func() {
	Context("", func() {
		It("case2", func() {
			Expect(nil).NotTo(HaveOccurred())
		})
	})
})

• 03_isbn_test.go

package books_test

import (
	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
)

var _ = Describe("isbn:", func() {
	Context("", func() {
		It("case3", func() {
			Expect(nil).NotTo(HaveOccurred())
		})
	})
})

• 04_uninstall_test.go

package books_test

import (
	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
)

var _ = Describe("uninstall:", func() {
	Context("", func() {
		It("case4", func() {
			Expect(nil).NotTo(HaveOccurred())
		})
	})
})

when I run my test twice:

ssli@order-demo(master)$ ginkgo -v
Running Suite: Books Suite
==========================
Random Seed: 1604045666
Will run 4 of 4 specs

uninstall:
  case4
  /Users/ssli/share/git/go_practice/ginkgo-demo/order-demo/04_uninstall_test.go:10
•
------------------------------
read:
  case2
  /Users/ssli/share/git/go_practice/ginkgo-demo/order-demo/02_read_test.go:10
•
------------------------------
install:
  case1
  /Users/ssli/share/git/go_practice/ginkgo-demo/order-demo/01_install_test.go:10
•
------------------------------
isbn:
  case3
  /Users/ssli/share/git/go_practice/ginkgo-demo/order-demo/03_isbn_test.go:10
•
Ran 4 of 4 Specs in 0.000 seconds
SUCCESS! -- 4 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Ginkgo ran 1 suite in 1.563022387s
Test Suite Passed
ssli@order-demo(master)$ ginkgo -v
Running Suite: Books Suite
==========================
Random Seed: 1604045673
Will run 4 of 4 specs

isbn:
  case3
  /Users/ssli/share/git/go_practice/ginkgo-demo/order-demo/03_isbn_test.go:10
•
------------------------------
uninstall:
  case4
  /Users/ssli/share/git/go_practice/ginkgo-demo/order-demo/04_uninstall_test.go:10
•
------------------------------
install:
  case1
  /Users/ssli/share/git/go_practice/ginkgo-demo/order-demo/01_install_test.go:10
•
------------------------------
read:
  case2
  /Users/ssli/share/git/go_practice/ginkgo-demo/order-demo/02_read_test.go:10
•
Ran 4 of 4 Specs in 0.000 seconds
SUCCESS! -- 4 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Ginkgo ran 1 suite in 1.039894448s
Test Suite Passed

I get 2 orders:

• 04_uninstall_test.go -> 02_read_test.go -> 01_install_test.go -> 03_isbn_test.go
• 03_isbn_test.go -> 04_uninstall_test.go -> 01_install_test.go -> 02_read_test.go

I hope my test run in the following order:

• 01_install_test.go -> 02_read_test.go -> 03_isbn_test.go -> 04_uninstall_test.go

we need to split the test case by feature or module. This makes it easier to maintain and add tests. so I'll ave multiple test files(01_install_test.go/02_read_test.go/03_isbn_test.go/04_uninstall_test.go). and I hope the tests runs as following order:

• 01_install_test.go -> 02_read_test.go -> 03_isbn_test.go -> 04_uninstall_test.go

[onsi]

hi @AdarshdeepCheema I recommend reading this section of the documentation:
https://onsi.github.io/ginkgo/#mental-model-ginkgo-assumes-specs-are-independent

in short - Ginkgo does not guarantee order of specs. in fact it prefers to enable randomized spec orders to prevent spec pollution. splitting tests by feature or module is not a problem. requiring strict ordering of tests in the way you are proposing is a problem and can be somewhat hard to maintain.

in your case you can place all these specs in a single Ordered Container, or you could set up a test suite that installs things to start, then runs a variety of tests agianst them, then cleans up at hte end. if you could provide more detail about the problem you are trying to solve, the system you are trying to test, and the constraints you are facing ("start up is slow" etc.) i'd be happy to help you think through how best to organize your specs.

[AdarshdeepCheema]

The example that I gave you above is just a prototype. In our case things are way too big in numbers and size.
I cannot add all the specs in a single Ordered Container as we will have more than 100 test specs.
For example:
We are working on a Kubernetes ENV, where we are creating CRDs, PODS, Secrets, PVC, Services, Service Accounts, Roles and other stuff. We do these in a specific order as if the order changes the things can fail and break.
We do not want to bring everything up all at once as it takes like 20 mins, before running the tests.
We want to do it step by step and if anything fails the testing should stop right away and avoid waiting for 20 mins and verify the k8 resources later.
So in our testing we want to generate small stuff at a time in k8 and then test it and then move to next step. Because all the test specs are randomized, I cannot make sure that all tests will pass every-time, unless they are run in a specific order.
Can we do something or play with with RandomSeed/ seed to achieve it?

[onsi]

hey @AdarshdeepCheema - there are many examples in the k8s community of folks using ginkgo to accomplish what you are aiming to do - including the k8s e2e suite itself which is ~7000 specs that run in parallel in random order. i'm confident i can help you figure this out for your usecase and would like to encourage you to not reach for ordered tests as a solution.

a common usecase that i've seen looks like this:

• spin up k8s
• set it up as desired/needed for all specs
• start running the specs, each spec performs whatever additional setup/tear down it needs
• clean up at the end

often, test suites do this in such a way that each spec is independent and works within a namespace such that the specs can be run in parallel. this allows a suite comprised of individually slow specs to run much more efficiently in parallel.

i don't have enough detail to fully help you - the more you can provide the better. or if you have a repo you can point me to i can help. but, in general, you don't need to worry about having to set up each little step in order. for example imagine this pseudocode:

BeforeEach(func() {
    Expect(createUser()).To(Succeed())
    Expect(createOperator()).To(Succeed())
    Expect(deployApp()).To(Succeed())
})

It("validates the app is running", func() {
 ...
})

If any of those setup steps fails Ginkgo will cease execution, record a failure, and not run the test in question. you don't need to string the separate steps together as a set of ordered specs.

[AdarshdeepCheema]

Can we use Ordered Container and then run the specs from a specific file from it ? if possible how ??

Describe("checking out a book", func() {


  It("can specs in 01_install_test.go file", func() {
    err := call to specs in  01_install_test.go   
    Expect(err).NotTo(HaveOccurred())
  })

  It("can specs in 02_read_test.go file", func() {
    err := call to specs in  02_read_test.go   
    Expect(err).NotTo(HaveOccurred())
  })

  It("can specs in 03_isbn_test.go file", func() {
    err := call to specs in  03_isbn_test.go   
    Expect(err).NotTo(HaveOccurred())
  })

  It("can specs in 04_uninstall_test.go file", func() {
    err := call to specs in  04_uninstall_test.go   
    Expect(err).NotTo(HaveOccurred())
  })

})

Can you consider it as an enhancement that you can implement in future. Though it will againt the Ginkgo design, But user should have the choice to choose it and its consequences?

[onsi]

hi - you can accomplish this today - simply define your code in a function in the external file and call that function in the It.

I will not be adding first-class support for this to Ginkgo, however. I'd be happy to help you analyze and build out solutions for your problem space that better match Ginkgo's semantics and structure - as I've mentioned I'll need more detail (e.g. you can point me to an open source repo; or give me just one or two actual fleshed out examples so I can engage in the conversation more productively). I don't doubt we can figure something out.

[AdarshdeepCheema]

This is not an opensource project and we cant share any github links.

I want to have 10-30 specs in each of the files that I mentioned above, Can I add Describe {it{spec 1 spec2} structure of ginkgo in these files wrapped in a function ?? and then calling these functions from another file?
If yes then can you please point me to an example ??

for example i have created the below in FILE_A
Can I call FILE_A.TestNamespaces() from FILE_B ?

func TestNamespaces() (string) {
	var _ = Describe("Control Plane Opearator", Ordered, Label("controlplane", "operator", "test 02"), func() {

		Context("***************TESTING NAMESPACES***************", func() {
	
			It("All required Namespaces should be created", func() {
				By("checking if controlplane namespace is created")
				_, err := utils.GetNamespace(clientSet, CONTROLPLANE_NAMESPACE)
				Expect(err).NotTo(HaveOccurred())
	
				By("checking if fission namespace is created")
				_, err = utils.GetNamespace(clientSet, FISSION_NAMESPACE)
				Expect(err).NotTo(HaveOccurred())
			})
		})
	})

	return "";
}

[onsi]

i can reply in a bit more detail later but this section of the docs has some discussion of this:

https://onsi.github.io/ginkgo/#dynamically-generating-specs

your example will work but you will need to pass variables like clientSet in.

I think it could be helpful to understand why defining these in a different file is an important design consideration (eg is it for reuse? so you can do the same thing in multiple different tests?)

i'll have more later today but i'm on my phone and wanted to share at least that doc link

[AdarshdeepCheema]

I am getting error when tried to use above concept

Ginkgo detected an issue with your spec structure
var _ = Describe("Control Plane Opearator", Ordered, Label("controlplane", "operator", "test 02"), func() {
/Users/adarshdeepsinghcheema/Desktop/Work_Repo/testing/controlplane-operator/tests/e2e/nsTest/02_call_namespace.go:19
  It looks like you are trying to add a [Container] node
  to the Ginkgo spec tree in a leaf node after the specs started running.

  To enable randomization and parallelization Ginkgo requires the spec tree
  to be fully constructed up front.  In practice, this means that you can
  only create nodes like [Container] at the top-level or within the
  body of a Describe, Context, or When.

  Learn more at:
  http://onsi.github.io/ginkgo/#mental-model-how-ginkgo-traverses-the-spec-hierarchy

https://onsi.github.io/ginkgo/#dynamically-generating-specs this is not what we are looking for. we do not want to reuse a spec, we want to keep them separated in different files.

We want something like: File1 with Spec_1 to Spec_30 specs and File2 with Spec_31 to Spec_70 specs and so on til FIle_N with Spec_N to Spec_M
Here number of files can be from 10-20, with 5-30 specs in each.

Each file will generate some Workloads like pods, PVCs, replicasets,and CRDs via BeforeALL and then we will run specs, written in the same file, to test what was generated via BeforeALL section. We want to divide the application of workloads and testing those workloads in small and sequential tasks.

If we decide to bring everything up at once and only run all the tests later randomly. Then in this case because some workloads are dependent on others and we will need to set wait for 20-30 mins to allow enough time for all workloads to complete. We do not want to wait for 20-30 mins before initiating the tests.

Suppose in File13 we want to verify the status of pvc_13, pods_13 or replicasets_13 that will not run or complete and keep on failing unless the things created in File1-file12 are completed without error. That is why we want to run the tests in a specific order and this is how we want to divide the tests.

[onsi]

@AdarshdeepCheema I'd be happy to join a zoom call to have a face-to-face conversation about this if you would like. You can send me an e-mail at onsijoe@gmail.com and we can arrange a time - I am in Denver, so Mountain Time Zone. I have structural concerns with the approach you are trying to take and feel a face-to-face conversation would help me better surface that actual underlying problem you are trying to solve and show you how I might solve it with Ginkgo.

[onsi]

In case there's interest, please take a look at the new proposal for managing shared resources and having finer-grained control over parallelism here: #1292

[farazkhawaja]

Need to carry out a simple CRUD test operation on k8s resource for eg; configmap.

can i have 4 it blocks with separate op in each block? or do i need beforeeach aftereach functions to carry out the CRUD test