View source for ActivityPub/Primer/Testing

Automated and semi-automated testing of ActivityPub implementations can improve the reliability of the entire network. Here are some guidelines for testing various aspects of ActivityPub software.

== API server testing ==

Because the Social API is specified as a RESTful Web API, it is amenable to automated testing with a number of standard API testing tools.

A typical unit test for the Social API would start an API server on <code>localhost</code> or another server and make HTTP requests to it, and check that the results are compliant with the specification.

Another option would be to use containerization tools like Docker or Kubernetes to configure a test environment, and run automated tests against it.

Some tools for testing APIs in different programming languages:

* [https://www.npmjs.com/package/supertest supertest] for NodeJS
* ...

One challenge for automated server testing is that ActivityPub depends heavily on HTTPS for API calls. Calls to a server on a containerized local environment, or running on localhost, may generate errors due to validation of the SSL cert, which might be self-signed. Ideally, the unit testing environment should include an option to turn off SSL cert validation.

* In NodeJS, setting the environment variable <code>NODE_TLS_REJECT_UNAUTHORIZED</code> to <code>0</code> will disable SSL checks throughout the whole process. NodeJS will emit a warning the first time it makes an HTTPS request with this environment variable set.
* In Python, with the <code>requests</code> HTTP client package, passing <code>verify=false</code> as a parameter will disable SSL checks.

== API client testing ==

The Social API, as a RESTful API, is also relatively easy to test from the client side.

The typical testing pattern is to use a [https://en.wikipedia.org/wiki/Mock_object mock object] to stand in for the ActivityPub API server, and run client-side unit tests.

Another option is to use a throw-away ActivityPub API server just for testing. However, it's important in testing that the tests are idempotent; that is, running the same test suite twice doesn't give different results. One mechanism is for a test to clean up any effects it has (for example, deleting any objects it creates). Another option is clearing the server's data storage before each test run, so that the tests run against an identical environment each time.

Mock object frameworks for different programming languages:

- TBD

ActivityPub API servers that may work well for testing:

* [https://github.com/evanp/onepage.pub onepage.pub]
* [https://pleroma.social/ Pleroma]

Again, SSL cert validation provides a stumbling block for API client testing. API clients should have an option to disable cert validation, but even in this case it can be hard to test correct behaviour when a real cert validation issue arises.

== Self-federation protocol testing ==

Another step in testing ActivityPub implementations is testing the federation protocol, if it's implemented. The easiest way to test this functionality is testing federation between two instances of the same implementation.

A common pattern is to start two instances of the software, either on separate servers, or on different ports on <code>localhost</code>. Containerization technologies like Docker or Kubernetes can also be useful for this stage of testing.

The unit test code can use the ActivityPub API, or another API, to create activities in one implementation, and then use an API to check if the correct side effects have occurred.

A difficult part of this kind of testing is that most implementations include a queueing mechanism, so that changes made due to an API call may not send an activity across the federation protocol for some period of time. Unit tests can either inspect the state of the queue, if an API is provided for that, or they can wait for a fixed period of time -- typically on the order of seconds -- before checking for side effects. They can also repeat checks for a fixed number of repetitions.

Federation also requires calls using HTTPS, so similar issues with SSL cert validation will occur if <code>localhost</code> or containerized servers are used. In self-federation protocol testing, it should be possible to disable HTTPS cert validation.

Lastly, in containerized environments, routing for server names is necessary. To make one server available to another in a Docker enviroment, [TBD].

== Integrated federation protocol testing ==

Integrated federation testing is testing an implementation against another implementation to ensure that activities generated by one will cause the expected side effects in another.

Integrated federation testing is similar to self-federation testing in that one starts two implementations, and uses APIs to create activities and measure the effects. Especially in this situation, containerization such as Docker or Kubernetes or Helm can help.

However, it can be complicated by the fact that different implementations may have different APIs, which can increase the code burden in unit test code.

Because Mastodon is the most popular ActivityPub implementation, many implementers will test against Mastodon first, and will only occasionally test with other implementations.

Federation also requires calls using HTTPS, so similar issues with SSL cert validation will occur if <code>localhost</code> or containerized servers are used. In integration protocol testing, it may be hard or impossible to disable cert validation in the other implementation. To disable cert validation in Mastodon, [TBD].

Lastly, in containerized environments, routing for server names is necessary. To make one server available to another in a Docker enviroment, [TBD].

== Compliance testing ==

Formalized compliance testing in ActivityPub has been, traditionally, self-reported. [https://activitypub.rocks/ activitypub.rocks] includes a script for a manual self-reported test at [https://test.activitypub.dev/ test.activitypub.dev], however, it is currently down as of 2023.

An additional formal compliance test suite has been created by Steve Bate at https://github.com/steve-bate/activitypub-testsuite .

== Alpha or beta testing ==

Finally, alpha or beta testing of live services is necessary to test an implementation in real-world conditions. Some rules of thumb for live testing:

* At least initially, use test accounts on other servers, rather than real people's accounts. 
* Well-known ActivityPub implementers, such as the Mastodon or Pleroma team, did not volunteer to be testing subjects for your beta tests. Please be considerate with generating test activities for these real human people.
* Bulk testing, like creating 100,000 new Note objects, can overwhelm other federated servers.
* Once your server is federating with other servers, links to objects, actors, activities, etc. on your server are stored in those other systems. Changing URL endpoint templates, mass-deleting objects or actors, or other major changes can be destabilizing for these other systems. Once a system is live on the fediverse, you should treat it as a production system.