Unit test

[awesomejay]

Need unit testing framework for REST URIs. Here is a script that I created to test streaming with its options: https://github.com/rweigel/datacache/blob/master/repeat.sh
Please propose how we should do this. Ideally we could also have an option to monitor certain unit tests for timing and send a report. Here is something that I threw together a while back: https://github.com/rweigel/urlwatch

[awesomejay]

I think there are several concerns here and they require different types of tests.

tl;dr To me these testing tasks would better be separated.

For correctness of code, we need unit tests. We need to break code into smaller components which focus on small tasks. Then we can write unit tests to make sure these small unit of code are logically correct. Unit tests should be fast and deterministic. So they should be run every time when new code comes in to ensure new code doesn't break existing code. The component under test should be put into a mock environment (like fake HTTP request) to produce fast and deterministic results.

To ensure these components actually work correctly together in a real world environment, we need the so-called system test. In this test we treat the system as a black box and use scripts to simulate how a user would use our system. And we can profile the performance during this test as well. System test doesn't need to run as often and usually is triggered manually.

To check whether the server is still alive, we can just periodically probe a certain URL ( /index.html should be good enough). We don't need to test the whole URL set. I guess there might be some free service out there we can use to monitor the server.

[rweigel]

I am not too worried about the system tests right now.

Here are the types of tests we need:
(1) Test all of the demos with forceUpdate=false (say every 5 minutes)
(2) Test all of the demos with forceUpdate=true&forceWrite=true (every hour)
(3) Test all of the URLs that caused a crash in the past (every hour)

After each commit, test (2) and (3). I suggest that we create a file with URLs in the repo, say urlwatch.txt. Then I'll grab this file before each url check with urlwatch.

Most of these can be done by just hitting a list of URLs. The urlwatch code also has options for checking the MD5; I think this is needed in some cases where it is important that the response is identical.

After each major change, run a stress test. We'll need to set up a few remote servers with data files of various sizes, delete the cache and then start hitting datacache with requests.

[awesomejay]

OK I see. I think these are system tests thou, in the sense that they treat
the system as a blackbox.

Why do you want to run these tests so frequently? If we don't checkin new
code, these test results should stay the same unless upstream servers are
down.

On Sat, Jun 22, 2013 at 4:38 AM, rweigel notifications@github.com wrote:

I am not too worried about the system tests right now.

Here are the types of tests we need:
(1) Test all of the demos with forceUpdate=false (say every 5 minutes)
(2) Test all of the demos with forceUpdate=true&forceWrite=true (every
hour)
(3) Test all of the URLs that caused a crash in the past (every hour)

After each commit, test (2) and (3). I suggest that we create a file with
URLs in the repo, say urlwatch.txt. Then I'll grab this file before each
url check with urlwatch.

Most of these can be done by just hitting a list of URLs. The urlwatch
code also has options for checking the MD5; I think this is needed in some
cases where it is important that the response is identical.

After each major change, run a stress test. We'll need to set up a few
remote servers with data files of various sizes, delete the cache and then
start hitting datacache with requests.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-19855382
.

[awesomejay]

Maybe we can use Mocha: http://visionmedia.github.io/mocha/

So the test code will look like this:

// steam_test.js

var TEST_URLS = [
    "http://datacache.org/dc/demo/file1.txt",
    "http://datacache.org/dc/demo/file2.txt"
]

describe("/sync", function()){
    it("should success for URL" + TEST_URLS[0], function(){
        request("http://localhost:8000/sync?source="+TEST_URLS[0], function(res){
            assert(res.statusCode).equalsTo(200);
        })
    })
}

And you can run tests by:

// in datacache folder
npm install -g mocha
mocha 

What do you think?

[rweigel]

This is a light-weight test. We really need to be testing timing and md5
in addition to HTTP response code. I find that with this async code,
sometimes there are surprises that only show up once every 10 hits or only
when network conditions are unusual.

Take a look at the urlwatch project. It does just about everything I
need. It lets me know when the md5 has changed and when something is
taking too long.

On Sun, Jun 23, 2013 at 4:15 PM, awesomejay notifications@github.comwrote:

Maybe we can use Mocha: http://visionmedia.github.io/mocha/

So the test code will look like this:

// steam_test.js

var TEST_URLS = [
"http://datacache.org/dc/demo/file1.txt",
"http://datacache.org/dc/demo/file2.txt"
]

describe("/sync", function()){
it("should success for URL" + TEST_URLS[0], function(){
request("http://localhost:8000/sync?source="+TEST_URLS[0], function(res){
assert(res.statusCode).equalsTo(200);
})
})
}

And you can run tests by:

// in datacache folder
npm install -g mocha
mocha

What do you think?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-19880341
.

[awesomejay]

We can add more assertions like:

    request("http://localhost:8000/sync?source="+TEST_URLS[0], function(res){
        assert(res.statusCode).equalsTo(200);
        assert(md5(res.body)) == ...)
    })

But is this framework what you are looking for? Or maybe you just want to add on top of urlwatch?

[rweigel]

I suggest looking into integrating this with URLWatch. I have a very
rudimentary assert check that should be improved.

On Sun, Jun 23, 2013 at 8:32 PM, awesomejay notifications@github.comwrote:

We can add more assertions like:

request("http://localhost:8000/sync?source="+TEST_URLS[0], function(res){
    assert(res.statusCode).

equalsTo(200);
assert(md5(res.body)) == ...)
})

But is this framework what you are looking for? Or may you just want to
add on top of urlwatch?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-19884592
.

[awesomejay]

I just uploaded a patch 465c0f4 which added a testing framework.

I wasted a little bit time on Mocha and found it not flexible enough to meet our needs. So I wrote our own. You can run the test by

// under datacache directory
npm install
npm test

// or 
npm install
node test/main.js

[rweigel]

I think the next step is to translate the tests in repeat.sh and repeat2.sh
to this framework.

Inputs:

• Number of times to test
• Time between launching batch of tests
• Run each test in batch async = true/false
• Pre-process commands (e.g., cache directory or subdirectory)
• URL to test
• Post process pipe (e.g., sort, md5, head, tail)
• Async #'s. (Not a high priority) Run this batch of tests at the same
  time as other test given in list. Otherwise run all batches of tests
  serially.
• Expected md5 (if not given create one based on first result)
• Expected file size (if not given create one based on first result)
  or
• Expected equivalence test number (md5 and/or file size of this result
  should be same as that of test number)

Outputs:

• md5
• file size
• times for each test in batch

Ideally I would be able to get a text listing of the test results, e.g.,
http://datacache.org/dc/tests?testNumber=2 would give rows with columns of

Universal time of test launch, md5, file size, pass (0 or 1), times for
each test in batch

and

http://datacache.org/dc/tests?testNumber=2&output=all would give

Universal time of test launch, md5, file size, times for each test in
batch, async, preprocess commands, URL, postprocess commands, expected md5,
expected file size, async #'s.

I tried installing your patch:

npm test

datacache@0.5.0 test /home/weigel/git/datacache
node test/main.js

dir /home/weigel/git/datacache/test

buffer.js:434
throw new Error('Unknown encoding');
^
Error: Unknown encoding
at Buffer.toString (buffer.js:434:13)
at Object.fs.readFileSync (fs.js:236:33)
at Object.
(/home/weigel/git/datacache/test/lib/testReporter.js:4:25)
at Module._compile (module.js:449:26)
at Object.Module._extensions..js (module.js:467:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:362:17)
at require (module.js:378:17)
at Object.
(/home/weigel/git/datacache/test/lightTests.js:4:19)
npm ERR! Test failed. See above for more details.
npm ERR! not ok code 0

On Sun, Jun 30, 2013 at 11:53 PM, awesomejay notifications@github.comwrote:

I just uploaded a patch 465c0f4465c0f4d6cf360619da33ed9d70417889f2e531ewhich added a testing framework.

I wasted a little bit time on Mocha and found it not flexible enough to
meet our needs. So I wrote our own. You can run the test by

// under datacache directory
npm install
npm test

// or
npm install
node test/main.js

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-20262356
.

[awesomejay]

I just fixed the error. The problem was they changed readFileSync(path, "utf8") to readFileSync(path, {encoding: "utf8" }) in Node vesion0.9.11 .

I can't believe they made such a change and just put a vague line in the changelog:

"fs: Support mode/flag options to read/append/writeFile (isaacs) "

I will use version0.8.16 to develop in the future.

[awesomejay]

Don't be surprised if you run the test and see it reports a failure in testing URL "http://www.google.com".

It is because timeout is set to 1: https://github.com/rweigel/datacache/blob/master/test/lightTests.js#L26

[rweigel]

worse: it would have been trivial to add the new flag options and still
support the old syntax.

On Wed, Jul 3, 2013 at 2:12 AM, awesomejay notifications@github.com wrote:

I just fixed the error. The problem was they changed readFileSync(path,
"utf8") to readFileSync(path, {encoding: "utf8" }) in Node vesion0.9.11 .

I can't believe they made such a change and just put a vague line in the
changelog:

"fs: Support mode/flag options to read/append/writeFile (isaacs) "

I will use version0.8.16 to develop in the future.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-20397471
.

[awesomejay]

Update:

1. Added support to run tests concurrently.
2. Ported repeat2.sh to test/repeat2.js. It can be run by:

// under datacache folder
node test/repeat2.js

1. Added a brief document for the testing framework: https://github.com/rweigel/datacache/wiki/Testing-framework

[rweigel]

Looks good and worked for me.

On Sun, Jul 7, 2013 at 10:59 PM, awesomejay notifications@github.comwrote:

Update:

1. Added support to run tests concurrently.
2. Ported repeat2.sh to test/repeat2.js. It can be run by:

// under datacache folder
node test/repeat2.js

1. Added a brief document for the testing framework:
   https://github.com/rweigel/datacache/wiki/Testing-framework

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-20583469
.