22:23 < sam_moore> For the cost estimate: Yes, we are supposed to have a technical diary (hand written)
22:23 < sam_moore> Including all notes and times worked on the project
22:40 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 22.0/20130618035212]"]
+--- Day changed Fri Aug 16 2013
+09:48 < jtanx> I was wondering what sort of model you have for reading/storing sensor values and controlling actuators
+10:02 < sam_moore> At the moment each sensor has a thread that continuously polls the sensor and dumps data to a binary file
+10:03 < sam_moore> For reading there is a thread that fills a buffer from the file when it gets a request
+10:04 < sam_moore> It might be better to change to a single thread for sensors/actuators though
+10:04 < jtanx> yeah, I was about to say if it was necessary to have one for each
+10:05 < sam_moore> Probably not, if you had varying polling speeds it would be useful, because you could poll the fast sensors seperately from the slow ones
+10:05 < jtanx> how do you wish to pass the data to the fcgi module?
+10:06 < jtanx> right now
+10:06 < jtanx> i've kinda set it so
+10:06 < jtanx> there's this thing called a ModuleHandler
+10:07 < jtanx> so you can pass it some data (tba what this is), and it also receives the query string from the use
+10:09 < sam_moore> Well, the ModuleHandler gets called whenever the right HTTP request is made right?
+10:10 < sam_moore> So, that function retrieves the data it needs to respond to the request
+10:10 < jtanx> well rightnow the typedef is
+10:10 < jtanx> typedef void (*ModuleHandler) (Data *data, char *params);
+10:10 < jtanx> so it's up to you to write a module handler (eg SensorsHandler)
+10:11 < sam_moore> Ok, data's a bit of an ambigous name, so is that something passed to the ModuleHandler when the FastCGI gets a request?
+10:11 < jtanx> well the data thing was a placeholder for whatever sensors/actuators data there was
+10:11 < sam_moore> Is it meant to be sensor data, or is it a value that the user has passed through HTTP
+10:11 < jtanx> the query string is part of params
+10:11 < sam_moore> Ok, right, that makes more sense
+10:12 < sam_moore> My view had the ModuleHandler getting the appropriate data itself
+10:12 < jtanx> Ok
+10:12 < sam_moore> Given the params
+10:13 < jtanx> I'm just not really used to using global variables
+10:13 < sam_moore> The global sensors array?
+10:13 < jtanx> yeah
+10:13 < jtanx> i take it that's where you access data from?
+10:14 < sam_moore> Yes
+10:14 < jtanx> oh yeah one more thing, when you get a request, should it only return the latest data?
+10:20 < sam_moore> We should probably think some more about this as a whole group
+10:20 < sam_moore> So, you have a sensor that can (probably) poll a lot faster than you get requests via jQuery
+10:21 < sam_moore> You can write it so that you only actually query the sensor when you get a request, but then you're wasting the chance to get data that could be useful
+10:22 < sam_moore> Or you can have it so that the sensor is continuously polling (What my code is simulating at the moment)
+10:22 < sam_moore> With the continuosuly polling sensor; when you get a request, you can either return the most recent block of data you got
+10:23 < sam_moore> Or you can have it so that the data will be sent in sequential order (which is what my code currently does)
+10:23 < jtanx> hmm
+10:23 < sam_moore> I'm pretty sure continously polling is better than querying each time you get a request, since after all this is an experiment and you want the data
+10:23 < jtanx> yeah
+10:24 < jtanx> I agree with you there
+10:24 < jtanx> so you're saying, continuously poll and log the results
+10:24 < jtanx> and when a request comes in, send back all the data up to the point when the last request came in
+10:25 < sam_moore> Well that's what the server program I wrote simulates at the moment.
+10:25 < sam_moore> However it might be more sensible for the request to just get the most recent data.
+10:25 < jtanx> hmm
+10:25 < sam_moore> Ah, yeah that's probably better actually
+10:25 < jtanx> then you're logging more points than youdisplay
+10:26 < sam_moore> Yes, but you can always add a function to pull all the data points and save to a csv or something like that.
+10:26 < sam_moore> In fact I think we'd want to do that.
+10:27 < jtanx> yeah ok
+10:27 < jtanx> that sounds not too bad
+10:27 < jtanx> but if that's the case
+10:27 < sam_moore> If you actually use this for an experiment, you're not going to want to have to do all your analysis in a JavaScript GUI
+10:27 < jtanx> true
+10:27 < sam_moore> The GUI is for getting live information on the state of the system more than for data analysis
+10:27 < jtanx> but if that's the case you might want to allow direct access to the latest dataset
+10:28 < jtanx> instead of having to read from the file
+10:28 < jtanx> if that's possible
+10:28 < jtanx> then you can directly record the results in csv format
+10:28 < sam_moore> Yes, I thought about that; you can have a buffer of points accessed by the request thread easily enough
+10:29 < sam_moore> There are some difficulties though with managing the buffer
+10:30 < sam_moore> Wait, maybe it's easier than I though
+10:30 < sam_moore> Yeah, the reason I started saving stuff to a binary file was because I was thinking of the requests having to get the data in sequence
+10:32 < sam_moore> I have some other stuff to do today unfortunately
+10:33 < sam_moore> I have a feeling it won't be trivial to just access the most recent dataset, I'll have to think about it
+10:34 < sam_moore> However, currently it should be easy to change so that the request gets data from the end of the binary file, rather than keeping track of its position
+10:34 < sam_moore> The function QuerySensor is what I was thinking the SensorHandler would do to get data; it just fills a buffer and prints it at the moment
+10:35 < sam_moore> Binary file access is pretty fast, we could even just keep the binary file and change the data to csv when it goes to the client
+10:36 < jtanx> ok, I'll keep that in mind
+11:11 < sam_moore> Oh, you can replace the Data* pointer with a Sensor* pointer and then not have to use the global variable directly, that's probably best
+11:13 < jtanx> well it might not only be sensor data
+11:13 < jtanx> it could be actuator stuff or anything else
+11:13 < sam_moore> I suppose
+11:14 < jtanx> maybe I should change the typedef to void*
+11:14 < jtanx> then you can cast it to whatever
+11:14 < sam_moore> Yeah, that will work
+11:15 < sam_moore> Actuator stuff is going to be a bit of a seperate (hopefully easier) problem
+11:15 < sam_moore> In that case you can just wait for a query before doing anything
+11:15 < sam_moore> Anyway, I've distracted myself again, this is just too interesting :S
+11:15 < jtanx> hahaha
+11:16 < jtanx> anycase for the actuator you would have a separate handler function
+11:16 < sam_moore> Yes
+11:16 < jtanx> eg ActuatorHandler
+11:20 < jtanx> fcgi is pretty whack
+11:20 < jtanx> it replaces printf
+11:21 < jtanx> so when you printf (or fwrite for binary?), that gets sent to the client
+12:03 < jtanx> hey
+12:04 < jtanx> what if we stored stuff in an sqlite library
+12:04 < jtanx> sorry database
+12:04 < jtanx> on second thoughts nah
+15:02 -!- jtanx_ is now known as jtanx
+18:59 -!- jtanx_ is now known as jtanx
+21:28 < sam_moore> So, I did a bit of testing with sqlite vs a binary file
+21:28 < sam_moore> sqlite takes about 1000x as long for saving data
+21:28 < sam_moore> I haven't tested reading it back yet
+21:28 < sam_moore> But more worrying
+21:28 < sam_moore> I get a segmentation fault using sqlite
+21:29 < sam_moore> And it's in the sqlite library somewhere; not my test code
+21:29 < sam_moore> Running sqlite in valgrind shows a bunch of complaints about uninitialised values
+21:29 < sam_moore> So... I'd recommend we just use a binary file
+21:30 < sam_moore> A database is good for storing more complex data, but when you just need to store data recorded in a sequence, it's probably unnecessary
+21:36 < jtanx> yeah sqlite not so good for file performance
+21:36 < jtanx> because every insert it has to confirm the write to disk
+21:36 < jtanx> the segfault
+21:36 < jtanx> may actually be because you didn't initialse the mutex
+21:36 < jtanx> es
+21:36 < jtanx> i tried compiling it on mingw and it segfaults on the mutex unlock
+21:37 < sam_moore> Re: fcgi replacing printf - It probably calls dup2 to change stdout from the terminal to a domain socket or fifo that nginx listens to
+21:37 < jtanx> nah
+21:38 < jtanx> fcgi_stdio just has a bunch of defines
+21:38 < sam_moore> Oh really, that's wierd
+21:38 < jtanx> so printf becomes FCGI_printf
+21:38 < jtanx> so fcgi_stdio must be the first include in the fcgi module
+21:38 < sam_moore> I would have thought it was simpler to just change stdout than replace the whole printf function
+21:38 < sam_moore> But whatever, we're not implementing fcgi :P
+21:38 < jtanx> haha
+21:39 < sam_moore> Oh, the mutex needs initialising, yeah
+21:39 < sam_moore> But the test I did was just a serial program, no mutex
+21:39 < jtanx> ah
+21:39 < jtanx> well
+21:39 < sam_moore> We'll be using gcc to compile on the beaglebone/rpi
+21:40 < sam_moore> I think some implementations of pthreads need an initialiser but others don't, I'll have to check
+21:40 < sam_moore> But pthread_mutex_init doesn't show up in man pages on debian
+21:40 < jtanx> yeah it's not on ubuntu either
+21:41 < jtanx> have you ever worked with clang
+21:42 < sam_moore> No
+21:42 < jtanx> neither but it seems pretty cool
+21:45 < jtanx> when you compile it gives better diagnostic output when something goes wrong
+21:46 < jtanx> it's a drop in replacement for gcc
+21:47 < sam_moore> Interesting
+21:57 < jtanx> if you kept the file handles for the sensor data always open
+21:58 < jtanx> and immediately wrote to them when a data point is polled
+21:58 < jtanx> is there much performance difference to buffering first?
+22:05 < sam_moore> Yeah, it's probably better to keep the file handles always open
+22:05 < sam_moore> The operating system will use some kind of buffer for the file anyway
+22:06 < sam_moore> I've got some basic programs, maybe I'll make some performance graphs tomorrow
+22:07 < sam_moore> On the other hand, I need to do some of the meta stuff (list of tasks that need to be completed, etc) before Monday
+22:08 < sam_moore> I should look into a #define or something to initialise the mutexes if they need to be
+22:09 < sam_moore> Anyway, I need to sleep
+22:11 < sam_moore> I wonder how James and Callum are going, they haven't been in the channel for a while
+22:12 < jtanx> yeah, it's pretty quiet
+22:17 < sam_moore> If we can get a second meeting in the week, preferably a longer one that would be good
+22:18 < sam_moore> Particularly if we can start working on code at the same time
+22:18 < sam_moore> But we'll see
+22:18 < sam_moore> Everyone has other stuff to do after all
+22:18 < sam_moore> I'm spending way too much time on this unit compared to my others
+22:18 < sam_moore> Then again this unit will probably require the most work
+22:19 < sam_moore> Anyway, bye
+22:23 < jtanx> bye
+22:46 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 22.0/20130618035212]"]
+--- Day changed Sat Aug 17 2013
+22:25 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 22.0/20130618035212]"]
+--- Day changed Sun Aug 18 2013
+21:53 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+22:37 -!- Callum [
[email protected]] has quit ["ChatZilla 0.9.90.1 [Firefox 22.0/20130618035212]"]
+--- Day changed Mon Aug 19 2013
+11:43 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+13:34 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+20:07 < jtanx> just so you know, I'm changing the JSON functions
+21:04 < sam_moore> Ok, I already pushed some stuff to github
+21:05 < sam_moore> But we can get it to merge
+21:05 < sam_moore> I ended up just using printf to make part of the JSON in the SensorHandler anyway
+21:06 < jtanx> depending on where SensorHandler is
+21:06 < jtanx> that may or may not work
+21:06 < jtanx> inside of fastcgi.c it willwork
+21:06 < jtanx> outside it won't
+21:07 < jtanx> oh ok
+21:07 < jtanx> I see
+21:07 < jtanx> so that's fine
+21:08 < jtanx> it might be best not to use a do while loop
+21:08 < jtanx> because if no arguments are passed
+21:08 < sam_moore> Ah, right
+21:08 < jtanx> then the keypair function will return null
+21:09 < jtanx> while ((params = FCGI_KeyPair(params, &key, &value)))
+21:09 < jtanx> will work fine
+21:10 < sam_moore> The KeyPair function looks like it can return an empty string though, in which case I was getting the sensor_id parsed twice when I was testing it
+21:10 < jtanx> what was your input string?
+21:11 < sam_moore> "http://localhost/api/sensors?id=0"
+21:11 < jtanx> looks fine from this end
+21:11 < jtanx> http://mctx.us.to:8080/api/sensors?id=0
+21:11 < jtanx> it might be because of your do while loop
+21:12 < jtanx> yeah it will be because of that
+21:12 < sam_moore> The do while loop causes problems when there is an empty string
+21:13 < sam_moore> ie: No parameters are passed
+21:13 < jtanx> ok let's just put it this way; FCGI_KeyPair was designed to use a while loop
+21:13 < sam_moore> Ok, sure
+21:13 < sam_moore> I had some problems with just a while loop, but I'll try again
+21:13 < jtanx> yeah about that JSON stuff
+21:13 < jtanx> I'm still trying to think of a good way to do that
+21:13 < jtanx> especially with the array stuff
+21:19 < sam_moore> I'm not sure what I did before, but a while loop seems ok now
+21:19 < jtanx> heh
+21:21 < sam_moore> Ok, I have to go now
+21:21 < sam_moore> I might not be able to do much tomorrow, we'll see
+21:22 < jtanx> ok yep
+21:35 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+22:46 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+23:14 -!- Callum [
[email protected]] has quit ["ChatZilla 0.9.90.1 [Firefox 23.0.1/20130814063812]"]
+23:35 -!- Callum [
[email protected]] has quit ["ChatZilla 0.9.90.1 [Firefox 23.0.1/20130814063812]"]
+--- Day changed Tue Aug 20 2013
+15:06 < Callum> hey same
+15:06 < Callum> sam*
+15:20 < jtanx> :>
+15:22 < Callum> he literally set himself to away as soon as i said that
+15:45 < jtanx> urgh this json stuff is doing my head in
+15:57 < sam_moore> Callum: I've been set to away since last night
+15:57 < Callum> o.O not according to my client
+15:58 < sam_moore> Anyway, we are definitely using the Beaglebone, Electronics has ordered one, they seemed to think they'd have to program it but I corrected them
+15:58 < Callum> ...why would they have to program it? wtf did they think we were doing?
+15:59 < sam_moore> They probably thought all we had to do was the GUI and not any of the other software? I don't know. But the guy seemed relieved anyway.
+15:59 < Callum> and was gonna ask if you remembered anything about the pi meson decay question for physics (how do you get the anti-neutrinos momentum, or do you assume it to be 0 as well? but then it doesnt really make sense with conservation of momentum)
+16:00 < sam_moore> Woah, I don't remember the assignments in that much detail
+16:00 < Callum> was hoping you would :p
+16:00 < sam_moore> No, I just have vague memories of algebra
+16:01 < sam_moore> And looking through notes
+16:01 < Callum> hmm. i cant seem to find anything in the notes
+16:01 < Callum> because he says to assume the rest mass of the anti-neutrino is 0
+16:02 < Callum> and i don't really remember much in regards to energies and 0 rest mass ;/
+16:02 < jtanx> I talked to someone who did computer vision before
+16:02 < jtanx> and he's really doubtful that you can get accurate readings off a webcam
+16:02 < sam_moore> Callum: I don't think you can assume the momentum is zero, other than that, I don't have much to offer
+16:02 < Callum> the question also says to have it in terms of m(pi) m(e) and c
+16:03 < Callum> and yea thats what i thought. it doesnt make sense to do that. just cant remember how to do it D;
+16:03 < Callum> @jeremy yea well, its a pain but possible
+16:04 < Callum> really comes down to how good your camera is/sampling rate/how quickly you can process it.
+16:04 < sam_moore> How about you do some experimenting with a webcam and see what you can do with it?
+16:04 < Callum> but you can get some pretty good webcams nowadays (but then again the better it is the longer it takes to process)
+16:04 < jtanx> personally, I don't think it will work
+16:04 < sam_moore> It looks like we might just end up streaming images diretly to a website
+16:04 < Callum> i don't have any
+16:04 < Callum> yea well even if thats all we do we still need the camera
+16:05 < Callum> spose i could use my laptop one but i doubt that would be very good
+16:05 < Callum> could run it through the canny algorithm and see what it produces
+16:06 < sam_moore> Sounds like an idea
+16:07 < sam_moore> A good idea specifically
+16:07 < jtanx> about the sensorhandler
+16:07 < sam_moore> Yes?
+16:07 < jtanx> do you envision leaving it in fastcgi.c permanently
+16:07 < jtanx> or for that matter any other handlers
+16:07 < sam_moore> I was kind of thinking there would be a "handlers.h" and "handlers.c"
+16:08 < sam_moore> Just to make things more organised
+16:08 < jtanx> yeah
+16:08 < jtanx> I'm trying to export enough of the functionality
+16:08 < jtanx> to do that
+16:08 < jtanx> but the json thing is annoying
+16:08 < jtanx> especially when you need to spit out arrays
+16:09 < jtanx> unless you have something like FCGI_Printf
+16:09 < jtanx> and it's up to you to format it correctly
+16:09 < Callum> bloody physics lecture video is laggy. gonna have to download it and hope for the best. fuck echo
+16:10 < jtanx> compared to lectopia you get 2x filesize with no visible benefit in quality
+16:10 < Callum> they're both shit.
+16:10 < sam_moore> You could have seperate "BuildJSON_Key" and "BuildJSON_Value" functions, with a "BuildJSON_ValueArray" maybe
+16:10 < Callum> haha
+16:11 < sam_moore> FCGI_Printf is Ok though, it's not too much formating
+16:11 < jtanx> the problem with the buildjson_* stuff
+16:11 < jtanx> is it gets very verbose
+16:12 < jtanx> and you can probably come up witha situation that breaks it too
+16:12 < jtanx> and don't get me started on value types
+16:12 < sam_moore> Haha
+16:14 < sam_moore> We can always just send plain text and make James turn it into JSON :P
+16:14 < jtanx> ahaha
+16:14 < jtanx> yeah that could work
+16:15 < jtanx> mm
+16:15 < jtanx> this is where java is good
+16:15 < jtanx> or any other higher level language
+16:21 < jtanx> ok so it's a bit of both, but how about: FCGI_JSONKey(key) and FCGI_JSONValue(format, ...)
+16:22 < sam_moore> That looks good
+16:25 < jtanx> I'm also adding long/double types for the BuildJSON function, just for convenience
+16:25 < jtanx> any preference to naming conventions?
+16:26 < jtanx> FCGI_BuildJSONLong
+16:27 < sam_moore> Seems OK
+16:28 < sam_moore> I need to go do some ENSC1001 stuff (hooray)
+16:29 < jtanx> yuck
+16:30 < Callum> hhaha
+16:30 < Callum> have fun :p
+16:30 < Callum> although i cant really talk. about a days worth of physics and a days worth of ensc3016 shit i should get through, on top of mctx and geng4402 stuff. yay for being behind already
+16:38 < Callum> well iv got an answer for the first part (second part should be the same process). just hope its right :s
+16:46 < jtanx> ok, time to merge this into the server code
+19:47 < jtanx> hmm interesting - it crashes if compiled with clang but not with gcc
+19:49 < jtanx> probably just a bug in clang 3.2
+20:08 < jtanx> ok, just submitted a pull request to update the fastcgi stuff
+20:08 < jtanx> for now I moved Sensor_Handler to sensor.c
+20:09 < jtanx> the status codes have all changed
+20:09 < jtanx> If you absolutely cannot process anything given the input arguments, you call FCGI_RejectJSON
+20:10 < jtanx> If you fail for some other reason (e.g unauthorized access), you use FCGI_BeginJSON with the appropriate status code
+20:10 < jtanx> With RejectJSON, it sends a HTTP 400 code so any query through AJAX/jQuery will fail with no extra info
+20:11 < jtanx> With BeginJSON, the HTTP code is always 200 OK, so you are able to transmit extra info if it failed for another reason
+20:12 < jtanx> BeginJSON will automatically add the module handler name + the status code
+21:47 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+--- Day changed Wed Aug 21 2013
+11:57 < jtanx> hmm
+11:57 < jtanx> I just had a play with the sensor stuff
+11:57 < jtanx> and I trialled a 'double buffer' scheme
+11:57 < jtanx> instead of the binary file idea
+11:57 < jtanx> seems to work okay, and it guarantees that a point won't be returned to the user if they have already received it
+12:35 < jtanx> urgh
+12:35 < jtanx> just worked through some stupid bug
+12:37 < jtanx> I think it's because make didn't recompile something because it thought it hadn't changed
+12:38 < jtanx> probably the header files
+12:50 < jtanx> you can see the double buffer method in this branch: https://github.com/jtanx/MCTX3420/tree/doublebuffer
+12:58 < jtanx> one issue though is that writing out csv instead of binary file takes up a lot more space
+14:49 < james__> Hey
+14:50 -!- james__ [
[email protected]] has quit ["ChatZilla 0.9.90.1 [Firefox 23.0.1/20130814063812]"]
+--- Day changed Thu Aug 22 2013
git.ucc.asn.au / matches / MCTX3420.git / blobdiff
UCC git Repository :: git.ucc.asn.au