22:23 < sam_moore> For the cost estimate: Yes, we are supposed to have a technical diary (hand written)
22:23 < sam_moore> Including all notes and times worked on the project
22:40 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 22.0/20130618035212]"]
+--- Day changed Fri Aug 16 2013
+09:48 < jtanx> I was wondering what sort of model you have for reading/storing sensor values and controlling actuators
+10:02 < sam_moore> At the moment each sensor has a thread that continuously polls the sensor and dumps data to a binary file
+10:03 < sam_moore> For reading there is a thread that fills a buffer from the file when it gets a request
+10:04 < sam_moore> It might be better to change to a single thread for sensors/actuators though
+10:04 < jtanx> yeah, I was about to say if it was necessary to have one for each
+10:05 < sam_moore> Probably not, if you had varying polling speeds it would be useful, because you could poll the fast sensors seperately from the slow ones
+10:05 < jtanx> how do you wish to pass the data to the fcgi module?
+10:06 < jtanx> right now
+10:06 < jtanx> i've kinda set it so
+10:06 < jtanx> there's this thing called a ModuleHandler
+10:07 < jtanx> so you can pass it some data (tba what this is), and it also receives the query string from the use
+10:09 < sam_moore> Well, the ModuleHandler gets called whenever the right HTTP request is made right?
+10:10 < sam_moore> So, that function retrieves the data it needs to respond to the request
+10:10 < jtanx> well rightnow the typedef is
+10:10 < jtanx> typedef void (*ModuleHandler) (Data *data, char *params);
+10:10 < jtanx> so it's up to you to write a module handler (eg SensorsHandler)
+10:11 < sam_moore> Ok, data's a bit of an ambigous name, so is that something passed to the ModuleHandler when the FastCGI gets a request?
+10:11 < jtanx> well the data thing was a placeholder for whatever sensors/actuators data there was
+10:11 < sam_moore> Is it meant to be sensor data, or is it a value that the user has passed through HTTP
+10:11 < jtanx> the query string is part of params
+10:11 < sam_moore> Ok, right, that makes more sense
+10:12 < sam_moore> My view had the ModuleHandler getting the appropriate data itself
+10:12 < jtanx> Ok
+10:12 < sam_moore> Given the params
+10:13 < jtanx> I'm just not really used to using global variables
+10:13 < sam_moore> The global sensors array?
+10:13 < jtanx> yeah
+10:13 < jtanx> i take it that's where you access data from?
+10:14 < sam_moore> Yes
+10:14 < jtanx> oh yeah one more thing, when you get a request, should it only return the latest data?
+10:20 < sam_moore> We should probably think some more about this as a whole group
+10:20 < sam_moore> So, you have a sensor that can (probably) poll a lot faster than you get requests via jQuery
+10:21 < sam_moore> You can write it so that you only actually query the sensor when you get a request, but then you're wasting the chance to get data that could be useful
+10:22 < sam_moore> Or you can have it so that the sensor is continuously polling (What my code is simulating at the moment)
+10:22 < sam_moore> With the continuosuly polling sensor; when you get a request, you can either return the most recent block of data you got
+10:23 < sam_moore> Or you can have it so that the data will be sent in sequential order (which is what my code currently does)
+10:23 < jtanx> hmm
+10:23 < sam_moore> I'm pretty sure continously polling is better than querying each time you get a request, since after all this is an experiment and you want the data
+10:23 < jtanx> yeah
+10:24 < jtanx> I agree with you there
+10:24 < jtanx> so you're saying, continuously poll and log the results
+10:24 < jtanx> and when a request comes in, send back all the data up to the point when the last request came in
+10:25 < sam_moore> Well that's what the server program I wrote simulates at the moment.
+10:25 < sam_moore> However it might be more sensible for the request to just get the most recent data.
+10:25 < jtanx> hmm
+10:25 < sam_moore> Ah, yeah that's probably better actually
+10:25 < jtanx> then you're logging more points than youdisplay
+10:26 < sam_moore> Yes, but you can always add a function to pull all the data points and save to a csv or something like that.
+10:26 < sam_moore> In fact I think we'd want to do that.
+10:27 < jtanx> yeah ok
+10:27 < jtanx> that sounds not too bad
+10:27 < jtanx> but if that's the case
+10:27 < sam_moore> If you actually use this for an experiment, you're not going to want to have to do all your analysis in a JavaScript GUI
+10:27 < jtanx> true
+10:27 < sam_moore> The GUI is for getting live information on the state of the system more than for data analysis
+10:27 < jtanx> but if that's the case you might want to allow direct access to the latest dataset
+10:28 < jtanx> instead of having to read from the file
+10:28 < jtanx> if that's possible
+10:28 < jtanx> then you can directly record the results in csv format
+10:28 < sam_moore> Yes, I thought about that; you can have a buffer of points accessed by the request thread easily enough
+10:29 < sam_moore> There are some difficulties though with managing the buffer
+10:30 < sam_moore> Wait, maybe it's easier than I though
+10:30 < sam_moore> Yeah, the reason I started saving stuff to a binary file was because I was thinking of the requests having to get the data in sequence
+10:32 < sam_moore> I have some other stuff to do today unfortunately
+10:33 < sam_moore> I have a feeling it won't be trivial to just access the most recent dataset, I'll have to think about it
+10:34 < sam_moore> However, currently it should be easy to change so that the request gets data from the end of the binary file, rather than keeping track of its position
+10:34 < sam_moore> The function QuerySensor is what I was thinking the SensorHandler would do to get data; it just fills a buffer and prints it at the moment
+10:35 < sam_moore> Binary file access is pretty fast, we could even just keep the binary file and change the data to csv when it goes to the client
+10:36 < jtanx> ok, I'll keep that in mind
+11:11 < sam_moore> Oh, you can replace the Data* pointer with a Sensor* pointer and then not have to use the global variable directly, that's probably best
+11:13 < jtanx> well it might not only be sensor data
+11:13 < jtanx> it could be actuator stuff or anything else
+11:13 < sam_moore> I suppose
+11:14 < jtanx> maybe I should change the typedef to void*
+11:14 < jtanx> then you can cast it to whatever
+11:14 < sam_moore> Yeah, that will work
+11:15 < sam_moore> Actuator stuff is going to be a bit of a seperate (hopefully easier) problem
+11:15 < sam_moore> In that case you can just wait for a query before doing anything
+11:15 < sam_moore> Anyway, I've distracted myself again, this is just too interesting :S
+11:15 < jtanx> hahaha
+11:16 < jtanx> anycase for the actuator you would have a separate handler function
+11:16 < sam_moore> Yes
+11:16 < jtanx> eg ActuatorHandler
+11:20 < jtanx> fcgi is pretty whack
+11:20 < jtanx> it replaces printf
+11:21 < jtanx> so when you printf (or fwrite for binary?), that gets sent to the client
+12:03 < jtanx> hey
+12:04 < jtanx> what if we stored stuff in an sqlite library
+12:04 < jtanx> sorry database
+12:04 < jtanx> on second thoughts nah
+15:02 -!- jtanx_ is now known as jtanx
+18:59 -!- jtanx_ is now known as jtanx
+21:28 < sam_moore> So, I did a bit of testing with sqlite vs a binary file
+21:28 < sam_moore> sqlite takes about 1000x as long for saving data
+21:28 < sam_moore> I haven't tested reading it back yet
+21:28 < sam_moore> But more worrying
+21:28 < sam_moore> I get a segmentation fault using sqlite
+21:29 < sam_moore> And it's in the sqlite library somewhere; not my test code
+21:29 < sam_moore> Running sqlite in valgrind shows a bunch of complaints about uninitialised values
+21:29 < sam_moore> So... I'd recommend we just use a binary file
+21:30 < sam_moore> A database is good for storing more complex data, but when you just need to store data recorded in a sequence, it's probably unnecessary
+21:36 < jtanx> yeah sqlite not so good for file performance
+21:36 < jtanx> because every insert it has to confirm the write to disk
+21:36 < jtanx> the segfault
+21:36 < jtanx> may actually be because you didn't initialse the mutex
+21:36 < jtanx> es
+21:36 < jtanx> i tried compiling it on mingw and it segfaults on the mutex unlock
+21:37 < sam_moore> Re: fcgi replacing printf - It probably calls dup2 to change stdout from the terminal to a domain socket or fifo that nginx listens to
+21:37 < jtanx> nah
+21:38 < jtanx> fcgi_stdio just has a bunch of defines
+21:38 < sam_moore> Oh really, that's wierd
+21:38 < jtanx> so printf becomes FCGI_printf
+21:38 < jtanx> so fcgi_stdio must be the first include in the fcgi module
+21:38 < sam_moore> I would have thought it was simpler to just change stdout than replace the whole printf function
+21:38 < sam_moore> But whatever, we're not implementing fcgi :P
+21:38 < jtanx> haha
+21:39 < sam_moore> Oh, the mutex needs initialising, yeah
+21:39 < sam_moore> But the test I did was just a serial program, no mutex
+21:39 < jtanx> ah
+21:39 < jtanx> well
+21:39 < sam_moore> We'll be using gcc to compile on the beaglebone/rpi
+21:40 < sam_moore> I think some implementations of pthreads need an initialiser but others don't, I'll have to check
+21:40 < sam_moore> But pthread_mutex_init doesn't show up in man pages on debian
+21:40 < jtanx> yeah it's not on ubuntu either
+21:41 < jtanx> have you ever worked with clang
+21:42 < sam_moore> No
+21:42 < jtanx> neither but it seems pretty cool
+21:45 < jtanx> when you compile it gives better diagnostic output when something goes wrong
+21:46 < jtanx> it's a drop in replacement for gcc
+21:47 < sam_moore> Interesting
+21:57 < jtanx> if you kept the file handles for the sensor data always open
+21:58 < jtanx> and immediately wrote to them when a data point is polled
+21:58 < jtanx> is there much performance difference to buffering first?
+22:05 < sam_moore> Yeah, it's probably better to keep the file handles always open
+22:05 < sam_moore> The operating system will use some kind of buffer for the file anyway
+22:06 < sam_moore> I've got some basic programs, maybe I'll make some performance graphs tomorrow
+22:07 < sam_moore> On the other hand, I need to do some of the meta stuff (list of tasks that need to be completed, etc) before Monday
+22:08 < sam_moore> I should look into a #define or something to initialise the mutexes if they need to be
+22:09 < sam_moore> Anyway, I need to sleep
+22:11 < sam_moore> I wonder how James and Callum are going, they haven't been in the channel for a while
+22:12 < jtanx> yeah, it's pretty quiet
+22:17 < sam_moore> If we can get a second meeting in the week, preferably a longer one that would be good
+22:18 < sam_moore> Particularly if we can start working on code at the same time
+22:18 < sam_moore> But we'll see
+22:18 < sam_moore> Everyone has other stuff to do after all
+22:18 < sam_moore> I'm spending way too much time on this unit compared to my others
+22:18 < sam_moore> Then again this unit will probably require the most work
+22:19 < sam_moore> Anyway, bye
+22:23 < jtanx> bye
+22:46 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 22.0/20130618035212]"]
+--- Day changed Sat Aug 17 2013
+22:25 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 22.0/20130618035212]"]
+--- Day changed Sun Aug 18 2013
+21:53 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+22:37 -!- Callum [
[email protected]] has quit ["ChatZilla 0.9.90.1 [Firefox 22.0/20130618035212]"]
+--- Day changed Mon Aug 19 2013
+11:43 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+13:34 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+20:07 < jtanx> just so you know, I'm changing the JSON functions
+21:04 < sam_moore> Ok, I already pushed some stuff to github
+21:05 < sam_moore> But we can get it to merge
+21:05 < sam_moore> I ended up just using printf to make part of the JSON in the SensorHandler anyway
+21:06 < jtanx> depending on where SensorHandler is
+21:06 < jtanx> that may or may not work
+21:06 < jtanx> inside of fastcgi.c it willwork
+21:06 < jtanx> outside it won't
+21:07 < jtanx> oh ok
+21:07 < jtanx> I see
+21:07 < jtanx> so that's fine
+21:08 < jtanx> it might be best not to use a do while loop
+21:08 < jtanx> because if no arguments are passed
+21:08 < sam_moore> Ah, right
+21:08 < jtanx> then the keypair function will return null
+21:09 < jtanx> while ((params = FCGI_KeyPair(params, &key, &value)))
+21:09 < jtanx> will work fine
+21:10 < sam_moore> The KeyPair function looks like it can return an empty string though, in which case I was getting the sensor_id parsed twice when I was testing it
+21:10 < jtanx> what was your input string?
+21:11 < sam_moore> "http://localhost/api/sensors?id=0"
+21:11 < jtanx> looks fine from this end
+21:11 < jtanx> http://mctx.us.to:8080/api/sensors?id=0
+21:11 < jtanx> it might be because of your do while loop
+21:12 < jtanx> yeah it will be because of that
+21:12 < sam_moore> The do while loop causes problems when there is an empty string
+21:13 < sam_moore> ie: No parameters are passed
+21:13 < jtanx> ok let's just put it this way; FCGI_KeyPair was designed to use a while loop
+21:13 < sam_moore> Ok, sure
+21:13 < sam_moore> I had some problems with just a while loop, but I'll try again
+21:13 < jtanx> yeah about that JSON stuff
+21:13 < jtanx> I'm still trying to think of a good way to do that
+21:13 < jtanx> especially with the array stuff
+21:19 < sam_moore> I'm not sure what I did before, but a while loop seems ok now
+21:19 < jtanx> heh
+21:21 < sam_moore> Ok, I have to go now
+21:21 < sam_moore> I might not be able to do much tomorrow, we'll see
+21:22 < jtanx> ok yep
+21:35 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+22:46 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+23:14 -!- Callum [
[email protected]] has quit ["ChatZilla 0.9.90.1 [Firefox 23.0.1/20130814063812]"]
+23:35 -!- Callum [
[email protected]] has quit ["ChatZilla 0.9.90.1 [Firefox 23.0.1/20130814063812]"]
+--- Day changed Tue Aug 20 2013
+15:06 < Callum> hey same
+15:06 < Callum> sam*
+15:20 < jtanx> :>
+15:22 < Callum> he literally set himself to away as soon as i said that
+15:45 < jtanx> urgh this json stuff is doing my head in
+15:57 < sam_moore> Callum: I've been set to away since last night
+15:57 < Callum> o.O not according to my client
+15:58 < sam_moore> Anyway, we are definitely using the Beaglebone, Electronics has ordered one, they seemed to think they'd have to program it but I corrected them
+15:58 < Callum> ...why would they have to program it? wtf did they think we were doing?
+15:59 < sam_moore> They probably thought all we had to do was the GUI and not any of the other software? I don't know. But the guy seemed relieved anyway.
+15:59 < Callum> and was gonna ask if you remembered anything about the pi meson decay question for physics (how do you get the anti-neutrinos momentum, or do you assume it to be 0 as well? but then it doesnt really make sense with conservation of momentum)
+16:00 < sam_moore> Woah, I don't remember the assignments in that much detail
+16:00 < Callum> was hoping you would :p
+16:00 < sam_moore> No, I just have vague memories of algebra
+16:01 < sam_moore> And looking through notes
+16:01 < Callum> hmm. i cant seem to find anything in the notes
+16:01 < Callum> because he says to assume the rest mass of the anti-neutrino is 0
+16:02 < Callum> and i don't really remember much in regards to energies and 0 rest mass ;/
+16:02 < jtanx> I talked to someone who did computer vision before
+16:02 < jtanx> and he's really doubtful that you can get accurate readings off a webcam
+16:02 < sam_moore> Callum: I don't think you can assume the momentum is zero, other than that, I don't have much to offer
+16:02 < Callum> the question also says to have it in terms of m(pi) m(e) and c
+16:03 < Callum> and yea thats what i thought. it doesnt make sense to do that. just cant remember how to do it D;
+16:03 < Callum> @jeremy yea well, its a pain but possible
+16:04 < Callum> really comes down to how good your camera is/sampling rate/how quickly you can process it.
+16:04 < sam_moore> How about you do some experimenting with a webcam and see what you can do with it?
+16:04 < Callum> but you can get some pretty good webcams nowadays (but then again the better it is the longer it takes to process)
+16:04 < jtanx> personally, I don't think it will work
+16:04 < sam_moore> It looks like we might just end up streaming images diretly to a website
+16:04 < Callum> i don't have any
+16:04 < Callum> yea well even if thats all we do we still need the camera
+16:05 < Callum> spose i could use my laptop one but i doubt that would be very good
+16:05 < Callum> could run it through the canny algorithm and see what it produces
+16:06 < sam_moore> Sounds like an idea
+16:07 < sam_moore> A good idea specifically
+16:07 < jtanx> about the sensorhandler
+16:07 < sam_moore> Yes?
+16:07 < jtanx> do you envision leaving it in fastcgi.c permanently
+16:07 < jtanx> or for that matter any other handlers
+16:07 < sam_moore> I was kind of thinking there would be a "handlers.h" and "handlers.c"
+16:08 < sam_moore> Just to make things more organised
+16:08 < jtanx> yeah
+16:08 < jtanx> I'm trying to export enough of the functionality
+16:08 < jtanx> to do that
+16:08 < jtanx> but the json thing is annoying
+16:08 < jtanx> especially when you need to spit out arrays
+16:09 < jtanx> unless you have something like FCGI_Printf
+16:09 < jtanx> and it's up to you to format it correctly
+16:09 < Callum> bloody physics lecture video is laggy. gonna have to download it and hope for the best. fuck echo
+16:10 < jtanx> compared to lectopia you get 2x filesize with no visible benefit in quality
+16:10 < Callum> they're both shit.
+16:10 < sam_moore> You could have seperate "BuildJSON_Key" and "BuildJSON_Value" functions, with a "BuildJSON_ValueArray" maybe
+16:10 < Callum> haha
+16:11 < sam_moore> FCGI_Printf is Ok though, it's not too much formating
+16:11 < jtanx> the problem with the buildjson_* stuff
+16:11 < jtanx> is it gets very verbose
+16:12 < jtanx> and you can probably come up witha situation that breaks it too
+16:12 < jtanx> and don't get me started on value types
+16:12 < sam_moore> Haha
+16:14 < sam_moore> We can always just send plain text and make James turn it into JSON :P
+16:14 < jtanx> ahaha
+16:14 < jtanx> yeah that could work
+16:15 < jtanx> mm
+16:15 < jtanx> this is where java is good
+16:15 < jtanx> or any other higher level language
+16:21 < jtanx> ok so it's a bit of both, but how about: FCGI_JSONKey(key) and FCGI_JSONValue(format, ...)
+16:22 < sam_moore> That looks good
+16:25 < jtanx> I'm also adding long/double types for the BuildJSON function, just for convenience
+16:25 < jtanx> any preference to naming conventions?
+16:26 < jtanx> FCGI_BuildJSONLong
+16:27 < sam_moore> Seems OK
+16:28 < sam_moore> I need to go do some ENSC1001 stuff (hooray)
+16:29 < jtanx> yuck
+16:30 < Callum> hhaha
+16:30 < Callum> have fun :p
+16:30 < Callum> although i cant really talk. about a days worth of physics and a days worth of ensc3016 shit i should get through, on top of mctx and geng4402 stuff. yay for being behind already
+16:38 < Callum> well iv got an answer for the first part (second part should be the same process). just hope its right :s
+16:46 < jtanx> ok, time to merge this into the server code
+19:47 < jtanx> hmm interesting - it crashes if compiled with clang but not with gcc
+19:49 < jtanx> probably just a bug in clang 3.2
+20:08 < jtanx> ok, just submitted a pull request to update the fastcgi stuff
+20:08 < jtanx> for now I moved Sensor_Handler to sensor.c
+20:09 < jtanx> the status codes have all changed
+20:09 < jtanx> If you absolutely cannot process anything given the input arguments, you call FCGI_RejectJSON
+20:10 < jtanx> If you fail for some other reason (e.g unauthorized access), you use FCGI_BeginJSON with the appropriate status code
+20:10 < jtanx> With RejectJSON, it sends a HTTP 400 code so any query through AJAX/jQuery will fail with no extra info
+20:11 < jtanx> With BeginJSON, the HTTP code is always 200 OK, so you are able to transmit extra info if it failed for another reason
+20:12 < jtanx> BeginJSON will automatically add the module handler name + the status code
+21:47 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+--- Day changed Wed Aug 21 2013
+11:57 < jtanx> hmm
+11:57 < jtanx> I just had a play with the sensor stuff
+11:57 < jtanx> and I trialled a 'double buffer' scheme
+11:57 < jtanx> instead of the binary file idea
+11:57 < jtanx> seems to work okay, and it guarantees that a point won't be returned to the user if they have already received it
+12:35 < jtanx> urgh
+12:35 < jtanx> just worked through some stupid bug
+12:37 < jtanx> I think it's because make didn't recompile something because it thought it hadn't changed
+12:38 < jtanx> probably the header files
+12:50 < jtanx> you can see the double buffer method in this branch: https://github.com/jtanx/MCTX3420/tree/doublebuffer
+12:58 < jtanx> one issue though is that writing out csv instead of binary file takes up a lot more space
+14:49 < james__> Hey
+14:50 -!- james__ [
[email protected]] has quit ["ChatZilla 0.9.90.1 [Firefox 23.0.1/20130814063812]"]
+--- Day changed Thu Aug 22 2013
+13:20 < callum> hey
+15:07 < callum> sam you still at uni?
+15:22 < callum> or jeremy if you remember what he used to compile the file. i'v managed to get it to recognise the header files but now its complaining about not being able to find the libraries.
+15:34 < jtanx> um
+15:34 < jtanx> I can't remember
+15:34 < jtanx> didn't you use pkg config to find out
+15:36 < jtanx> try this http://opencv.willowgarage.com/wiki/CompileOpenCVUsingLinux
+22:08 < sam_moore> gcc -o opencv opencv.c -I/usr/include/opencv -lopencv_core -lopencv_highgui -lopencv_imgproc
+22:08 -!- Irssi: #mctxuwa_softdev: Total of 1 nicks [0 ops, 0 halfops, 0 voices, 1 normal]
+22:08 < sam_moore> Oh... there's no one here
+22:08 < sam_moore> Well, if you read the IRC logs when they're commited to git, you'll see it. Good luck.
+--- Day changed Fri Aug 23 2013
+07:49 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+08:59 -!- jtanx [
[email protected]] has quit ["ChatZilla 0.9.89 [Firefox 23.0.1/20130814063812]"]
+10:12 -!- jtanx [
[email protected]] has quit ["http://www.mibbit.com ajax IRC Client"]
+12:30 < jtanx_> um
+12:30 < jtanx_> do you know how you connected the relay board to the sensor board
+12:30 < jtanx_> for the soldering lab
+12:45 < jtanx_> and what sort of wire did you use?
+12:58 < jtanx_> brb
+12:58 -!- jtanx_ [
[email protected]] has quit ["http://www.mibbit.com ajax IRC Client"]
+--- Log opened Sat Aug 24 17:07:53 2013
+17:07 -!- ServerMode/#mctxuwa_softdev [+nt] by irc.eversible.com
+17:07 -!- Irssi: #mctxuwa_softdev: Total of 1 nicks [1 ops, 0 halfops, 0 voices, 0 normal]
+17:07 -!- Irssi: Join to #mctxuwa_softdev was synced in 1 secs
+17:08 -!- You're now known as sam_moore
git.ucc.asn.au / matches / MCTX3420.git / blobdiff
UCC git Repository :: git.ucc.asn.au